Vector Data Engineering: Proven pgvector, Milvus & RAG 24×7

The work begins at the source of truth. Product catalogues, support tickets, contracts, clinical notes and telemetry already live in PostgreSQL, MySQL, SQL Server, MongoDB or a lake, and the embeddings derived from them are just that: derived. Our Vector Data Engineering pipelines treat vectors as a rebuildable projection of operational data, keyed idempotently on the document, the chunk and the embedding model version, so a model change, a chunking change or a corrupted index is a rebuild, never a data-loss event.

Between source and index sit the decisions that determine quality. Chunk size and overlap change recall more than any index parameter does. Layout-aware parsing of PDFs and tables changes what a chunk even contains. The embedding model fixes the dimensionality, the distance metric and the memory bill for the life of the index. Each of these is a Vector Data Engineering decision made against a golden query set for your corpus, not from a vendor tutorial.

On the serving side, Vector Data Engineering means a retrieval path that combines approximate nearest-neighbour search with lexical matching and metadata filters, fuses the candidates, reranks the top of the list and hands the result to an application, whether that is a retrieval-augmented generation assistant, a recommendation service or an anomaly detector on transaction streams. Every stage has a latency budget and a quality metric, and both are measured in production, not estimated in a notebook.

The evaluation loop closes the system. Recall at k against an exact scan, nDCG on the golden set, faithfulness of generated answers, p95 latency per stage, index lag versus the source and embedding drift after a model change are the signals that tell us whether to change the chunker, the model, the index parameters or the engine. That loop is what separates Vector Data Engineering from a proof of concept that happened to work on the demo corpus.

HNSW: a navigable graph

Hierarchical Navigable Small World indexes keep every vector in a bottom layer with M links to near neighbours, and a shrinking set of vectors in upper layers that act as express lanes. A query enters at the top, greedily descends and then widens its beam at layer zero to ef_search candidates. Higher M and ef_construction build a better graph at the cost of build time and memory; higher ef_search raises recall and latency together. This is the index in pgvector, Milvus, ClickHouse, Redis, Valkey and MariaDB, under slightly different names, and it is where most Vector Data Engineering tuning time goes.

The trap is filtered search. When a tenant or date filter removes most of the graph, a naive HNSW traversal runs out of matching candidates before it reaches k results and returns fewer, worse rows. pgvector 0.8.0 added iterative index scans that keep walking until k rows pass the filter; Milvus pushes the filter into the segment scan; Qdrant and others use filter-aware graph traversal. Which mechanism your engine has, and how it behaves below about 2% selectivity, is the first thing our Vector Data Engineering assessment measures.

IVF: partition then probe

Inverted-file indexes cluster the space into nlist cells around trained centroids and search only the nprobe closest cells. They build faster and use less memory than HNSW, and pair naturally with product or scalar quantisation, which is why Milvus IVF_PQ and the newer IVF_RABITQ are the memory-economic choice at hundreds of millions of vectors. The cost is that the centroids go stale as data shifts: a re-embedding, a new tenant with a different vocabulary, or a large backfill all warrant a retrain, and our Vector Data Engineering runbooks schedule it from a recall alert rather than a calendar.

Quantisation and two-stage search

A 1,536-dimension float32 vector is 6,144 bytes before any graph overhead. Half precision halves it (pgvector halfvec, Milvus FLOAT16_VECTOR), scalar int8 quarters it, and binary or RaBitQ representations bring it to under 200 bytes. Recall drops with each step unless the quantised index is used only to generate candidates and a full-precision rerank of the top 100 to 200 restores the ordering. Milvus reports IVF_RABITQ with SQ8 refinement at roughly 95% recall on a quarter of the memory; that is a vendor figure, and we reproduce it on your corpus before it enters a sizing decision.

Reading the plan as Vector Data Engineering does

On PostgreSQL the question is always whether the planner used the vector index at all. An ORDER BY embedding <=> $1 LIMIT 10 with a selective WHERE clause can flip to a sequential scan with an exact sort, which is correct and slow. We read it the same way we read any other plan.

-- pgvector 0.8.x on PostgreSQL 16+: confirm the HNSW index is used and
-- see how many candidates the iterative scan had to walk.
SET hnsw.ef_search = 120;
SET hnsw.iterative_scan = relaxed_order;
EXPLAIN (ANALYZE, BUFFERS)
SELECT chunk_id,
       doc_id,
       embedding <=> ${QUERY_VECTOR} AS distance
FROM   doc_chunks
WHERE  tenant_id = ${TENANT_ID}
ORDER  BY embedding <=> ${QUERY_VECTOR}
LIMIT  10;
-- Expect: Index Scan using idx_doc_chunks_embedding_hnsw on doc_chunks
-- Not:    Seq Scan ... Sort ... (exact but unbounded at scale)

In Vector Data Engineering, idempotency comes from the key. Every chunk is addressed as doc_id + chunk_no + model_version, so the same chunk re-processed from a retry, a replay or a backfill is an upsert rather than a duplicate, and a delete on the source tombstones every chunk of that document. Content hashes let the pipeline skip chunks whose text has not changed, which is most of them on a daily run.

Replayability, the second Vector Data Engineering property, comes from change data capture. Debezium on PostgreSQL logical replication, MySQL binlog or MongoDB change streams publishes inserts, updates and deletes to Kafka with the log position attached. A stream processor in Flink, Kafka Streams or plain workers chunks, dedupes and batches 64 to 256 chunks per embedding call, and the consumer offset is the recovery point. Index lag, measured as source updated_at against vector indexed_at, is a service-level indicator with an alert.

Embedding serving runs in your VPC on vLLM, Text Embeddings Inference or Triton when data governance requires it, or against a managed endpoint when it does not; either way it is batched, rate-limited and retried with back-pressure to the stream, so a slow model never becomes a lost event.

Re-embedding is the operation most teams postpone because it looks like downtime. Under our Vector Data Engineering pattern a new model, or a new dimension, is a second consumer on the same stream from the same offset, writing into a second collection or table built in parallel. The application reads through an alias. Promotion is gated on parity (chunk counts per tenant within 0.1%), quality (recall at 10 and nDCG on the golden set at least equal to the live index) and latency (p95 at production concurrency within budget, with ef_search or nprobe retuned for the new dimension).

The alias moves in one metadata operation, the old index stays warm for a week, and rollback is pointing the alias back.

The same mechanism handles a chunking change, a corrupted index, an engine migration and a major-version upgrade of the vector store itself. Nothing is rewritten in place, which is the property that lets us commit to change windows measured in minutes.

Sizing the index: Vector Data Engineering arithmetic

The estimate starts from vector count times bytes per vector at the chosen precision, plus graph overhead of roughly M × 2 × 4 bytes per vector for HNSW at layer zero, plus the metadata columns that filters need. Ten million 1,536-dimension float32 vectors is about 61 GB of vectors alone; at half precision it is 31 GB, and at binary with full-precision rerank the index is under 2 GB with the rerank served from disk. We produce this table for your corpus at three precisions and three engines, then load a representative sample and measure, because page layout, segment sealing and index metadata differ enough between engines that arithmetic alone is not a sizing.

Build performance in Vector Data Engineering

HNSW builds are CPU-bound and, on PostgreSQL, memory-bound: pgvector builds the graph in maintenance_work_mem and spills to disk when it does not fit, at which point build time grows by an order of magnitude. Parallel builds with max_parallel_maintenance_workers help on PostgreSQL 15 and later; 0.8.2 fixed a buffer overflow in the parallel path, so we pin that release. On Milvus the build runs on data nodes in 2.6 and its throughput sets your re-embedding window. On ClickHouse the index is built per part at merge time, so index_granularity and merge settings determine both build cost and query pruning.

Query-time knobs, per engine

Each dial is swept against the golden set in every Vector Data Engineering tuning cycle and plotted as recall against p95, and the operating point is chosen from that curve with headroom for growth. The sweep is repeated after any re-embedding, because the curve moves with the model.

-- Ground truth for recall@10: exact scan on a 1,000-query sample.
-- Run once per model version; store the result as the golden set.
SET LOCAL enable_indexscan = off;
SELECT q.query_id,
       ARRAY_AGG(c.chunk_id ORDER BY c.embedding <=> q.embedding) AS exact_top10
FROM   eval_queries AS q
CROSS  JOIN LATERAL (
         SELECT chunk_id, embedding
         FROM   doc_chunks
         WHERE  tenant_id = q.tenant_id
         ORDER  BY embedding <=> q.embedding
         LIMIT  10
       ) AS c
GROUP  BY q.query_id;