Enterprise RAG Architecture in Your Own VPC: 5 Proven Layers on pgvector or Milvus

The demos all work. A few hundred documents, a hosted embedding API, a vector index, a chat window, and the answers look right. What we get asked to fix is what happens after that: the corpus is two million documents across four repositories with four permission models, the data cannot leave the region, the answers are confidently wrong about the one policy that matters, and nobody can say whether the retrieval or the generation is at fault. An enterprise RAG architecture is the set of design choices that survive that transition, and most of them are database choices, not model choices.

This post describes the enterprise RAG architecture we build inside customers' own cloud accounts or data centres, on PostgreSQL with pgvector or on Milvus, with the retrieval layer measured before the generation layer is touched. It covers where the corpus lives and how it is chunked, how to choose between pgvector and Milvus on numbers rather than preference, the hybrid retrieval queries we actually run, the evaluation harness that turns "the answers feel wrong" into a recall figure, and the access, deletion and audit controls that GDPR, India's DPDP Act and HIPAA-class programmes require.

Everything in this enterprise RAG architecture runs on PostgreSQL 16 or later with pgvector 0.8 or later, or Milvus 2.6; where a feature is version-specific we say so. Latency and recall figures are illustrative unless a measurement source is named.

Measure retrieval first, because generation cannot fix what retrieval missed

The single discipline that separates a working enterprise RAG architecture from a demo is a golden set: two to five hundred real questions from the people who will use the system, each paired with the document passages a domain expert says are the correct evidence. It is built before any index exists, and it is the yardstick for every decision that follows. Chunk size, embedding model, index type, hybrid weighting and reranker are each chosen because they moved recall on the golden set, not because a benchmark elsewhere said so.

Generation quality in an enterprise RAG architecture is evaluated separately and later, because a language model handed the wrong passages will produce a fluent wrong answer, and no prompt engineering repairs that.

-- Golden set and retrieval evaluation, PostgreSQL 16+
CREATE TABLE rag.golden_question (
    question_id       BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    question          TEXT        NOT NULL,
    asked_by_role     TEXT        NOT NULL,          -- claims handler, underwriter, engineer
    tenant_id         INTEGER     NOT NULL,
    created_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE rag.golden_evidence (
    question_id       BIGINT  NOT NULL REFERENCES rag.golden_question (question_id),
    chunk_id          BIGINT  NOT NULL,              -- a chunk the expert marked as correct evidence
    CONSTRAINT golden_evidence_pk PRIMARY KEY (question_id, chunk_id)
);

-- Every evaluation run records what each configuration retrieved, per question
CREATE TABLE rag.retrieval_run (
    run_id            BIGINT  NOT NULL,
    config_label      TEXT    NOT NULL,              -- 'hnsw_m16_ef200_hybrid_rrf60_rerank'
    question_id       BIGINT  NOT NULL,
    rank              SMALLINT NOT NULL,
    chunk_id          BIGINT  NOT NULL,
    score             REAL,
    CONSTRAINT retrieval_run_pk PRIMARY KEY (run_id, question_id, rank)
);

-- Recall@k and MRR per configuration: the numbers every design choice is judged on
SELECT
    r.config_label,
    round(AVG((hit.first_rank IS NOT NULL AND hit.first_rank <= 5)::int), 3)   AS recall_at_5,
    round(AVG((hit.first_rank IS NOT NULL AND hit.first_rank <= 10)::int), 3)  AS recall_at_10,
    round(AVG(COALESCE(1.0 / hit.first_rank, 0)), 3)                            AS mrr
FROM (SELECT DISTINCT run_id, config_label FROM rag.retrieval_run) AS r
CROSS JOIN rag.golden_question AS q
LEFT JOIN LATERAL (
    SELECT MIN(rr.rank) AS first_rank
    FROM rag.retrieval_run AS rr
    JOIN rag.golden_evidence AS ge USING (question_id, chunk_id)
    WHERE rr.run_id = r.run_id AND rr.question_id = q.question_id
) AS hit ON TRUE
GROUP BY r.config_label
ORDER BY recall_at_10 DESC;

Recall at ten is the enterprise RAG architecture number we optimise first, because it bounds everything downstream: a reranker can reorder ten candidates but cannot recover a passage that was never retrieved. In our experience a corpus that starts at recall@10 around 0.6 on a dense-only index typically reaches 0.85 to 0.9 with hybrid retrieval and a reranker; those are illustrative and the golden set will give the real figure for the corpus in question.

The corpus stays where the permissions are

An enterprise RAG architecture inherits its access model from the source systems, and the mistake we see most is flattening that at ingestion. Documents from SharePoint, Confluence, a ticketing system and a document management system arrive with four different notions of who may read them. If the chunk store drops that, the assistant becomes the one place in the company where everyone can read everything. So each chunk in the enterprise RAG architecture carries the source, the tenant, an access-control list resolved to groups at ingestion, and a content hash, and retrieval filters on the caller's groups before similarity is computed rather than after.

-- Chunk store on PostgreSQL with pgvector 0.8+; access filters are columns, not afterthoughts
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE rag.chunk (
    chunk_id          BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    tenant_id         INTEGER      NOT NULL,
    source_system     TEXT         NOT NULL,
    source_doc_id     TEXT         NOT NULL,
    doc_version       INTEGER      NOT NULL,
    chunk_ord         INTEGER      NOT NULL,
    allowed_groups    TEXT[]       NOT NULL,           -- resolved at ingestion from the source ACL
    content           TEXT         NOT NULL,
    content_hash      BYTEA        NOT NULL,
    content_tsv       TSVECTOR     GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
    embedding         VECTOR(1024) NOT NULL,           -- dimension pinned to the embedding model version
    embed_model       TEXT         NOT NULL,
    ingested_at       TIMESTAMPTZ  NOT NULL DEFAULT now(),
    deleted_at        TIMESTAMPTZ,
    CONSTRAINT chunk_doc_version_uq UNIQUE (tenant_id, source_system, source_doc_id, doc_version, chunk_ord)
);

CREATE INDEX chunk_embedding_hnsw_idx
    ON rag.chunk USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 200);

CREATE INDEX chunk_tsv_idx      ON rag.chunk USING gin (content_tsv);
CREATE INDEX chunk_groups_idx   ON rag.chunk USING gin (allowed_groups);
CREATE INDEX chunk_tenant_live_idx ON rag.chunk (tenant_id) WHERE deleted_at IS NULL;

-- Row-level security so the filter cannot be forgotten by an application path
ALTER TABLE rag.chunk ENABLE ROW LEVEL SECURITY;
CREATE POLICY chunk_group_read ON rag.chunk
    FOR SELECT
    USING (
        tenant_id = current_setting('rag.tenant_id')::int
        AND allowed_groups && string_to_array(current_setting('rag.caller_groups'), ',')
        AND deleted_at IS NULL
    );

Row-level security is the point of putting the enterprise RAG architecture chunk store in PostgreSQL. The application sets two session variables per request and cannot retrieve a chunk the caller may not read, whatever the query says. The same guarantee on Milvus is a partition key on tenant plus a filter expression on the group list, enforced by the service in front of it rather than by the store, which is workable but is one more thing to audit.

Chunking is an enterprise RAG architecture data-modelling decision

Chunk boundaries decide what a query can retrieve, so an enterprise RAG architecture treats chunking as it treats table grain. Fixed token windows are the wrong enterprise RAG architecture default for structured enterprise documents; a policy clause split across two chunks is retrieved by neither. We chunk on document structure where it exists, headings, clauses, table rows with their header, and fall back to sentence-bounded windows of roughly 300 to 500 tokens with a 10 to 15 percent overlap where it does not.

Each chunk in the enterprise RAG architecture is prefixed with its document title and section path before embedding, which costs a few tokens and measurably lifts recall on questions that name a document. Chunk size is then tuned on the golden set like everything else.

pgvector or Milvus: a decision on measured numbers

Both are legitimate enterprise RAG architecture choices, and we run both in production. The decision turns on four measurements: corpus size in vectors, query rate at the p95 latency the application needs, how often the corpus changes, and whether the surrounding metadata queries are relational. Under a few tens of millions of vectors with a p95 budget above 50 milliseconds, pgvector on a well-provisioned PostgreSQL instance is simpler to operate, joins directly to the permission and document tables, and needs no second system.

Above that, or when the enterprise RAG architecture needs thousands of queries a second with sub-20-millisecond p95, or when the index must be rebuilt frequently against a corpus that changes by the hour, Milvus with its dedicated index nodes and GPU-capable builds is the right tool.

Two pgvector specifics matter for the filtered case. Before 0.8, an HNSW scan with a selective WHERE could return fewer rows than requested because the filter was applied after the approximate search; 0.8 introduced iterative index scans, controlled by hnsw.iterative_scan, which keep scanning until the limit is satisfied.

hnsw.ef_search is the enterprise RAG architecture knob that trades recall for latency at query time; we set it per request class and record it in the evaluation run's config_label. EXPLAIN (ANALYZE, BUFFERS) on the retrieval query should show an index scan on the HNSW index with the filter applied inside the scan; a sequential scan or a post-filter with a heap sort is the signal to revisit the index or the settings.

Hybrid retrieval: the query we actually run

Dense retrieval alone misses exact identifiers, product codes, clause numbers and rare terms, which is precisely what enterprise questions are full of. The enterprise RAG architecture we ship runs a dense search and a lexical search in parallel and fuses them with reciprocal rank fusion, then hands the top candidates to a cross-encoder reranker. On PostgreSQL the whole retrieval step is one query.

-- Hybrid retrieval with reciprocal rank fusion, PostgreSQL 16+ and pgvector 0.8+
-- Session set by the application before the query:
--   SET LOCAL rag.tenant_id = '42'; SET LOCAL rag.caller_groups = 'claims,eu-staff';
--   SET LOCAL hnsw.ef_search = 100; SET LOCAL hnsw.iterative_scan = 'relaxed_order';
WITH
    dense AS (
        SELECT chunk_id,
               ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rnk
        FROM rag.chunk
        ORDER BY embedding <=> $1::vector
        LIMIT 40
    ),
    lexical AS (
        SELECT chunk_id,
               ROW_NUMBER() OVER (ORDER BY ts_rank_cd(content_tsv, q) DESC) AS rnk
        FROM rag.chunk, websearch_to_tsquery('english', $2) AS q
        WHERE content_tsv @@ q
        ORDER BY ts_rank_cd(content_tsv, q) DESC
        LIMIT 40
    ),
    fused AS (
        SELECT chunk_id,
               SUM(1.0 / (60 + rnk)) AS rrf_score        -- k = 60, the usual starting constant
        FROM (SELECT * FROM dense UNION ALL SELECT * FROM lexical) AS u
        GROUP BY chunk_id
    )
SELECT c.chunk_id, c.source_system, c.source_doc_id, c.chunk_ord, c.content, f.rrf_score
FROM fused AS f
JOIN rag.chunk AS c USING (chunk_id)
ORDER BY f.rrf_score DESC
LIMIT 20;                                               -- candidates for the reranker

In this enterprise RAG architecture, row-level security applies inside both CTEs because it is a property of the table, so the fusion never sees a chunk the caller cannot read. The twenty fused candidates go to a cross-encoder reranker running in the same VPC, which returns the five to eight passages that reach the prompt. The reranker is the most cost-effective upgrade in an enterprise RAG architecture; in our evaluations it typically adds more recall@5 than any change to the embedding model, and it is the component to add first when the golden set says retrieval is the problem.

# Milvus 2.6 equivalent: hybrid dense + sparse (BM25) search with a ranker, pymilvus 2.6
from pymilvus import MilvusClient, AnnSearchRequest, RRFRanker

client = MilvusClient(uri="http://${MILVUS_HOST}:19530", token="${MILVUS_TOKEN}")

dense_req = AnnSearchRequest(
    data=[query_embedding],
    anns_field="embedding",
    param={"metric_type": "COSINE", "params": {"ef": 100}},
    limit=40,
    expr='tenant_id == 42 and array_contains_any(allowed_groups, ["claims", "eu-staff"]) and deleted == false',
)
sparse_req = AnnSearchRequest(
    data=[query_text],                       # BM25 function on the collection embeds the text server-side
    anns_field="sparse",
    param={"metric_type": "BM25"},
    limit=40,
    expr='tenant_id == 42 and array_contains_any(allowed_groups, ["claims", "eu-staff"]) and deleted == false',
)
candidates = client.hybrid_search(
    collection_name="chunks",
    reqs=[dense_req, sparse_req],
    ranker=RRFRanker(k=60),
    limit=20,
    output_fields=["source_system", "source_doc_id", "chunk_ord", "content"],
)

Generation is the last enterprise RAG architecture layer, and it is constrained

Once enterprise RAG architecture retrieval is measured and acceptable, the generation layer in an enterprise RAG architecture has a narrow job: answer from the passages supplied, cite them, and refuse when they do not contain the answer. We run the model inside the customer's VPC or on a provider with a contractual no-training, in-region commitment, and we log the prompt, the passages, the answer and the citations for every request into ClickHouse, which is what makes faithfulness measurable. Faithfulness, the share of answer claims supported by a cited passage, is scored on the golden set with a judge model and spot-checked by the domain experts who built the set; it is reported beside recall, never instead of it.

Deletion, residency and audit are schema, not policy documents

Three obligations follow every regulated enterprise RAG architecture, and each becomes a table or a job. Deletion: when a source document is removed or a data subject exercises erasure, every chunk derived from it, every cached embedding and every logged prompt that quoted it must go, within the statutory window. That is a propagation job keyed on source_doc_id and content_hash, and it is tested quarterly with a real request.

Residency: the chunk store, the embedding service, the reranker and the generation endpoint all run in the region the data belongs to, and the deployment manifest is the evidence. Audit: every retrieval and every answer is logged with the caller, the groups in force, the chunks returned and the model version, so a question about what the assistant told a specific user on a specific day is a query.

-- Erasure propagation: soft-delete every chunk of a document and record the request
BEGIN;
INSERT INTO rag.erasure_request (source_system, source_doc_id, requested_by, legal_basis)
VALUES ('sharepoint', 'HR-2024-00913', 'dpo@example.com', 'DPDP s.12 erasure');

UPDATE rag.chunk
SET    deleted_at = now()
WHERE  source_system = 'sharepoint' AND source_doc_id = 'HR-2024-00913' AND deleted_at IS NULL;

-- Verification query, run before COMMIT and recorded with the request
SELECT count(*) AS still_live
FROM   rag.chunk
WHERE  source_system = 'sharepoint' AND source_doc_id = 'HR-2024-00913' AND deleted_at IS NULL;
COMMIT;

-- Hard delete runs later, in batches, after the answer log that cited these chunks is purged;
-- the HNSW index does not shrink on DELETE, so reindex on a schedule and track its size:
SELECT pg_size_pretty(pg_relation_size('rag.chunk_embedding_hnsw_idx')) AS hnsw_index_size;

Soft delete first, hard delete after the dependent logs are purged, is the enterprise RAG architecture order that keeps the audit trail coherent. On Milvus the equivalent is a deleted flag in every filter expression followed by a scheduled hard delete and compaction; either way, the erasure test is the query that proves zero live chunks, not a note in a ticket.

Operating the retrieval layer

The enterprise RAG architecture has the same operational needs as any database platform plus two of its own. The embedding model is versioned in the chunk table, and a model change means re-embedding the corpus into a new column or collection, evaluating on the golden set, and cutting over; running two embedding versions in one index is the classic silent-recall failure.

The HNSW index on PostgreSQL grows with deletes and updates until it is rebuilt, so index size and autovacuum on the chunk table are watched alongside recall. Enterprise RAG architecture retrieval latency is measured at p95 per request class from the application's own log, and the golden-set evaluation runs nightly against production, so a recall drop from a bad ingestion batch is a morning alert rather than a quarter's complaints.

Where this enterprise RAG architecture is the wrong answer

Three cases push an enterprise RAG architecture elsewhere. A corpus that is mostly structured, where the questions are really queries against tables, wants a text-to-SQL layer over a governed metrics layer, not passage retrieval; retrieving prose about revenue is a poor way to answer a question about revenue. A team that already runs a document platform with a competent enterprise search and permission model may get most of the value by putting a generation layer over that search's results and skipping the vector store entirely.

A use case where the answer must be exactly right, a dosage, a contractual figure, a compliance rule, is better served by retrieval that surfaces the source passage to a person than by generation that paraphrases it.

Working with MinervaDB on enterprise RAG architecture

Enterprise RAG architecture is delivered through our enterprise generative AI consulting practice, typically as a six-week build that starts with the golden set and the corpus permission audit, chooses PostgreSQL with pgvector or Milvus on the measurements above, and hands over a retrieval layer with a nightly evaluation and an erasure test that passes. The feature versioning and skew discipline is shared with our feature store architecture work, and the access and lineage controls with our data governance practice.

Under managed operations the retrieval p95, the nightly recall figure, the HNSW index size and the erasure job are monitored by our 24×7 teams under the standard S1 to S4 commitments, with a retrieval path serving results outside the caller's permissions treated as an S1. As always: test every schema, index setting and query here against your own corpus and golden set before applying it to production, and keep a tested restore posture for the chunk store and the answer log; for a regulated assistant, both are records.