Self-Hosted RAG: A Privacy-First Architecture

Self-hosted RAG isn't just a deployment choice — it's an architecture decision driven by data sovereignty, regulatory compliance, and avoiding vendor lock-in. What it actually means and when you need it.

By Tulasinath Reddy · May 15, 2026 · Tags: Graph RAG Architecture Self-Hosted

"Self-hosted RAG" gets thrown around to mean three different things. Sometimes it means the LLM runs in your network. Sometimes it means your data never touches a third-party API. Sometimes it just means you've negotiated a no-retention contract with OpenAI and called it good. These are not the same architecture and they don't give you the same privacy guarantees.

This post draws the lines clearly: what the components actually are, what each one costs, and a pragmatic middle ground that preserves real privacy without forcing you to run a 70B parameter model on your own GPUs.

The three definitions, ranked by strictness

1. Fully self-hosted (everything in your network)

Your LLM, your embedding model, your vector store, your graph store, your application — all running on infrastructure you control. No outbound traffic to third parties for inference or storage.

Use cases: classified environments, healthcare with strict HIPAA scope, defense, anywhere data egress is regulated or contractually impossible.

Cost: high. You need GPU capacity for the LLM (an H100 or two for a 70B model at usable latency), a team that can operate model serving (vLLM, TGI, or similar), and you accept that you'll trail the SOTA by 6–18 months because the best models aren't self-hostable.

2. Self-hosted retrieval, commercial LLM

Your vector store, your graph store, your retrieval pipeline — all on your infra. The LLM is a commercial API (OpenAI, Anthropic, Google) but you only send it the retrieved chunks, not your raw corpus. Your data lives in your environment; only what's needed for a specific answer leaves.

Use cases: most teams who say "we need privacy" actually need this. SOC 2 / GDPR scopes shrink dramatically because the LLM sees ephemeral, query-scoped slivers instead of bulk corpus exposure.

Cost: medium. You operate retrieval infrastructure (still real work) but you skip the model-serving layer and stay on the SOTA frontier.

3. Managed retrieval with privacy guarantees

A trusted vendor runs the retrieval stack on physically isolated infrastructure under contractual no-retention / no-third-party-sharing terms. Your data lives at the vendor but with audit trails and isolation guarantees stronger than the typical cloud SaaS posture.

Use cases: teams that need privacy without the operational overhead of building it themselves.

Cost: low. You pay per query or per credit; you don't operate anything.

What "self-hosted" doesn't fix

People deploy fully self-hosted RAG and assume the privacy work is done. Two common gaps:

The model still needs telemetry / updates. Most open-weight model deployments call home for usage analytics by default. You have to disable that explicitly.
Embedding models leak. Sentence-transformers and similar libraries often phone home on first load. Either pin specific versions and pre-bake them into your container, or audit the network behavior.
RAG itself doesn't encrypt at rest. Self-hosting puts the responsibility on you. If your vector store sits in plaintext on a shared filesystem, the threat model isn't actually better than a SaaS with proper encryption.

The realistic architecture

For 80% of teams claiming they need self-hosted RAG, option 2 (self-hosted retrieval + commercial LLM) is the sweet spot. The architecture:

┌──────────────────────────────────────────────────────────┐
│  YOUR INFRASTRUCTURE                                     │
│                                                          │
│  Documents → Chunker → Entity Extractor → Vector + Graph │
│                              ↓                           │
│  Query → Hybrid Retrieval → Top-K Chunks                 │
│                              ↓                           │
└──────────────────────────────┼───────────────────────────┘
                               ↓ (only retrieved chunks + question)
                  ┌────────────────────────┐
                  │  Commercial LLM API    │
                  │  (zero-retention SKU)  │
                  └────────────────────────┘

Critical detail: you're sending the LLM query-scoped chunks at runtime, not your bulk corpus at ingest. The LLM provider never sees your full data; they see slivers of it tied to specific user queries. This is dramatically smaller exposure than the typical "send the whole document" assumption.

What VertexRAG does

We're option 3 — managed retrieval with hard privacy guarantees: physical isolation per tenant on our servers in Hyderabad, India; no third-party LLM sharing (your documents never leave our environment except as retrieved chunks for a specific query); no training on your data, ever.

If you need option 1 (truly air-gapped), we're not the answer — you should be running everything yourself or talking to a defense-specialized vendor. If option 2 or 3 covers your privacy requirements, the VertexRAG API skips the operational burden of standing up the retrieval stack yourself.

The honest takeaway

Self-hosted RAG is a real architecture choice, not just a checkbox. Pick the level of self-hosting that matches your actual threat model, not the one that sounds most reassuring. Most teams over-engineer for a privacy concern that option 2 or 3 already addresses, and pay for it in operational drag for years.

Audit what data actually needs to stay where, pick the lightest architecture that satisfies it, and stop carrying privacy theater costs that don't change your real exposure.

Read more on the VertexRAG blog or try the Graph RAG API.