Your RAG Pipeline Isn't Hallucinating. Your Data Layer Is.

A team spends three months wiring a RAG pipeline into the internal knowledge base. The demo runs. The system confidently cites a policy document that was superseded eighteen months ago. Or it merges pricing from two contradictory spreadsheets into a single wrong answer. Or it fabricates a procedure that sounds plausible and never existed.

The instinct is to blame the model. Swap in a bigger one. Tune the retrieval. Add reranking. The root cause sits one layer up. The documents are stale, contradictory, duplicated, or chunked into fragments that strip the context away.

Gartner puts the cost directly: roughly 60% of AI projects may be abandoned by the end of 2026 because organizations lack AI-ready data^[1]. The directional number tracks with what practitioners report. Around 61% of companies say their data is not ready for generative AI; about 42% killed at least one AI initiative in 2025 because data quality could not be fixed in the timeline available^[5]. Most production RAG pipelines stall at 65% accuracy — and the cause isn't the model, it's the ingestion layer^[9].

The counterintuitive part: the data exists. Every organization with three years of operational history has a knowledge problem, not a knowledge gap. The failure mode is unmanaged surplus, not scarcity.

When humans read documents, they apply judgment in real time. They notice the date on a policy. They check whether it was superseded. They weigh conflicting sources. A language model does none of that. Every chunk in the retrieval window arrives at the same authority level.

The data layer has to do the judgment work before retrieval, because the model will not. Single source of truth is not a database. It is a discipline enforced inside the pipeline so every chunk that reaches the model is current, canonical, and unambiguous.

Most organizations conflate having data with having a source of truth. The data is everywhere — Confluence, Google Docs, Notion, Slack, SharePoint, PDFs in a shared drive nobody has opened in two years. Multiple versions of the same content coexist with no signal about which one is authoritative. The pipeline either picks a winner upstream or the model picks one downstream. There is no third option.

Every broken RAG deployment maps to one or more of three failure modes. Name the mode first. Then decide where to spend the engineering hours.

Failure Mode 1: Staleness. Documents that were correct when written are now wrong. The Q2 2024 pricing page still sits in the index next to the current one. Both look equally relevant to the embedder. The model has no way to prefer the newer version. The staleness problem compounds with deletion bugs — removed content can remain retrievable and appear as legitimate evidence^[9]. Analysis of enterprise RAG deployments traces roughly 38% of retrieval errors to outdated content that was never archived or versioned — the rate varies by corpus and organization, but the pattern repeats^[3].

Failure Mode 2: Contradiction. Different teams maintain their own versions of shared information. Sales documents one set of product capabilities, marketing documents another, engineering's internal docs describe a third. Retrieval pulls chunks from all three. The model reconciles by inventing a plausible synthesis that matches none of the originals. The output looks fluent. It is wrong in a way that is hard to debug because no single chunk is the source of the error.

Failure Mode 3: Implicit Knowledge. Business rules, decision criteria, escalation triggers — the things every experienced employee carries in their head — were never written down. The model cannot retrieve what does not exist as text. The RAG system answers the documented question correctly and misses the operational context any human would apply by reflex. On average, 31% of real user queries fall outside the distribution of queries used to train or evaluate the embedding model, and in these zero-shot cases retrieval failure rates jump by 40%^[9].

Five stages. Each owns one responsibility. Each has a quality gate before content moves forward.

Stage 1: Collection. Connectors pull from every source — Confluence, Notion, Google Docs, SharePoint, Slack, email archives, PDFs. The discipline here is completeness. Cover the messy sources, not just the tidy ones. The Slack threads and one-off Google Docs are where institutional knowledge actually lives. Skip them and the pipeline goes live missing the context that matters most.

Stage 2: Ingestion and Normalization. Raw content collapses to a common format. HTML stripped, PDFs parsed into structured text, images OCR'd where needed. Every document gets a standard metadata envelope: source system, original URL, last-modified timestamp, author, content hash. The envelope is non-negotiable. Documents without it cannot be filtered downstream.

Stage 3: Quality Gate. This is where most pipelines are weakest. Each document passes validation. Is the content parseable? Is the modification date inside the staleness threshold? Does it duplicate an existing document by hash or by semantic similarity above the threshold? Failures route to quarantine. Failures do not silently disappear. A document you cannot find is worse than a document you rejected on purpose.

Stage 4: Canonical Resolution. When two documents cover the same topic, the pipeline picks a winner. This is the hardest engineering problem in the stack and the one most teams skip. Resolution strategies: prefer the most recently modified, prefer the document from the designated authoritative source for that domain, escalate to human review when confidence is low. Picking the winner upstream is the only mechanism that prevents the model from picking the wrong one downstream.

Stage 5: Metadata Enrichment. The surviving canonical content gets tagged — topic categories, content type (policy, procedure, reference, tutorial), confidence score, expiration date, ownership. This metadata is what powers filtering at query time. Without it, retrieval is a popularity contest among chunks. With it, retrieval can prefer authoritative, fresh, in-domain content.

Most teams ship with whatever their vector DB documentation suggests — 512 tokens, 50-token overlap — and never revisit it. That default destroys multi-section policy documents, fractures procedural guides mid-step, and produces fragments so short they carry no context.

A NAACL 2025 peer-reviewed study from Vectara evaluated 25 chunking configurations across 48 embedding models^[10]. The finding runs counter to the assumption that semantic chunking is always better: on realistic document sets, fixed-size chunking consistently outperformed semantic chunking end-to-end. Semantic chunking's topic-detection algorithm produced fragments averaging just 43 tokens — clean in isolation, but too short to give the LLM enough context to construct useful answers. Recursive 512-token splitting achieved 69% accuracy on the benchmark; semantic chunking landed at 54%^[10].

The exception matters. A peer-reviewed clinical study found adaptive chunking hit 87% accuracy versus 13% for fixed-size baselines on medical documents^[10]. Domain specificity is the variable. When your corpus is homogeneous — all API docs, all legal contracts — semantic chunking can outperform fixed splitting. When it's mixed, start fixed and measure.

When raw enterprise data — multiple iterations of the same report, overlapping API documentation, cross-referenced knowledge-base articles — is naively chunked and embedded, the resulting vector store contains near-identical coordinate points. Standard top-k retrieval at those duplicated points returns multiple copies of the same underlying passage. The model sees three versions of the same paragraph, treats them as corroboration, and confidently answers with whatever that paragraph says — even if it's outdated.

Approximately 80% of accuracy issues in production RAG originate from data-quality problems including duplication^[11]. Two complementary techniques handle this at scale:

Exact deduplication catches byte-identical or near-identical documents using SHA-256 content hashes. Fast, cheap, catches the obvious case of the same PDF uploaded twice.

Semantic deduplication catches paraphrase-level duplicates using embedding cosine similarity. Milvus introduced native MinHash-LSH indexing in 2025 specifically for customers deduplicating multi-billion-document indices^[11]. For most enterprise corpora under 10M documents, cosine similarity above 0.92 is a practical threshold — below that, you start collapsing genuinely distinct documents.

There is a persistent fantasy that a smart-enough model can clean up messy documentation on the way through. It cannot. Documentation quality is the hard ceiling on what the AI system can produce. Every retrieval error is bounded by the worst document in the corpus.

Documentation written for humans does not survive chunking. Humans tolerate ambiguity, skip sections, and infer context from layout. A retrieval pipeline shreds documents into fragments, and each fragment has to stand on its own. The doc that reads well end-to-end becomes incoherent the moment its third paragraph lands in a context window without the first two.

The hardest data quality problem is not fixing bad documents. It is capturing the rules that were never documented in the first place. Every organization runs on a layer of implicit business rules carried in employees' heads.

A support agent knows that when a customer mentions 'enterprise plan,' they should check whether the account was migrated from the legacy billing system, because legacy accounts have a different rate structure. An engineer knows the staging environment has a 2GB memory limit nobody documented and several test suites depend on. A salesperson knows that deals above $500K need VP approval even though the CRM workflow does not enforce it.

These rules are invisible to the RAG system. They are also exactly the context that separates a useful AI answer from a technically-correct, practically-wrong one. The fix is not to hope the model figures it out. The fix is to extract the rules, store them as structured content, and feed them into retrieval as first-class documents.

When the pipeline finds two documents covering the same topic and saying different things, something has to pick the authoritative version. This is canonical resolution. Get it wrong and the RAG system inherits every internal contradiction the organization has ever produced.

We defaulted to recency-weighted resolution at first — the most recently modified document wins. It worked for 80% of cases. It failed catastrophically on the remaining 20%. The HR policy that got a typo fix in the title last month became 'more recent' than the finance policy that was comprehensively rewritten six months ago. Recency is a weak proxy for authority. The right resolution depends on who owns the domain.

Three strategies, ranked by reliability.

Raw document content is necessary and not sufficient. The metadata envelope around each document is what lets the RAG system make filtering decisions the embedder cannot — preferring authoritative sources, dropping expired content, boosting in-domain results.

The schema below is the starting point. Every document in the canonical store carries these fields. Add fields as the pipeline matures; do not start with the full 20-field schema and spend two months on enrichment before measuring retrieval quality.

A clean data layer is not a one-time project. Without active maintenance, the entropy that produced the original mess will reproduce it within months. Drift is the default. The monitoring stack runs three layers.

The most overlooked metric is quarantine rate by source. When a particular source consistently fails the quality gate, that is not a pipeline problem. It is a source problem upstream of you. Talk to the team that owns the source. The pipeline is doing its job by rejecting the content. The fix is not in the pipeline.

A weekly data quality digest closes the loop: total documents in the canonical store, new documents ingested, documents quarantined with reasons, documents expired, retrieval accuracy score. The digest goes to whoever owns the data layer — not into a dashboard nobody opens. Ownership is observable when someone reads the digest and pushes back on the source teams whose quarantine rates are climbing.

The model is not the bottleneck. The corpus is. Every team that fixed retrieval by swapping the model is back at the same accuracy ceiling six months later, paying more per token. The teams that fixed the data layer are still shipping on whatever model is current — and their retrieval accuracy keeps climbing because the foundation is doing the work the model cannot.

Clean data is not a foundation you finish. It is a discipline you maintain. The pipeline runs every day. The audits run every month. The ownership is named and observable. Skip any of that and the entropy wins.