Context Engineering: Build the Memory Layer Enterprise AI Actually Needs

Every enterprise deployment hits the same wall. The model is capable. The prompts are tight. The framework is wired. Then someone asks a question that requires knowing what was in a Confluence page from eight months ago, cross-referenced with a Slack thread from last Tuesday, filtered by what the user is actually authorized to see.

The agent hallucinates. Or returns something generic. Or — the failure mode that costs the most — returns a confidently wrong answer pulled from a stale document that was superseded three revisions ago.

This is the context engineering problem. It is now the primary bottleneck in enterprise AI. Gartner's framing — "context engineering is in, prompt engineering is out" — lands once you have competent models, because the quality of what you feed them outweighs how you ask^[2].

Context engineering is the discipline of structuring everything an LLM needs — prompts, memory, retrieved documents, tool outputs, conversation history — so it can make reliable decisions. For enterprise systems that means a memory layer: persistent infrastructure that gives agents access to institutional knowledge across sessions, users, and time^[1].

The non-obvious leverage point: upgrading the model is the least efficient investment once an agent is working. The same improvement in answer quality costs far less when it comes from better retrieval than from a more capable model. Spend the engineering hours on the memory layer first.

The standard RAG tutorial: chunk the documents, embed them, store them in a vector database, retrieve top-k, stuff the prompt. It works on a single PDF.

Then point it at 40,000 Confluence pages, 200,000 Slack messages, a Salesforce instance, three SharePoint sites, and a Notion workspace. Different problem class entirely.

The failure rate is stark. Between 70% and 80% of enterprise RAG programs never reach production — organizations that went wide on RAG in 2025 are hitting the same failure point: architectures built for document retrieval don't hold at agentic scale^[10]. And when they do fail, roughly 80% of RAG failures trace back to the ingestion and chunking layer, not the LLM^[5]. Most teams discover this after weeks of prompt tuning and model swaps, only to find the retrieved context was wrong before the model ever saw it.

Enterprise RAG fails in three predictable ways. Retrieval noise — the vector search returns documents that are semantically similar but factually irrelevant. "Q3 budget planning" matches "Q1 budget planning" with high cosine similarity. The Q1 doc is useless. Stale context — documents indexed once, never updated. The agent cites a policy revised two months ago. Permission leakage — the retrieval layer has no concept of who is asking. An intern's query returns the same executive compensation data as the CFO's.

Fixing these requires treating context as an engineering discipline, not a hook bolted onto an agent framework.

The instinct is to reach for semantic chunking — splitting along "meaning boundaries" instead of fixed token counts. It sounds obviously better. The benchmarks tell a different story.

A February 2026 benchmark by PremAI across 50 academic papers put recursive 512-token splitting at roughly 69% accuracy, with semantic chunking at 54% — fifteen points behind^[4]. The mechanism: semantic chunking produced fragments averaging 43 tokens. Too small. Too fragmented. The chunks lost the surrounding context that made them useful. Caveat — the benchmark used academic texts. Enterprise corpora behave differently.

We deployed semantic chunking across an enterprise corpus of 30,000 documents because it sounded correct. After six weeks we ran a retrieval recall comparison against recursive splitting. Recall was 12% lower with semantic. The corpus was mostly internal policy prose, which benefited from the larger, more stable window of recursive splitting. We migrated everything back. A painful week of re-embedding. Run the comparison before committing.

The practical default is simpler than most teams expect: recursive character splitting at 256-512 tokens with 10-20% overlap. Roughly 50-100 overlap tokens for 512-token chunks. As of early 2026 this default outperforms more elaborate strategies on most enterprise corpora. Benchmark against your own data before locking it in.

Late chunking is the most promising development here. Instead of splitting first and embedding each chunk independently, you feed the whole document into a long-context embedding model and split the resulting embeddings. Each chunk retains awareness of the full document. Pronouns resolve. Headers carry through. Cross-references stay intact.

Jina's embeddings-v4 and Voyage AI's 32K-token context window both support the pattern. The tradeoff is compute: you embed the full document instead of individual chunks. For corpora where accuracy outweighs marginal embedding cost, late chunking is worth evaluating.

The embedding market shifted hard in 2025-2026. The old default — OpenAI's text-embedding-ada-002 — is two generations behind. The current landscape has three tiers, each with sharp tradeoffs.

Voyage AI's voyage-3-large beats OpenAI's text-embedding-3-large by roughly 9.74% and Cohere's embed-v3 by roughly 20.71% on MTEB retrieval as of early 2026^[8]. Its 32K-token context window means longer documents embed without truncation — or with late chunking to preserve context across boundaries. Benchmark rankings shift as new models ship. Verify against the current MTEB leaderboard before locking in a choice.

Raw scores are not the whole story. The constraints decide:

• Already in the OpenAI ecosystem? The text-embedding-3 family gives you good-enough quality with the simplest integration. The small variant at $0.02/M is hard to beat for cost-sensitive workloads.
• Multi-modal retrieval? Jina's v4 handles text and images through one pathway, with both dense and ColBERT-style multi-vector embeddings.
• Multilingual corpus? Google's Gemini embedding leads on cross-lingual benchmarks and costs nothing through the API.
• Regulatory constraints? Self-hosted BGE-M3 or E5-Mistral give you full data sovereignty and an operational tax to match.

The vector database market consolidated fast in 2025-2026. Four options cover most enterprise use cases. The decision is primarily operational.

The honest threshold: pgvector is production-ready up to roughly 50-100M vectors, and for most enterprise RAG workloads that never becomes the constraint. If you already run Postgres, the killer advantage is transactional joins — filter by tenant_id, user_id, document_type, and created_at > now() - interval '30 days' in a single SQL query. No separate filter pipeline. No consistency gap between metadata and vector stores.

Above 50M vectors or with heavy multi-region requirements, Qdrant or Pinecone earns its place. Self-hosting Qdrant saves 3–10x over Pinecone at scale, but below roughly 60–80M queries per month, the engineering time to maintain it offsets the savings. If your team does not want to think about index compaction at 2am, pay for Pinecone.

The retrieval backend is one of the most consequential architectural choices in the stack. Three viable patterns. The right answer for most enterprises is the combination.

Vector stores win on semantic similarity over unstructured text. "How do we handle customer refunds?" surfaces documents on refund policies, return procedures, and chargeback handling — even when none of them use the word "refund." This is the breadth play. Vector search covers messy, unstructured knowledge bases where relationships between documents are implicit.

Knowledge graphs win on structured relationships and multi-hop reasoning. "Which teams depend on the payments service and what are their SLAs?" resolves through explicit traversal: payments-service → consumed-by → [checkout, subscriptions, invoicing] → SLA nodes. This is the depth play. Graphs win when the question requires traversing relationships, enforcing permissions, or reasoning across connected entities^[7].

Microsoft's GraphRAG changed the graph RAG cost calculus significantly. Full GraphRAG indexing previously cost $33,000 for a typical corpus. Microsoft's LazyGraphRAG (June 2025) defers LLM-based summarization to query time, reducing indexing cost to 0.1% of full GraphRAG while achieving comparable answer quality. At $33 for the same corpus, graph-augmented retrieval is no longer a luxury item^[11]. Still: knowledge graph construction from unstructured text requires real engineering effort. Start with a manually curated graph for the highest-value entities and expand incrementally.

The hybrid uses vectors for breadth and graphs for depth. A query router classifies the incoming question and fans out to both systems when both are needed. Results merge, deduplicate, and pass through a cross-encoder re-ranker before context assembly. Schema App's published evaluation reports 15-30% improvements in faithfulness and answer relevancy with hybrid retrieval^[6]. Actual gains depend heavily on corpus structure, re-ranking configuration, and caching. Calibrate against your own data.

Models keep getting larger context windows. Claude supports 200K tokens. Gemini 2M. GPT-4o 128K. Why does context window management still matter?

Because attention degrades with length. Mid-2025 research showed retrieval quality drops measurably even for models with massive context windows when the prompt is stuffed with too much retrieved text. Shorter, more precise context consistently produces better answers than 50K tokens of "potentially relevant" documents. The threshold varies by model and task type, but the pattern is consistent.

There's also an economic constraint that never fully disappears: at current pricing, filling a 1M-token context costs $1-5 per query. Not viable for high-volume applications. Even where the technical limit is removed, selective retrieval is the correct architecture.

The context window is a budget. You allocate a finite resource across competing demands.

Static allocation is the easy part. The real fight is dynamic rebalancing as conversations extend. As history grows, the space available for retrieved context shrinks. Four strategies handle this.

Sliding window with summarization. Keep the last N turns verbatim. Summarize everything older. The summary consumes fewer tokens while preserving key context. Simplest approach. Works for most conversational agents.

Retrieval budget scaling. Reduce the number of retrieved chunks as conversation history grows. Top-8 on the first turn, top-3 by turn fifteen. Trades retrieval breadth for conversation continuity.

Hierarchical context. Maintain two tiers — a compact summary layer always included, a detail layer included only when the query requires depth. The summary costs 200-500 tokens and provides ambient context. The detail layer carries specifics on demand.

Context eviction with recency bias. Score each piece of context by relevance to the current query and recency of insertion. Evict the lowest scores first. Requires scoring infrastructure. Holds up under long-running sessions.

Institutional memory is the accumulated knowledge an organization has about how it works. Not just documents — the decisions, the context around those decisions, the informal rules that never got written down, and the relationships between all of these.

Most enterprises have this fragmented across dozens of silos: Confluence, Jira, SharePoint, Slack, Google Drive, CRMs, ticketing systems, code repos, and the heads of long-tenured employees. Context engineering for enterprise AI means building a unified memory layer agents can query across all of these sources with appropriate access controls intact.

Production-grade agent memory in 2026 is more than document retrieval. Mature systems implement multiple memory types, each serving a distinct purpose in the agent's reasoning.

The structural insight is that these memory types have different write cadences and eviction policies. Working memory is ephemeral — it exists only for the current interaction. Episodic memory accumulates per-user and needs periodic summarization to prevent unbounded growth. Semantic memory reflects organizational knowledge and updates on the cadence of the source systems. Procedural memory changes only when agent behavior is updated.

Mem0 is the most widely deployed framework for episodic and semantic memory as of mid-2026. Its April 2025 paper (ECAI 2025) measured 91% lower p95 latency (1.44s vs 17.12s) and 90% lower token cost than the full-context approach on the LOCOMO benchmark^[12]. The mechanism: Mem0 extracts only factual knowledge from each conversation turn — condensing 26,000 tokens of conversation history to roughly 1,800 tokens of structured memory facts. That compression is what makes cross-session memory economically viable at scale.

Frameworks like Mem0 and LangGraph's checkpointing handle working and episodic memory adequately. Semantic memory is where most enterprise teams need custom infrastructure, because it ties directly into the ingestion pipeline and access control layer — the parts that cannot be outsourced.

Roughly 70% of teams running RAG in production have no systematic eval for retrieval quality^[10]. They tune prompts, swap models, and wonder why answers keep degrading — because they have no signal pointing at the actual failure.

Three metrics carry the load:

Retrieval recall — what fraction of the relevant chunks appeared in the retrieved set? Measures whether your retrieval architecture is fundamentally sound. Low recall means the right documents aren't indexed, the chunking lost critical context, or the embedding model misaligns with your query types.

Context precision — what fraction of retrieved chunks were actually relevant? Low precision wastes context window budget on noise. A cross-encoder re-ranker is the fastest fix.

Answer faithfulness — does the model's response stay grounded in the retrieved context? Catches both hallucinations and cases where the model ignores retrieved context in favor of its parametric knowledge.

RAGAS and TruLens automate all three using LLM-as-judge scoring — no labeled ground truth required for faithfulness and relevancy, though a labeled test set is still needed for recall.

The memory layer is not one thing. It's a retrieval architecture, an ingestion pipeline, an access control system, an evaluation harness, and a set of eviction policies that keep five distinct memory types consistent. Most enterprise teams build these sequentially, under pressure, and discover the architectural constraints the hard way.

The teams that get it right start with the test set and the access control design — not the embedding model choice or the vector database selection. Those decisions are easier when you know what "correct" looks like and who is allowed to see what.