Firm Sovereignty Is a System, Not a Sprint: The Continuous Model Training Playbook

Nadella's framing was correct. The implementation advice that followed was mostly wrong.

In January 2026 at Davos, he said it directly: "If you're not able to embed the tacit knowledge of the firm in a set of weights in a model that you control, by definition you have no sovereignty. That means you're leaking enterprise value to some model somewhere."^[1][2] Organizations heard this as a sprint. Train a model on your internal knowledge. Own the weights. Call it sovereignty.

Fourteen months on, teams running that playbook have sovereignty over a snapshot. The weights they control reflect organizational knowledge from the moment they were trained. Since then, three pricing tiers changed, two product lines were discontinued, the support workflow was restructured, and four major policy documents were updated. The fine-tuned model talks confidently — and incorrectly — about all of it. Internal agents surface wrong answers with complete authority. That's not sovereignty. It's a well-branded liability that compounds quietly until someone notices.

Firm sovereignty is a continuous operating system, not a one-time project. The machinery that makes it real has three hard problems the "train once, deploy, done" playbook skips entirely: deciding when to retrain without wasting compute on unnecessary runs, deciding what to retrain on without triggering catastrophic forgetting, and validating the new version without disrupting the live agents that depend on the current one. This is the playbook for those three problems.

Teams that retrain on a fixed six-month schedule are usually doing it wrong in both directions simultaneously: over-training on knowledge that barely changes, and under-training on knowledge that changed last month.

Organizational knowledge decays at four distinct rates. Product specifications and pricing logic change in weeks — a new model release, a deprecated API, a quarterly pricing update. Process and policy documentation changes in months — workflow changes, compliance updates, team restructures. Org structure and people data turns over across quarters. Strategic frameworks and company values change across years, if at all.

A fine-tuned model trained on a corpus that mixes all four categories will drift in ways that are confusing to monitor. The degradation won't be uniform — the model stays accurate on slow-decay knowledge while diverging sharply on fast-decay content. When you evaluate accuracy against a golden set that samples across all categories, the aggregate score looks acceptable while the product catalog answers are dangerously wrong. The aggregate is hiding the problem.

The knowledge half-life framework classifies your training corpus by decay rate before building any retraining pipeline. Each knowledge domain gets tagged with a decay class. Fast-decay domains trigger more frequent retraining runs, or run in a hybrid architecture where a retrieval layer over a live data source handles high-churn content while fine-tuning handles stable organizational knowledge. Slow-decay domains are excluded from frequent retraining cycles — retraining on them constantly just adds noise and regression risk.

This classification also feeds the knowledge audit: before each retraining run, the audit identifies which domains have changed since the last training checkpoint, what the magnitude of that change is, and whether the change crosses a threshold that justifies retraining. Teams that skip the audit retrain on everything every time, which is expensive, slow, and adds unnecessary catastrophic forgetting risk.

Three classes of trigger should drive retraining decisions, applied in combination. No single class is sufficient on its own.

Performance-based triggers fire when your golden eval set — a fixed collection of 50–200 representative queries with verified correct answers — drops below a threshold. This is the most rigorous trigger, but it has detection lag: by the time the eval set shows degradation, real users have been getting degraded answers for weeks. Supplement with PSI-based input distribution monitoring, which detects when the kinds of questions users ask diverge from what the model was trained to handle.^[3] PSI scores below 0.1 indicate stable distributions. Scores between 0.1 and 0.25 warrant investigation. Scores above 0.25 signal major shift and should queue a retraining review.^[5]

Event-based triggers fire when a specific organizational event occurs that you know invalidates a portion of your training corpus: a major product launch, a pricing change, a policy update, an org restructure. These are the triggers that the performance-based system misses entirely — because performance hasn't degraded yet. The model simply hasn't been trained on information that now exists. Event-based triggers require a structured process where the teams that own fast-decay knowledge domains have a clear mechanism to signal the platform team that their domain has changed.

Scheduled triggers run on a fixed cadence regardless of signal. They catch slow, gradual drift that statistical tests miss until it accumulates to a detectable level. A quarterly review that checks which training corpus domains have been updated since the last checkpoint is the minimum floor for a serious sovereignty program.

The failure mode is over-triggering, not under-triggering. Teams that configure overly sensitive thresholds retrain too frequently, and frequent retraining on narrow deltas creates catastrophic forgetting — where learning new information degrades performance on previously-mastered knowledge. Require all three trigger classes to independently support a retraining decision before committing a run, rather than acting on any single signal in isolation.

The knowledge audit runs before every retraining job. Its output is not a training set — it's a decision: retrain on these domains, skip those, and here's the minimum corpus to do it safely.

The audit produces three outputs per knowledge domain:

Change magnitude measures how much the domain has changed since the last training checkpoint. A documentation update that corrects three typos has near-zero magnitude. A policy document that reverses a core business rule has high magnitude. Change magnitude is computed from diff coverage — percentage of tokens that changed — not just binary "updated or not."

Gap severity measures how bad the model's current performance is on queries that touch this domain. If the domain changed but model accuracy remains acceptable (possible when changes don't affect the semantics of common queries), retraining on that domain isn't urgent. Gap severity is measured by running the eval set filtered to that domain and comparing against human-validated reference answers.

Retraining cost estimates what the run will take. For fine-tuned adapters using LoRA or QLoRA, the cost of retraining on a narrow domain delta is low enough that the bar should also be low. For full fine-tuning runs, the cost is high enough that both high change magnitude and confirmed gap severity are required before committing.

MCP context servers are a natural data source for the audit and for the training corpus itself. If your organization runs MCP context servers for inference-time retrieval — documentation servers, CRM connectors, policy stores — those same servers feed the knowledge audit. They're already curated, structured, and maintained by the teams that own them. Converting MCP-served documents to fine-tuning examples via synthetic Q&A generation creates a consistent organizational knowledge layer: the same source drives both inference-time retrieval and periodic fine-tuning, rather than two separate systems that diverge over time.^[8]

The standard validation playbook — run the candidate against your golden eval set, check scores haven't regressed, deploy if it passes — is correct for text quality. It misses two failure modes that are specific to fine-tuned models running in agentic workflows.

Tool call schema drift is the first. When a fine-tuned model learns a slightly different output format for tool calls — a different key name, a changed response structure, an extra field it started emitting — live agents that depend on specific JSON schemas for tool invocation will either fail silently or throw parse errors. Text quality metrics don't catch this because the model is still producing high-quality text. It just happens to break the machine-readable structure that agents depend on. The fix is a dedicated tool call compliance check separate from the standard eval: run the candidate model against every tool invocation scenario in your agent's toolkit, parse every JSON it emits, and validate it against each tool's schema. Compliance must be 100%. A 99% pass rate means roughly 1 in 100 tool calls fails — and in a multi-step workflow where 5 sequential tool calls are normal, a 1% per-call failure rate translates to a ~5% failure rate per workflow completion. That's not acceptable.

Behavioral consistency is the second. A model's implicit patterns — its default strategies, confidence calibration, escalation tendencies — can shift after retraining in ways that are hard to quantify on a static eval set but immediately visible to users. The agent might become more aggressive about concluding tasks without confirmation, start producing longer responses for simple queries, or change how it handles ambiguous instructions. None of these show up as eval failures. They show up as user confusion and escalating support requests after deployment.

Shadow deployment is the production-safe gate that catches both.^[6] The candidate model runs alongside production — receiving all real production traffic in parallel, generating responses, but never serving them. The shadow model's tool call patterns and response behaviors are compared against the production model on every request. When the shadow model matches or exceeds production across 95%+ of queries with no regressions on critical cases, proceed to canary. Canary at 1–5% traffic split with automated rollback gates adds real user signal before full promotion.

The first quarter is infrastructure, not training. Teams that rush to their first retraining run before instrumentation is in place have no way to know whether the new model is better or worse than the one it replaced. Q1 deliverables: production telemetry instrumented and logging, golden eval set built from real production queries with verified correct answers, knowledge corpus classified by decay rate, and baseline drift scores established. That last point matters more than it sounds — you cannot detect drift without a stable baseline to measure against. If you start monitoring on the same day you start deploying, every signal looks like drift.

Q2 is the first full cycle. By now you have enough telemetry history to see where drift is occurring. Run the first knowledge audit, identify the fast-decay domains that have changed most since your initial training, and run a targeted fine-tuning job. The first cycle will surface gaps in your validation pipeline — your golden set will miss edge cases that shadow deployment catches, and canary will reveal behavioral inconsistencies that neither caught. Document these as improvements to the pipeline, not failures of the approach. The first cycle is supposed to be messy.

By Q3, the continuous loop should run with minimal manual intervention. Event-triggered jobs fire when fast-decay domains update. Scheduled quarterly reviews run the broader knowledge audit. The monitoring dashboard gives the platform team drift visibility without daily manual review. This is also when to run the first governance review: which domains are actually changing, which retraining jobs produced measurable accuracy improvements, and which domains might be better served by a retrieval layer than continued fine-tuning. Some fast-decay content changes faster than any retraining cadence can follow — that's a retrieval problem, not a training problem.

Q4 is measurement and course correction. At twelve months, you can compare the model's performance on proprietary knowledge queries to where it stood at initial deployment. Teams often discover two things: specific domains are drift-resistant enough to reduce retraining frequency (less cost, same accuracy), and specific domains change so frequently that fine-tuning can never keep up and a live retrieval layer is the right long-term architecture. Both discoveries inform the Year 2 operating model. Sovereignty isn't a static target — the right architecture for it shifts as you learn how your organizational knowledge actually ages.