Self-Improving Agents: Turn Dismiss/Modify/Escalate Logs Into Thresholds That Converge

Ship an agent. It triages tickets, flags transactions, routes leads. Week one, it works. Week three, the team is dismissing 40% of its alerts. Week six, someone has built a private spreadsheet to track which alerts are worth opening.

That spreadsheet is the failure mode and the fix in the same place. The dismiss, modify, and escalate actions your reviewers already perform are labeled training data. The model is generating the calibration signal it needs, every day, in production. Most teams log that signal as a debug artifact and forget about it.

This is the architecture that uses it. An audit log that captures every decision with the threshold that triggered it. A weekly tuner agent that reads four weeks of human responses and proposes evidence-backed threshold changes. A human approval gate that keeps the system accountable. Eight weeks in, the agent matches your team's judgment without anyone editing a rule.

Drift is structural, not exceptional. The fraud patterns that defined last quarter's training data are not this quarter's. The triage agent calibrated for 200 daily tickets behaves differently at 2,000. The world keeps moving. The prompt does not.

Manual tuning is the obvious response and a bad one. It depends on someone noticing — usually a frustrated stakeholder filing a complaint long after dismissals have normalized. It then routes through an engineer who has to diagnose, adjust, redeploy. And every fix addresses the symptom that escalated, not the structure underneath it. The cycle is whack-a-mole because the inputs are anecdotes.

The scale of the problem is concrete in domains where alerting agents have been deployed longest. Transaction monitoring false positive rates sit at 93% across the industry in 2025 — down from 97% in 2015, which means a decade of investment bought four percentage points of improvement with no feedback loop in place.^[9] A 2025 ISACA analysis tracked self-modifying AI systems with no structured feedback mechanism and found roughly 3x higher rates of unexpected behavioral change versus systems with formal oversight — the magnitude varies by system type but the direction does not.^[5]

The fix is not heroics. It is plumbing. Every human response to an agent decision becomes structured data. A second agent reads that data and proposes calibrations. A human approves them. The loop closes.

Before you can tune anything, you need something to tune from. That means logging every agent decision with enough structure to reconstruct why it fired and what happened next.

The audit log is not a debug log. A debug log records what code ran. The audit log records what the agent decided, what evidence it used, and how a human responded. Three fields, one row, queryable for pattern analysis.

Store the entries somewhere queryable with time-range indexing. A PostgreSQL table with JSONB columns covers most teams. High-volume systems should partition by week — the tuner only ever reads the last four.

The human response field starts empty and gets backfilled as reviewers process their queue. That asynchrony is the whole point. The agent does not block waiting for a human. The human responds when they get to it. The tuner reads the joined record on a weekly cadence. Every layer runs at the speed it can sustain.

One implementation detail that bites teams: the timeToRespond field is not just metadata. Fast responses (under 60 seconds) from the same reviewer on the same day are a signal that the reviewer is triaging by habit, not evidence — exactly the pattern that produces an unreliable training signal. The tuner should weight rapid dismissals lower than deliberate ones. Build that into the clustering logic from the start, not as a retrofit.

Four components, four cadences. Decoupling them is the architecture.

The Primary Agent runs in real time. It reads the current threshold config, makes decisions, writes every one to the action log. It never blocks on a human.

The Action Log accumulates decision records. Reviewer responses backfill async — most teams see the human field populated within 24–48 hours.

The Tuner Agent runs once a week, typically Sunday night or Monday morning. It reads four weeks of joined data, clusters dismissals and escalations, and emits a structured proposal with evidence counts.

The Approval Interface routes the proposals to a designated approver — a team lead or ops owner. They accept, reject, or modify each one. Nothing reaches the live config without explicit human sign-off.

The cadence gap between components is not a weakness. It is why the loop is safe. Real-time agent decisions can't accidentally trigger threshold changes. Threshold changes can't bypass human review. Each boundary is a deliberate control point, not an accident of scheduling.

The tuner is not a fine-tune. It is not a training run. It is an LLM analyst that reads structured logs and produces structured proposals. Treat it as the data analyst who works exclusively on your agent's behavior, on a weekly schedule, with no other distractions.

Three phases per cycle: pattern detection, root cause clustering, proposal generation.

We got the third phase wrong on the first build. We let the tuner emit as many proposals as it found patterns for, assuming throughput would speed convergence. It did the opposite. Approvers waved through too many changes in a single week, and when missed escalations spiked the following week, no one could attribute the regression to a specific change. The 3-proposal cap looked arbitrary until we ran it. The cap forces the tuner to rank by expected impact, which forces it to write better proposals. The constraint is the leverage point.

The approval gate is the structural difference between a self-improving system and an unsupervised one. The tuner emits proposals. A human accepts, rejects, or modifies. The gate is not decoration — it is the mechanism that keeps the system from optimizing against criteria it set for itself.

A 2026 OneReach AI enterprise study tracked agentic AI deployments and found that systems with human-in-the-loop oversight ran roughly 60% fewer production incidents than fully autonomous deployments — the spread varies by use case but the pattern holds.^[4] The approver does not need to read every statistical detail. They need to answer one question: does this change match how we want the system to behave?

The approver role is not a rubber stamp. It is the one place in the loop where institutional knowledge, regulatory context, and domain judgment enter the system. Engineers who treat approvals as overhead are misreading the architecture.

The convergence pattern is predictable enough to plan around. The reason teams need to know it: phase one looks like nothing is working, and the temptation to pull the plug peaks exactly when the loop is collecting the data it needs. Set expectations on the curve, not the first cycle.

Self-improving systems fail in ways that look like success until they don't. The loop can converge on the wrong target, amplify a latent bias in the training signal, or drift past equilibrium into excessive conservatism. All three failure modes share one feature: they are invisible unless you are tracking the right counters.

Reward hacking. The tuner optimizes the metric it can see — dismissal rate. If reviewers start approving more alerts simply because the tuner has pushed thresholds up and fewer alerts fire, dismissal rate drops, the system looks healthy, and actual detection quality is unchanged. Defense: track true positive rate separately from dismissal rate. The two should move together. If dismissal rate falls while confirmed escalations also fall, you have drifted into over-suppression.

Reviewer habit drift. This is the inverse problem. Reviewers who process 100+ alerts per day develop pattern recognition that shortcuts deliberate review — exactly what produces the 22% accuracy decline documented in high-volume alert environments.^[10] They dismiss quickly; the tuner reads the fast dismissals as signal; thresholds lift; more real signals get suppressed. Defense: the timeToRespond weight discussed in the schema section. Fast dismissals from the same reviewer in a single session get down-weighted in the clustering phase.

Threshold creep toward silence. The tuner optimizes against dismissals. Dismissals drop. The system slowly tightens until it suppresses real signals to maintain a clean dismissal rate. The loop has now confused silence with accuracy. Defense: alert when escalation rate falls below historical baseline alongside dismissal rate. Both should fall together in healthy convergence. If escalations fall faster than dismissals, the system is going quiet, not sharp.

Self-improving agents are not a research problem. They are a plumbing problem. The agents in production today that stay sharp are not running clever architectures. They are running boring, well-instrumented feedback loops. An audit log with the right schema. A tuner on a weekly schedule. A human at the gate. Eight weeks.

There is a counterpoint worth naming, because it is the failure mode this architecture does not solve. The loop assumes the underlying task is stable. If your business logic shifts every quarter — pricing rules, compliance rewrites, product catalog churn — automated threshold tuning will mask the need for a real architecture revision. The agent will converge beautifully on a target that is no longer the target. The loop is a precision tool for a stable domain. It is not a substitute for rethinking a misaligned system.

The team is already producing the calibration data. The infrastructure to use it is the work.