AI Agent Maintenance: The Work That Starts After Launch

AI agent maintenance is the recurring work that keeps a deployed agent correct, affordable, and running. It means moving off retired models, fixing tools when the APIs behind them change, refreshing the data the agent reads, rerunning evals, watching cost, and staffing on-call. You rent the model, and the world the agent acts on keeps moving.

Below: each kind of drift with current provider dates, a runbook table, and a hypothetical estimate of monthly hours.

What AI agent maintenance covers

An agent depends on six things that change on their own schedules. Five are outside your control: the model, the tool APIs, the data it reads, the mix of requests users send, and the price per token. The sixth is your own change stream of prompts, code, and config.

Hidden Technical Debt in Machine Learning Systems (NeurIPS 2015) argued that real-world ML systems carry large ongoing maintenance costs, and named changes in the external world as a risk factor. Agents have more external surface, because every tool is someone else’s API.

NIST’s AI Risk Management Framework covers it in MANAGE 4.1: “Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management.” It is voluntary, and a fair checklist.

LangChain’s State of Agent Engineering surveyed 1,340 people from November 18 to December 2, 2025. Among respondents with agents in production, 94% had some observability, yet 22.8% ran no evaluations. LangChain sells an evals product and 63% of respondents work in tech, so treat it as directional.

Decision rule: if you cannot name an owner for each drift source, fix that first with the roles in who owns the AI agent.

How long does a model version last?

Anthropic’s model deprecations page promises at least 60 days’ notice before retiring a publicly released model. Those dates cover Anthropic-operated platforms. Amazon Bedrock and Google Cloud set their own schedules.

OpenAI’s deprecations page sets minimum notice of 6 months for generally available models, 3 months for specialized variants, and as little as 2 weeks for previews, unless safety or compliance concerns require faster.

What is on the calendar as of October 15, 2026:

The GPT-4o notice gives 92 days, shorter than OpenAI’s stated six-month minimum for generally available models. The policy allows exceptions for safety or compliance, and the notice gives no reason. Plan from the notice date, not the policy minimum.

Across snapshots listed on those pages, measured from the date in the model ID to retirement, lifespans run roughly 12 to 40 months. claude-opus-4-1-20250805 lasted 12 months and claude-sonnet-4-5-20250929 gets 14. At the long end, gpt-4o-mini-2024-07-18 gets about 30 and gpt-4-0613 about 40. Planning rule: plan for the short end and budget one forced migration per model per year.

Automate the watch. OpenAI’s GET /v1/models returns an optional shutdown_date per model. Anthropic’s Console exports usage by API key and model, to find callers still on a deprecated ID.

Model migrations are code changes

Take the migration Anthropic now requires, Claude Sonnet 4.5 to Sonnet 5.5 before November 30. The Sonnet 5.5 migration guide lists settings Sonnet 4.5 accepts and Sonnet 5.5 rejects with a 400 error: thinking budgets, non-default temperature, top_p, or top_k, a prefilled assistant turn, and forced tool use (tool_choice of type any or tool). The last hits agents hardest. A classification step that forces a tool call fails outright until you switch to auto with strict tools and say in the prompt when to call it.

Cost moves too. The guide says Sonnet 5.5 produces about 30% more tokens than Sonnet 4.5 for the same text, depending on content. Claude pricing lists Sonnet 4.5 at $3 input and $15 output per million tokens, and Sonnet 5.5 at $2 and $10.

Net, per unit of the same text: price falls to two-thirds and tokens rise 1.3 times, so 0.667 × 1.3 = 0.87, about 13% cheaper. If your content tokenizes at 1.5 times, the two cancel. At 1.2 times, it is 20% cheaper. Images go the other way: a 2000 by 1500 image costs about 2.5 times as many tokens. Measure your own traffic before promising a saving.

Anthropic’s /claude-api migrate command in Claude Code applies the ID swap and parameter fixes, but does not run your evals. For the full test sequence, use the model upgrade checklist in change control for AI agents.

Behavior can drift even on a pinned model

Pin a dated snapshot, never an alias the provider can repoint. The classic evidence is Chen, Zaharia, and Zou. The service called GPT-4 identified prime versus composite numbers with 84% accuracy in March 2023 and 51% in June 2023, on the same questions.

Pinning does not freeze the serving stack. Anthropic’s postmortem of three issues (September 17, 2025) describes infrastructure bugs that degraded some Claude responses from August to early September 2025. One routing bug hit 0.8% of Sonnet 4 requests at first and 16% in the worst hour, on August 31. The company wrote: “The evaluations we ran simply didn’t capture the degradation users were reporting.” Only a test set built from your task catches a drop in your task.

The AgingBench preprint (May 2026, not peer reviewed) argues that an agent’s effective state shifts after deployment, even with frozen weights, through history compression, memory retrieval, and fact revisions. One finding: behavioral tests can stay clean while factual precision decays.

Detection rule (mine): run your golden set against the exact production configuration nightly for customer-facing agents and weekly otherwise. Alert if the pass rate falls 3 points below its four-week average, or any critical case fails. Sizing and gating are in agent evals in CI.

Tool and API drift

When a tool’s API changes, the agent often does not crash. It reads a field that moved and acts on a wrong value.

Three current examples. First, the Model Context Protocol revision 2026-07-28 removed the initialize handshake and the Mcp-Session-Id header. Every request now carries its protocol version in _meta, and a mismatch returns UnsupportedProtocolVersionError. It also changed the resource-not-found error code from -32002 to -32602. A client that branches on -32002 silently stops taking its not-found path.

Second, SDK upgrades can change API versions. Stripe’s upgrade guide says its server-side SDKs call the API version that was current when that SDK release shipped. So a routine dependency bump can change the response shape a tool parses. Set the version explicitly where the SDK allows it. In Stripe’s Java, Go, and .NET SDKs the version is fixed per release, so the SDK upgrade is the API upgrade.

Third, platform features retire like models. Per OpenAI’s deprecations page, the Assistants API shut down on August 26, 2026, and the Evals platform and reusable prompt objects shut down on November 30, 2026. Prompts and eval suites that live only in a vendor dashboard are on someone else’s retirement schedule. Keep them in your repo.

Test: give each tool a contract test. Call it in a sandbox with fixed inputs and validate the response against the schema the agent expects, weekly and on every dependency bump. See the tool design guide for schemas.

Data drift: the world moves, the agent does not

Data drift takes two forms, and neither shows up in error logs.

Knowledge drift is when documents, prices, and policies the agent retrieves go stale. The model is fine. It is faithfully reading last quarter’s refund policy.

Input drift is when what users ask shifts, after a launch, a new channel, or a season. The golden set is a snapshot of past traffic, so the pass rate stays high while a growing share of real traffic goes untested.

For knowledge drift, give every source document an owner and a review date, and track the share of retrieved passages from documents past it. My threshold: above 10%, refresh the content before touching the prompt.

For input drift, classify a weekly sample of real requests into the same categories as your golden set. If an uncovered category passes 5% of traffic, add cases for it. Each quarter, replace 10% to 20% of the golden set with fresh, labeled production samples. These are my starting points, not a standard.

Cost drift

A migration can change the tokenizer. Reasoning effort settings change output tokens per call. History and tool results grow the context on long tasks. Retries multiply calls quietly. Each new tool adds calls per task.

Prices move in both directions. A footnote on Anthropic’s pricing page says Sonnet 5’s $2 and $10 rates, announced as introductory pricing through August 31, 2026, are now standard, and the scheduled increase to $3 and $15 on September 1, 2026 will not occur.

Track one number weekly: cost per successful task, meaning total model and tool spend divided by tasks that passed your success check. Total spend rises with volume. This number rises with waste, retries, and failures.

Alert rule (mine): if cost per successful task rises 20% week over week with no release in that window, investigate. Build the baseline with the formula in what an AI agent costs to run per month, and find fixes in token cost reduction.

On-call and security re-tests

Rotation design and response targets are in who owns the AI agent, and live incident steps are in the incident response runbook. Maintenance adds three duties.

First, rehearse. Run a pause-and-fallback drill each quarter, so you know the manual fallback and the stop path still work. Give whoever is on call a one-page sheet listing each model ID, its retirement date, and the fallback.

Second, re-test security on a schedule, because every new tool or data source adds attack surface. OWASP released its Top 10 for Agentic Applications on December 9, 2025, a peer-reviewed list of the most serious risks for autonomous agents. Use it as the checklist for a quarterly prompt injection and tool misuse suite, and rerun the suite when a new tool or data source goes live.

Third, plan the end. NIST’s MANAGE 4.1 lists decommissioning next to monitoring. Write down what retires the agent, such as two quarters below its business target.

The AI agent maintenance runbook

Copy this and fill in names. Frequencies and thresholds are my defaults for an agent that writes to business systems.

How many hours does maintenance take?

The common rule that maintenance costs 15% to 25% of build cost per year has no primary source I could find, so here is a bottom-up estimate instead.

Hypothetical: one support triage agent, two models (main and a cheaper router), six tools, and a retrieval index. A month is 52 ÷ 12 = 4.33 weeks and 21 working days.

That is about 43 hours a month, or 0.27 of a full-time person at 160 hours. At three incidents a month, the total is 50.75 hours. With one model instead of two, migrations fall to 2.5 hours and the total is 40.25. Eval token costs come on top.

Bottom Line

Building an agent is a project. Running one is an operation with a calendar.

That calendar now includes OpenAI shutdowns on October 23, Claude Sonnet 4.5 on November 30, and GPT-4o on January 14, 2027.

Copy the runbook, put a name in every owner cell, and automate the deprecation check this week.

FAQ

How often do AI providers retire models?

Roughly 12 to 40 months, in the snapshots named above. Anthropic says it has committed to long-term preservation of model weights, but a retired model is not callable through its API.

What happens if my agent calls a retired model?

The request fails. OpenAI says some customers may be able to provision dedicated capacity for continued access after a shutdown date, through its sales team. Anthropic’s deprecations page does not describe a similar option.

Does a Claude model retire on the same date on Bedrock and Google Cloud?

Not necessarily. Anthropic’s Sonnet 5.5 guide notes that Claude Sonnet 4 is retired on the Claude API but still available on Amazon Bedrock and Google Cloud.

Can an AI tool do the model migration for me?

Partly. Anthropic’s Sonnet 5.5 migration guide says its /claude-api migrate command asks you to confirm scope before editing, and adjusts model ID formats for Amazon Bedrock clients. You still run your golden set and re-baseline cost.

Should I keep my evals in a vendor platform?

Keep the source of truth in your own repo. OpenAI’s hosted Evals platform becomes read-only on October 31, 2026 and shuts down on November 30, 2026. OpenAI publishes a guide for moving those evals to Promptfoo.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect