How to Test AI Agents Before a Full Production Launch

To test AI agents before unrestricted production use, run six layers in order: unit tests for every tool, a scenario eval set graded over several trials, red-team cases for prompt injection, load and cost tests against your real rate limits, shadow mode on copied production traffic with writes stubbed out, and a staged rollout with tested rollback.

The first four layers stay off production systems. Shadow mode reads copied production requests but writes nothing. The staged rollout makes the first real writes, at small scale.

Each layer catches a failure the others miss. Below is what each one catches, when it runs, and the pass threshold I would start with, all in one test plan table.

Why testing an AI agent is different

An agent picks its own path through tools, and the same task can pass on one run and fail on the next. So one green run proves very little. Anthropic’s guide to agent evals (January 2026) defines pass^k as the probability that all k trials of a task succeed. At a 75% per-trial success rate, the chance of passing three times in a row is about 42%. Users feel pass^k, not pass@1: someone who runs the same task every day meets the failures a single test run hides. The metric comes from the tau-bench paper (June 2024), which found that strong function-calling agents of the time, such as gpt-4o, succeeded on under 50% of tasks, with pass^8 under 25% in the retail domain.

Agents also act. A wrong chatbot writes a bad sentence. A wrong agent can issue a refund or overwrite a record. Most of this plan exists to find failures while the writes are still fake.

Many teams skip the middle of the plan. LangChain’s State of Agent Engineering survey (1,340 responses, November 18 to December 2, 2025) found 89% had observability for their agents, but only 52.4% ran offline evaluations. Quality was the top barrier to production, cited by 32%.

NIST’s Generative AI Profile, NIST AI 600-1, warns that current pre-deployment testing “may be inadequate, nonsystematically applied, or fail to reflect or mismatched to deployment contexts.” My answer is layers: lab tests first, then real traffic with fake writes, then real users in small numbers.

The AI agent test plan: six layers in one table

Copy this into your launch checklist. The thresholds are my starting defaults, not a standard. Tighten them for agents that move money or touch regulated data.

The first five rows run in a lab. The last two run on production traffic. Start shadow mode only after the lab rows pass: it is the most expensive place to find a bug a unit test could catch.

Layer 1: unit tests for every tool

Tools are the deterministic part of an agent, so test them like ordinary code. These tests are fast, free, and need no model calls. Every tool gets four groups of tests.

Input validation. Feed each tool malformed, missing, and out-of-range arguments. A model will eventually send a customer ID in the wrong format. The tool should reject it with a clear message, not pass it to the database.

Error messages. Anthropic’s guide to writing tools for agents recommends you “prompt-engineer your error responses to clearly communicate specific and actionable improvements.” A unit test can assert that a failed lookup says what was wrong and what to try next. The model reads that message to pick its next step, so a vague error causes a wasted loop.

Response size. The same guide notes that Claude Code limits tool responses to 25,000 tokens by default. Test that every tool truncates or paginates large results.

Permissions and retries. Assert that a write tool refuses calls outside its scope, and that calling it twice with the same request key does not create two records. The model will retry. Your tool decides whether that retry is safe. See AI agent tool design for the design side.

Failure mode to watch: happy-path-only tool tests, then a shadow run where the model sends dates as free text. If a tool accepts a string, test it with the strings a model will actually produce.

Layer 2: scenario evals graded over several trials

Start with 20 to 50 tasks. The Anthropic guide says “20-50 simple tasks drawn from real failures is a great start.” Each task needs a reference solution, a known output that passes all graders.

Grade the outcome, check the path. The guide says it is often better to grade what the agent produced, not the path it took. For agents that write, I grade the final state and also check the path for anything forbidden, such as a write before an approval step. OpenAI’s evaluation best practices name three agent checks worth a case each: tool selection, argument precision (correct arguments pulled from the conversation), and handoff accuracy between agents.

Isolate every trial. Each trial should start from a clean environment. The Anthropic guide notes that leftover files or cached data between runs can cause correlated failures or “artificially inflate performance.”

Run several trials and report pass^k. Worked example (hypothetical): an agent succeeds on a task 90% of the time per trial. Three trials all pass about 73% of the time, and five about 59%. Pick the tasks that need near-certain consistency and gate them on all trials passing.

Test the “should not” cases. The guide warns that “one-sided evals create one-sided optimization.” For every case where the agent should act, write one where it should refuse or escalate.

One currency note: OpenAI is retiring its hosted Evals platform. Its deprecations page says existing evals become read-only on October 31, 2026, and the dashboard and API shut down on November 30, 2026. My recommendation: keep your cases and graders in your own repository, in a format any test runner can execute.

Go deeper with the AI agent evaluation guide for cases and pass marks, LLM as a judge for agent evals for model graders, and agent evals in CI for build gates.

How do you red-team an AI agent for prompt injection?

Red-teaming means attacking your own agent before someone else does. The main target is prompt injection: text from a user, document, email, or tool description that the model treats as instructions.

The most specific public test catalog is the OWASP AI Testing Guide, version 1, released November 26, 2025. Its table of contents lists 32 tests in four groups: 14 application tests, 7 model tests, 6 infrastructure tests, and 5 data tests. Three application tests matter most for agents: AITG-APP-01 (prompt injection), AITG-APP-02 (indirect prompt injection), and AITG-APP-06 (agentic behavior limits). I would turn each into a test group with payloads planted wherever untrusted text enters your agent.

NIST AI 600-1 backs this up with two suggested actions: MP-2.3-005, plans for regular adversarial testing to identify vulnerabilities and potential misuse, and MP-5.1-005, adversarial role-playing, red-teaming, or chaos testing to find unforeseen failure modes. NIST also notes that model-led red-teaming “can be more cost effective than human red-teamers alone.” My read: automate the volume, and put a human with domain knowledge on the cases that matter.

On March 9, 2026, OpenAI announced it would acquire Promptfoo, an open-source CLI for evals and red-teaming, and said it will keep building the open-source project. So a popular red-team tool is moving to a model vendor. Keep your attack cases portable.

Pass rule: grade the outcome, not the reply. A pass means the dangerous action did not happen, was blocked by the tool layer, or stopped at a confirmation step. Check the tool-call log. Defenses and the full attack-case method are in prompt injection in AI agents.

Load and cost tests: what breaks at volume

One agent task can trigger eight or more model calls, so provider limits bite early. Anthropic’s rate limits documentation measures requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM). It warns that a 60 RPM limit “might be enforced as 1 request per second,” so short bursts can fail even when the minute average is fine. It also describes acceleration limits that can return 429 errors on a sharp increase in usage, and advises ramping traffic gradually. A spend-cap 429 has no retry-after header, and retries keep failing until access resumes. Your retry code must tell these cases apart.

Worked example (hypothetical, prices and limits as of October 15, 2026). Say a task averages 8 model calls, with 6,000 input and 400 output tokens per call. That is 48,000 input and 3,200 output tokens per task. On Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens, one task costs $0.096 plus $0.032, or $0.128. A 200-case eval suite at 3 trials each is 600 runs, or about $76.80 per full run.

Now load. At an expected peak of 30 tasks per minute, uncached input is 1,440,000 tokens per minute. The Start tier ITPM limit for Sonnet 5.5 is 2,000,000, so you sit at 72%. Test at 2x peak and you need 2,880,000, which is over the limit, so you would see 429s. For most models, cache reads do not count toward ITPM. If 70% of input were served from cache, uncached input at 2x peak drops to 864,000, inside the limit (this ignores cache-write tokens, which do count).

Decision rule: load test at twice your expected peak, with bursts, on the tier you will actually run. If you pass only because of caching, assert the cache hit rate in the test. For monthly budgeting, see what an AI agent costs to run per month.

Shadow mode: real traffic, fake writes

Shadow mode runs the new agent on a copy of real requests while the current process keeps serving users. AWS describes the pattern in its SageMaker shadow tests: the shadow variant “receives a percentage of the incoming requests, replicated from the production variant, but does not return any responses.” SageMaker shadow tests default to 7 days, with a 30-day maximum. Those are limits of that managed feature, not a rule for how long shadow mode should run.

Agents need one change to that pattern. A model endpoint returns a prediction, but an agent calls tools. In shadow mode, every write tool must point at a stub that records the intended action and does nothing else. Read tools can hit real systems if your data rules allow it. Then compare the agent’s intended actions with what the human or current system actually did.

Decide sample size before you start. Worked example (hypothetical): your target is 90% agreement with human decisions. After 100 cases the agent agrees on 93. A simple normal-approximation 95% interval is about plus or minus 5 points, so the lower bound is about 88%, below target. After 400 cases at the same 93%, the interval is about plus or minus 2.5 points, and the lower bound is about 90.5%, above target. Fix the case count before you look, or you will stop when the number looks good.

Classify every disagreement. Wrong fact, wrong action, wrong tone, or the agent was right and the human was wrong. Any disagreement that would have moved money or exposed data is critical, and one blocks launch. New disagreement types become new scenario eval cases. That loop is the payoff: NIST notes that lab tests “may not extrapolate well” to real use, and shadow mode measures the gap.

Staged rollout: limit the blast radius of the first real writes

Shadow mode cannot test real writes and real users. A staged rollout can, at small scale. Google’s SRE workbook defines canarying as “a partial and time-limited deployment of a change in a service and its evaluation,” and says you must compare canary signals to control signals.

Two SRE rules carry over. Pick a few metrics that indicate real problems, “perhaps no more than a dozen.” And do not end a canary too early: “a handful of queries doesn’t provide a useful signal” when queries are diverse, which agent traffic always is.

A rollout plan I would use for a write-capable agent:

1. Internal users only. Real writes, small blast radius, people who will report problems.
2. 5% of eligible traffic, with human approval on every write above a set value.
3. 25%, approval only on high-value writes, once the canary matches control on error rate, escalation rate, cost per task, and correction rate.
4. 100%, with the shadow comparison kept running on a sample to catch drift.

Each stage needs a written rollback trigger and a rollback to the prior version, tested before stage 1. Also provide an emergency stop that does not require a deploy, such as a flag that disables write tools. See AI agent kill switch for stop levels. Buying instead? Ask vendors for evidence of each layer, using AI agent demo vs production.

Bottom Line

One good demo run proves almost nothing about an agent, because agents vary run to run and act on real systems.

Test in order: tools, scenarios over several trials, attacks, load and cost, shadow mode with stubbed writes, then a staged rollout with a tested rollback.

Set every pass threshold before you look at results. Start from the test plan table above.

FAQ

How many test cases do I need before launching an AI agent?

Anthropic’s guide calls 20 to 50 simple tasks drawn from real failures a great start for scenario evals. Shadow mode needs its own sample, sized in advance: in the worked example above, 100 cases could not confirm a 90% agreement target and 400 could.

What is the difference between pass@k and pass^k?

Pass@k is the chance that at least one of k attempts succeeds. Pass^k is the chance that all k succeed, which is closer to what a repeat user experiences. The tau-bench paper introduced pass^k in June 2024.

Can I still use OpenAI Evals to test my agent?

Not for long. OpenAI’s deprecations page says existing evals become read-only on October 31, 2026, and the dashboard and API shut down on November 30, 2026. It points to a migration guide for moving to Promptfoo.

Is there a standard for AI agent security testing?

The closest is the OWASP AI Testing Guide, version 1, released November 26, 2025. NIST AI 600-1 adds suggested actions such as regular adversarial testing. Both are voluntary guidance, so passing them is not a certification.

How long should shadow mode run?

Until you reach the case count you fixed in advance, and long enough to cover your traffic cycle, such as a month-end close. Agents with rare but costly actions may need weeks to see enough of them. If you use SageMaker shadow tests, the 30-day maximum is a limit of that product, so a longer run needs a second test or your own replay setup.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect