The One-Week AI Pilot: A Day-by-Day Method to Ship Fast

A one-week AI pilot is five working days in which an agent runs on real cases from one narrow workflow, in shadow mode, and you measure three numbers: accuracy, critical errors, and cost per case. It only fits in a week if data access, a named owner, a baseline, and the scope are settled before day 1.

Five days is this method’s target once the readiness work is done. Below: the pre-day-1 checklist, the daily plan, what “done” means, and what a week cannot prove.

Why do AI pilots usually take months?

MIT’s Project NANDA, in The GenAI Divide: State of AI in Business 2025, says top-performing mid-market companies reported average timelines of 90 days from pilot to full implementation. Enterprises, which it defines as firms with over $100 million in annual revenue, took nine months or longer. The research ran January to June 2025 and drew on interviews, a survey of 153 senior leaders, and a review of over 300 public AI initiatives. The authors call their figures directionally accurate, based on interviews rather than official company reporting.

Gartner’s survey of 644 respondents in the U.S., Germany, and the U.K., fielded in late 2023, found that on average only 48% of AI projects make it into production, and it takes 8 months to go from AI prototype to production. Deloitte’s 2026 State of AI in the Enterprise surveyed 3,235 leaders in 24 countries in August and September 2025. Only 25% had moved 40% or more of their AI pilots into production.

None of these sources splits the months into build time and waiting time, so what follows is my read, not a measured fact. I think most of the calendar goes to access requests, a security review that starts from zero, an owner with no hours set aside, and a metric still argued in week six. In my view, building a narrow workflow takes days.

Decision rule: a one-week pilot does not delete the slow work. It moves it before day 1. If the readiness checklist below takes more than two weeks to clear, that delay is your most useful finding.

What has to be true before day 1

Every check is pass or fail. Clear them the week before. If a blocker fails on the Friday before, move the start date.

Check 6 follows OWASP’s guidance on Excessive Agency. Its prevention steps include limiting permissions to the minimum necessary. Read plus draft is the minimum that still produces something to measure.

For a fuller audit, use the 12-check AI agent readiness checklist. For the one-page charter behind check 9, use the scale-or-stop success criteria.

The one-week AI pilot methodology, day by day

Each day has one job, a deliverable, and an exit test. Fail the test and you fix it that day or stop.

Day 1 is plumbing. Most surprises live here: a field blank in many records, a tight API rate limit, a PDF that is a scanned image. Those are findings, not delays.

Day 2 is where the pilot gets honest. The locked set exists because you tune against whatever you read. Fix prompts while staring at the scoring cases and your score measures your memory, not the agent. Anthropic’s January 2026 guide to agent evals says 20 to 50 simple tasks drawn from real failures is a great start. It also says you will not know whether your graders work unless you read the transcripts and grades from many trials. Reading only failures misses wrong outputs graded as passes. So on day 2, review every failure and a sample of passes against a human’s judgment. Break-it tests are malformed inputs, missing fields, and an email or document that tells the agent to do something it should not.

Day 3 is shadow mode. AWS describes shadow testing as routing a copy of live requests to the new variant while only the production variant’s responses go back to the caller. For an agent pilot, the “production variant” is your people. The agent’s answers go to a log nobody outside the pilot sees.

Day 4 produces the number most pilots skip: how long it takes a person to judge the agent’s output. That number, not model price, usually decides the economics.

Day 5 is arithmetic and a decision, covered next. For a worked mailbox version of this plan, see the one-week email triage pilot.

What “done” means on day 5

Done is a one-page result with three numbers, each with its uncertainty, and a signed decision.

Number 1: accuracy, with an interval. NIST’s engineering statistics handbook recommends the Wilson interval for proportions over the common normal approximation. Say the agent gets 135 of 150 locked cases right. That is 90%, and the 95% Wilson interval is about 84.2% to 93.8%. If your floor was 85%, the point estimate clears it and the lower bound does not. These intervals, and the bound below, assume the cases are independent and drawn from the workflow you will run. A test set estimates production performance only as well as it represents production traffic.

Decision rule: compare the lower bound to the floor. Lower bound above the floor: proceed. Point estimate above but lower bound below: iterate for one more week and add cases. Point estimate below the floor: narrow the scope or stop.

Number 2: critical errors, with an upper bound. Define critical before day 1 (a wrong payee, a missed legal hold, a reply that promises a refund). If you see zero, you still do not know the rate is zero. The rule of three from Hanley and Lippman-Hand (JAMA, 1983) says that with zero events in n cases, the upper 95% limit on the true rate is about 3/n. Zero critical errors in 150 cases means the rate could still be as high as 2%. If 2% is too high, a week cannot clear this bar. The next step needs more volume or a human gate on that error type.

Number 3: cost per case, including people. Hypothetical: each case uses 15,000 input tokens and 1,200 output tokens on Claude Sonnet 5.5, which Anthropic’s pricing page lists at $2 per million input tokens and $10 per million output tokens. Model cost is $0.030 plus $0.012, or $0.042 per case. Today a person spends 9 minutes per case at a $48 loaded hourly rate, which is $7.20. In the projected setup, the agent drafts 80% of cases and a reviewer spends 2 minutes on each. The other 20% escalate and take the full 9 minutes. That is 3.4 minutes of people time per case, or $2.72, plus $0.042 of model cost: $2.76, a 62% saving.

Now stress it. If review takes 4 minutes, cost rises to $4.04 and the saving drops to 44%. If escalations hit 40% at 2-minute reviews, cost is $3.88 and the saving is 46%. In every version, the model is under 2% of the cost. That is why day 4 times the reviewers.

What a one-week pilot cannot prove

A week produces evidence, not a production system. Here are five limits and what to do about each.

Rare failures. As shown above, 150 clean cases still allow a 2% critical error rate. To get the upper bound to 1%, you need about 300 clean cases. Low-volume workflows need weeks of shadow mode for that, not days.

Calendar effects. One week misses month-end close, quarter-end spikes, open enrollment, and holiday backlogs. A cyclical workflow got one sample point. Say which.

Learning and drift. In MIT NANDA’s buyer interviews, one user said a tool was useful the first week, “but then it just repeats the same mistakes.” A week cannot show whether your feedback loop works. Plan a weekly review of the error table for the first 90 days.

Model change. Anthropic’s deprecation page commits to at least 60 days’ notice before retiring publicly released models. On September 30, 2026, it notified Claude Sonnet 4.5 users of retirement on November 30, 2026. Keep the 150 locked cases as your regression test for every model swap.

Adoption and operations. Shadow mode never touches the humans’ real work, so it cannot show whether they will trust the drafts, or who answers the page at 2 a.m. See PoC vs pilot vs production for stage exit tests and the experimentation-to-production playbook for identity, logging, and release gates.

Where one-week pilots break, and how to spot it early

Each of these has a signal you can see before Friday.

Access arrives on day 3. The signal is check 3 passing on a promise (“IT says Tuesday”). The fix is the rule in the checklist: a pulled sample, using the agent’s own access, or no start.

Scope grows mid-week. The classic line is “can it also handle X?” on Wednesday. Each addition resets the error table and spends locked cases. Write additions on a list for the decision meeting. If the owner insists, the pilot restarts on Monday with the new scope.

The labels disagree. If two reviewers would score the same case differently, your accuracy number measures their disagreement. Before day 1, have two people label the same 30 cases. If they disagree on more than about 1 in 10 (my rough threshold, not a standard), fix the label definitions first. This matters more with a model as grader. Anthropic’s eval guide says LLM-as-judge graders should be closely calibrated with human experts. My post on calibrating LLM-as-a-judge shows how.

The owner disappears. The signal is unreviewed shadow output piling up on day 4. No review means no live accuracy and no economics. Reschedule instead of reporting a half-measured result.

The demo replaces the numbers. The signal is a day 5 agenda that opens with three hand-picked cases. Lead with the one-page result and the worst failures.

Stopping gets treated as failure. The signal is a request to “give it another week” with no new cases or fixes planned. IDC research with Lenovo, reported by CIO, found that for every 33 AI proofs of concept a company launched, only four graduated to production. Its authors cited low organizational readiness in data, processes, and IT infrastructure. A “stop” on day 5 cost five days, not five months. For the longer list of causes, see why AI pilots fail.

Bottom Line

My view: a one-week AI pilot is fast because the waiting happens first.

Clear the nine checks before Monday. Run five days with one exit test each. On Friday, report accuracy with an interval, critical errors with an upper bound, and cost per case including review time.

Say plainly what the week could not prove. Next step: run the checklist on your best candidate workflow and count the blockers.

FAQ

Is a one-week AI pilot the same as a proof of concept?

No. A proof of concept usually tests whether the agent can do the task on a sample. This pilot also runs in shadow mode on live cases and measures review time and cost per case, so it answers whether the task is worth doing.

How many test cases does a one-week AI pilot need?

More cases buy a tighter number. At 90% observed accuracy, 150 scored cases give a 95% interval about 10 points wide. 400 cases narrow it to about 86.7% to 92.6%.

What does a one-week AI pilot cost to run?

The model bill is usually small. In the worked example, 1,000 cases cost about $42 in model fees. The real cost is people: the owner’s hours, the reviewers’ time on day 4, and the engineer building the path.

Can the agent take real actions during pilot week?

Not in this method. Writes to a system of record need a longer controlled release with approvals, logging, and rollback. OWASP’s guidance is to require a human to approve high-impact actions before they are taken. Move from drafts to approved actions only after shadow results clear the floor.

What happens after a successful pilot week?

You move to a controlled release on a small share of live work, with a human approving actions. Deloitte’s 2026 survey found that close to three-quarters of companies plan to deploy agentic AI within two years, but only 21% of those report a mature model for agent governance. Settle ownership, identity, and logging before you widen access.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect