How to Choose an AI Agent Framework in 2026: A 2-Day Spike Scorecard

To choose an AI agent framework, name the hard part of your workflow first. Then test two or three candidates, plus plain code, in a 2-day spike. Score each on eight criteria: control flow, durability, tools and MCP, observability, language fit, model portability, license, and maintenance health. Pick the smallest option that passes.

The backward approach is to pick the framework with the most stars, then discover what the job needed. It is common, and it is expensive.

This post is the method. For a head-to-head of three popular frameworks, see LangChain vs LlamaIndex vs CrewAI.

What are the AI agent framework options in 2026?

The options fall into layers that stack. A provider SDK runs the agent loop. An orchestration framework adds explicit control flow and state. A durable workflow engine makes a run survive crashes and long waits. A typed framework adds validated outputs across model vendors. Plain code is always an option. I pulled every version below on October 15, 2026: the Python package from PyPI, except Mastra, from npm.

Two status changes matter. Microsoft shipped Agent Framework 1.0 on April 3, 2026, merging Semantic Kernel’s enterprise foundations with AutoGen’s orchestration patterns, and told users “Now is the time to migrate.” The AutoGen README now says AutoGen “is now in maintenance mode.” Semantic Kernel still ships (1.45.0 on October 6, 2026), but Microsoft’s advice is to migrate.

Stacking is normal. Temporal’s OpenAI Agents SDK integration runs ordinary Agents SDK code inside a durable workflow. So the real question is which layers this job needs.

Control flow, durability, and tools: the criteria that shape the design

These three decide your architecture. Get them wrong and you rewrite.

1. Control flow and state. Who decides the next step: your code or the model? A provider SDK lets the model drive a loop of tool calls and handoffs. An orchestration framework lets you write the path as a graph, with typed state at each node. If your process has fixed steps, branches, and approvals, put the path in code, where you can test it. For open-ended research, a model-driven loop is fine. Anthropic’s Building effective agents (December 2024) still holds: “start by using LLM APIs directly: many patterns can be implemented in a few lines of code.”

2. Durability and resume. A run that holds state in process memory loses it when the process dies. Anything that waits on a human, runs for minutes, or writes to a system of record needs state outside the process. Pydantic AI’s durable execution docs list Temporal, DBOS, Prefect, Restate, and AWS Lambda as co-maintained backends that wrap “every model request and tool call” as a durable unit. The same page warns that “durability is not storage”: surviving a crash is a different feature from saving a conversation. Read the edges. Temporal’s integration page says model calls run as Activities that “retry durably and are not repeated during Workflow replay,” but its durability “does not extend to” MCP servers, which run outside the workflow. The durable execution guide covers idempotency keys and retry rules.

3. Tools and MCP. The provider SDKs, Microsoft Agent Framework, Pydantic AI, and Mastra all document MCP support. The question is which version. The current MCP revision is 2026-07-28. It removed the initialize handshake and protocol-level sessions, so every request now carries its protocol version. It also deprecated the Roots, Sampling, and Logging features. A client that only speaks the older handshake-based revisions (2025-11-25 and earlier) can still talk to servers that keep supporting them, which the spec allows. But you carry that gap, so know who closes it. For when MCP beats a direct API call, see MCP vs API integration.

Observability, portability, license, and health: the criteria that decide cost later

These five never show up in a demo. They show up in month six.

4. Observability and eval hooks. You need every model call, tool call, and token count, in a store you control. Most agent bugs are invisible state, and without a trace you guess. Check defaults. The OpenAI Agents SDK sends traces to OpenAI’s backend by default, and “Tracing is unavailable for organizations that use OpenAI’s APIs under a Zero Data Retention (ZDR) policy.” You can add or replace trace processors, so it is fixable, but tell security before launch. Compare backends in AI agent observability tools.

5. Language and runtime fit. A framework your team cannot debug in its own language is a liability. CrewAI is Python only. Mastra is TypeScript. Runtime matters too: the Claude Agent SDK docs describe it as “a library that runs the Claude Code binary,” so your container has to ship and run that binary as a subprocess.

6. Model portability. Provider SDKs vary. The OpenAI Agents SDK docs cover non-OpenAI providers, with adapters for LiteLLM and Any-LLM. Google says ADK “can work with almost any generative AI model,” per adk.dev. Mastra’s README claims 40+ providers through one interface. The Claude Agent SDK is Claude Code as a library. In my reading, that makes it the wrong layer if swapping vendors is a hard requirement.

7. License and commercial lock-in. Read the terms, not the badge. The Claude Agent SDK docs say its use “is governed by Anthropic’s Commercial Terms of Service,” except components with their own LICENSE file. Yet the Python package lists MIT on PyPI. That mismatch is why legal reads the terms. Mastra’s core is Apache-2.0, but code in ee/ directories needs a paid license in production. And the bill rarely comes from the open-source library. It comes from the hosted tracing and deployment platform.

8. Maintenance health. A version policy tells you how often you will fix your code. LangChain’s release policy says breaking changes “will only occur in major version releases.” The OpenAI Agents SDK uses 0.Y.Z versioning and increases Y “for breaking changes to any public interfaces that are not marked as beta.” Per its PyPI history, it shipped 21 minor versions (0.4 through 0.24) between October 17, 2025 and October 15, 2026. Not every one broke something, but each one was allowed to. Pin exact versions and budget upgrade time. Pydantic AI went from 1.0 (September 2025) to 2.0 (June 2026) in under ten months.

The 2-day spike scorecard: eight pass/fail tests

Build one reference workflow in each candidate, and always include plain code with your vendor’s client SDK. The workflow: read an input, call one real MCP tool, extract fields, write one record, pause for human approval, send a notification. Use your real model, CI, and deployment target. Day 1 covers tests 1 to 4. Day 2 covers tests 5 to 8.

My scoring rules. Tests 2, 5, and 7 are gates: one fail eliminates the candidate. (Skip test 2 if no run waits on a human or outlives one request.) Then count passes on tests 1, 3, 4, 6, and 8. Highest count wins. Ties go to fewer dependencies. If no framework beats plain code by at least two passes, use plain code.

Worked example (hypothetical). A Python team spikes an invoice approval agent. Plain code clears the gates with a small Postgres state table, then passes control flow and health but fails MCP, observability, and portability: 2 of 5. Framework A clears the gates and scores 5 of 5. Framework B scores 5 of 5 but fails the license gate: the approval UI it needs is enterprise only. A beats plain code by three passes, so A wins. At 3 of 5, A would win by one pass, and I would ship plain code.

Which AI agent framework should you use? A decision table by situation

Find the closest row. Where two apply, durability and audit win over convenience. These are my defaults, not rankings.

Two warnings. Multi-agent is a design choice, not a framework feature. If one agent with good tools does the job, extra agents add coordination to debug (when multi-agent pays). And if nobody on the team writes code, read n8n vs code for AI agents.

Picking a framework too early: how to detect it

The failure mode: you choose before you know the hard part, so you choose by popularity. Then the framework’s shape becomes your design. A role-based crew runs a fixed approval process. A graph runs a straight line. The cost shows up as debug time and upgrade churn, not as an outage. Anthropic’s warning names the mechanism: frameworks “often create extra layers of abstraction that can obscure the underlying prompts and responses, making them harder to debug.”

Five signals that you picked too early. Check them 30 days after adoption.

1. You cannot say the hard part in one sentence. “Approvals that survive restarts” is a hard part. “We wanted to be agentic” is not.
2. Your graph is a straight line. No branches or loops means you bought a state machine for a script.
3. You read framework source to see the prompt. If finding the exact text sent to the model takes over five minutes, the abstraction costs more than it saves.
4. You patch internals. Monkeypatching private classes in the first month means the framework fights your design.
5. You pin to avoid upgrades. A team afraid of the next minor version falls behind on security fixes.

Two or more signals means re-run the spike with plain code as a candidate. The defense is the same either way. Keep domain logic out of the framework: tools as plain functions, prompts in versioned files, eval cases in your own repo, run by your own CI. Agent evals in CI shows the setup. Then a framework swap is adapter work, not a rewrite.

Bottom Line

Knowing the hard part of your job matters more than the framework.

Name it. Run the 2-day spike with plain code as a candidate. Treat durability, runtime fit, and license as gates. Adopt a framework only if it beats plain code by two passes.

Keep tools, prompts, and evals outside it, so the choice stays reversible.

FAQ

Is AutoGen still maintained?

Only for fixes. Its README says contributions are limited to bug fixes, security patches, and documentation, and that the project is community managed. The last autogen-agentchat release on PyPI was 0.7.5, on September 30, 2025. New users are pointed to Agent Framework.

Is the OpenAI Agents SDK only for OpenAI models?

No, but swapping the model does not move the tracing. Traces still go to OpenAI’s backend by default unless you disable tracing or replace the processors. Built-in tracing is unavailable to Zero Data Retention organizations, so my advice for those teams is to set their own trace processor on day one.

Can I build a customer-facing product on the Claude Agent SDK?

Yes, under Anthropic’s Commercial Terms of Service. Use API key authentication: third-party products may not offer claude.ai login or rate limits without Anthropic’s approval. Branding rules allow “Powered by Claude” but not calling your product “Claude Code.”

Which MCP version should my framework support?

The current revision is 2026-07-28. The spec lets clients and servers support several versions at once, and deprecated features stay for at least twelve months (ninety days under an expedited exception). In my view, a client still on 2025-11-25 is a gap to track, not an emergency.

Does Microsoft Agent Framework 1.0 cover everything it ships?

No. The 1.0 post promises backward compatibility for agents, middleware, memory, workflows, and orchestration patterns. DevUI, Foundry hosted agents, and Skills are in preview, and those APIs “may evolve” before reaching stable.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect