AI Agent Observability: What to Log and Which Tool to Pick

AI agent observability means recording every agent run as one trace: each model call, tool call, and decision, with tokens, cost, latency, errors, and a quality score attached. You need it to explain any single run and to catch drift across thousands of runs. LLM observability tools such as Langfuse, LangSmith, Arize Phoenix, Braintrust, and Datadog store and query those traces.

Below: a per-run log schema mapped to OpenTelemetry names, and a tool selection table by requirement. Prices come from each vendor’s site as of October 15, 2026.

Why APM is not enough for AI agent observability

An agent can fail while every HTTP call returns 200. The model returns well-formed text that is wrong. A tool call succeeds with the wrong arguments. A loop that should take 4 steps takes 30. Your APM sees status codes and latency. It does not see which prompt version ran, which documents the agent read, or what the model decided at each step.

A run is a tree: a root span for the agent invocation, child spans for each model call and tool call, and attributes on each span. That tree is what agent observability tools are built to store.

Tracing is now common. LangChain’s State of Agent Engineering survey (1,340 responses, fielded November 18 to December 2, 2025) found 89% have some agent observability and 62% have step-level tracing. Among teams with agents in production, those numbers were 94% and 71.5%. Evaluation lags: 52.4% run offline evals and 37.3% run online evals. Quality was the top barrier to production, cited by 32%. LangChain sells observability to this audience, so treat these as directional.

My read: collecting traces is solved. Turning traces into quality and cost signals is not.

Decision rule: if you cannot answer “which agent version, prompt version, and model produced run X, what did it cost, and was it right?” in under five minutes, you do not have observability yet. The five minutes is my bar, not a standard.

What to log per run: a schema mapped to OpenTelemetry

This schema is for engineers debugging quality, latency, and cost. The audit trail post covers what an auditor needs: approvals, retention, and access. The fields overlap, so emit both from the same instrumentation.

The attribute names come from the OpenTelemetry GenAI attribute registry. Every gen_ai.* name below has Development status, so names can still change. Rows marked “custom” have no standard attribute today.

Three notes on using it.

Content is opt-in for a reason. The GenAI span conventions say instrumentations “SHOULD NOT capture” prompts and outputs by default, and recommend storing production content externally with references on spans, under separate access controls. My recommendation: metadata on every run, full content on a sample plus every failed or escalated run.

Compute cost at ingestion. There is no standard cost attribute. Multiply tokens by your price table when the span arrives and store the price table version. Otherwise a price change rewrites last quarter’s numbers.

Alert on truncation. A rising share of length in gen_ai.response.finish_reasons means outputs are being cut off. That failure looks like a quality drop, not an error.

Where the OpenTelemetry GenAI conventions stand in October 2026

On June 12, 2026, semantic conventions v1.42.0 deprecated all gen_ai.* attributes, metrics, events, and spans in the core repository and moved them to a dedicated semantic-conventions-genai repository. The span documentation there still carries Development status, so names can still change.

Vendors already build on it. Datadog’s OpenTelemetry instrumentation guide supports GenAI conventions from version 1.37 and also accepts Arize’s OpenInference conventions. If a framework previously supported a pre-1.37 version, Datadog says to set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Anthropic’s Claude API tracing uses GenAI attributes and sends to any OTLP endpoint, with Langfuse, LangSmith, Datadog, Arize, Braintrust, and others listed as partners.

Decision rule: instrument with OpenTelemetry anyway, because it keeps your switch cost low. But pin your instrumentation library versions, and treat every upgrade as a schema migration.

Failure mode: a renamed attribute throws no error. It empties a dashboard panel. Detect it with a daily count of model-call spans missing gen_ai.usage.input_tokens; a jump after a deploy means a name changed.

LLM observability tools in 2026: pricing, hosting, and ownership

List prices as published on October 15, 2026. “Units,” “traces,” “spans,” and “GB” are different billing meters, so do not compare the headline numbers directly.

One detail the table hides: Datadog bills only LLM spans, meaning single requests to an LLM provider, so one run produces several.

Ownership changed for three of these in 2026. ClickHouse acquired Langfuse on January 16, and Langfuse says “there are no planned changes to licensing.” Mintlify acquired Helicone on March 3; the founders promised that “security updates, new models, bug & performance fixes all keep shipping.” My read: that is maintenance language, not a roadmap. ServiceNow acquired Traceloop in March, and its CEO wrote that “OpenLLMetry will remain open-source.”

Decision rule: if your instrumentation is a vendor SDK only, price the migration before you sign a multi-year contract.

Provider-native tracing: free, but check where traces go

Model providers ship their own tracing. It costs nothing extra, but the defaults differ.

OpenAI Agents SDK. Tracing is on by default and sends traces to OpenAI’s backend. It is unavailable for organizations under a Zero Data Retention policy. trace_include_sensitive_data defaults to True, so model inputs and outputs are captured unless you turn it off. Disable it with OPENAI_AGENTS_DISABLE_TRACING=1, or route spans to external processors such as Langfuse, LangSmith, or Datadog.

Claude API. OpenTelemetry export is in beta. The default content mode is metadata only: identifiers, timing, status, and token counts. Full content is opt-in. With zero data retention you can still export full content; for Messages API requests, Anthropic holds it in memory only until delivery. Organizations with a HIPAA, FERPA, or K-12 restriction export metadata only from the API.

Claude Agent SDK. Telemetry is off until you set CLAUDE_CODE_ENABLE_TELEMETRY=1, traces are beta, and prompt and tool content stay out unless you opt in.

Amazon Bedrock AgentCore. Telemetry lands in your own CloudWatch account at CloudWatch rates. In AWS’s own example, 200,000 invocations produce about 10 million spans (10 GB) and cost $6.50 a month to ingest with event logs, before metrics and standard logs.

Decision rule: if you use one provider, its native tracing plus an OTLP export is a fine start. If you use two or more, make an OpenTelemetry backend your system of record and treat provider traces as one input. For retention terms by provider, see the zero data retention guide.

Which LLM observability tool fits your requirement?

Start from the constraint you cannot change.

My recommendation for a mid-size company starting now: instrument with OpenTelemetry GenAI conventions, send traces to Langfuse Cloud or a self-hosted Langfuse, and add Datadog only if it already runs your APM. Revisit after 90 days, when you know your real calls per run.

If data location is the deciding factor, the data sovereignty guide covers what to ask beyond the region name.

What does LLM observability cost? A worked example

Hypothetical: a support-triage agent handles 50,000 runs a month. Each run has 1 root span, 4 model calls, and 3 tool calls, so 8 spans. Each run gets 1 online eval score.

Langfuse Cloud. A billable unit covers traces, observations, and scores. Per run that is 1 trace + 8 observations + 1 score = 10 units, or 500,000 units a month. Core: $29 + (400,000 / 100,000) × $8 = $61 a month. Pro: $199 + $32 = $231 a month.

Datadog. Only model calls count as LLM spans: 4 × 50,000 = 200,000. With annual billing: $160 + 10 × $3.50 = $195 a month. Month-to-month: $200 + 10 × $4.20 = $242. On-demand: $240 + 10 × $5 = $290.

Double the volume to 100,000 runs. Langfuse Core: 1,000,000 units, so $29 + 9 × $8 = $101. Datadog annual: 400,000 LLM spans, so $160 + 30 × $3.50 = $265. Langfuse stays inside its first overage tier, which runs to 1 million units.

LangSmith Plus, 1 seat. Each run is one trace: 50,000 traces, 10,000 included. $39 + 40,000 × $0.005 = $239 a month. At 100,000 runs: $39 + 90,000 × $0.005 = $489. That buys base traces with 14-day retention.

Arize AX lists no overage rate, and Braintrust meters bytes, which depend on content capture, so neither is priced here.

Each tool meters something different. A tool that is cheap at 4 model calls per run can be expensive at 40. Price candidates on your calls per run and content setting, not on run count.

Failure modes that make traces lie

Double-counted tokens. Anthropic’s counting guidance warns that SDK and API spans share the name anthropic.messages.create, and that gen_ai.usage.input_tokens already includes cached tokens. Sum one source per request. Sum both and you count each request twice.

Lost spans. The Claude Agent SDK exports traces every 5 seconds by default, a killed process loses its buffer, and export errors are silent by default. Anthropic also calls API span delivery best effort. Compare runs in your application database with root spans daily, and reconcile spend against the provider’s usage API, not your traces.

Content you did not mean to keep. With OpenAI’s default of trace_include_sensitive_data=True, prompts and outputs go into traces. Check this before your first production deploy.

Retention shorter than detection. Datadog defaults to 15-day trace retention and LangSmith base traces last 14 days. If a bad answer surfaces three weeks later, the trace is gone. Set retention longer than your detection lag plus investigation time. The incident response runbook covers that lag.

Traces also feed two loops: failed runs become test cases for evals in CI, and outcome fields roll up into the three weekly KPIs.

Bottom Line

AI agent observability is one trace per run, with versions, tokens, cost, outcome, and a quality score on it.

Instrument with OpenTelemetry GenAI conventions, knowing they are still in Development. Pick the backend by your hardest constraint: data location, existing APM, or eval workflow. Price it on your calls per run, and check retention against how late you find problems.

FAQ

What is the difference between LLM observability and AI agent observability?

LLM observability traces single model calls. Agent observability adds the steps between them: tool calls, retrieval, planning, and sub-agents. The OpenTelemetry GenAI registry gives these their own operation names, such as invoke_agent, execute_tool, and plan.

Is Langfuse still open source after the ClickHouse acquisition?

Yes, as of October 2026. The repository is MIT licensed except the ee folders, and Langfuse said after the January 2026 deal that self-hosting stays “a first-class path.” A few add-on features need a license key, such as Organization Creators, the Instance Management API, and UI customization.

Can I use OpenTelemetry instead of a vendor SDK?

For traces, yes. The Claude Agent SDK, for example, injects W3C TRACEPARENT into its subprocess, so agent spans nest under your application’s own trace instead of starting a new one. Vendor SDKs still add extras such as prompt management and datasets. My recommendation: OpenTelemetry for traces, a vendor SDK only for those extras.

Does agent tracing work with zero data retention?

OpenAI’s Agents SDK tracing is unavailable for organizations under a Zero Data Retention policy. Anthropic allows full-content export under zero data retention for Messages API requests, but Claude Managed Agents is not eligible for zero data retention.

How long should I keep agent traces?

Plans set the ceiling. Langfuse gives 30 days of data access on Hobby, 90 on Core, and 3 years on Pro; automatic deletion is a separate opt-in setting on Pro. Arize AX keeps 15 days on Free and 30 on Pro. Datadog sells 15 to 90 days. I would keep full traces at least 30 days beyond your slowest detection path, and keep audit records on the audit schedule.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect