Agent Observability: Tracing, Debugging, and Monitoring in Production

An agent that took fourteen steps and cost $0.80 instead of two steps and $0.02 is failing in a specific way — and without span-level tracing you'll never see it. OpenTelemetry, LangSmith, Langfuse, and Arize Phoenix give you X-ray vision into every LLM call, every tool call, every retry. Instrument from day one; retroactive tracing helps no one at 3 AM.

Learning Objectives

After this lesson, you will be able to:

Add OpenTelemetry tracing to your agents so every LLM call and tool call is captured in a visual tree you can inspect
Read a LangSmith trace to diagnose exactly where an agent failed -- which tool call went wrong, which argument was bad, how many tokens were wasted
Build a monitoring dashboard that tracks cost and speed for every agent run, so cost surprises never happen again
Thread a trace ID through multi-agent handoffs so you can follow one task across multiple agents in your logs
Set up token budget alerts that warn you (or auto-kill the agent) before a runaway loop burns through your API credits

Prerequisites

Before You Start

Review

Observability is your X-ray vision into what your agent is actually doing. Without it, your agent is a black box -- you send in a question and get back an answer, with no idea what happened in between. This lesson gives you the tools to see everything.

#The Crash That Woke No One Up

from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor import anthropic import json # Set up the tracer provider = TracerProvider() provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter())) trace.set_tracer_provider(provider) tracer = trace.get_tracer("my-agent") client = anthropic.Anthropic() def run_agent(user_message: str, tools: list) -> str: """Run an agent loop with full OTel instrumentation.""" with tracer.start_as_current_span("agent.run") as root_span: root_span.set_attribute("agent.user_message", user_message) root_span.set_attribute("agent.tool_count", len(tools)) messages = [{"role": "user", "content": user_message}] total_input_tokens = 0 total_output_tokens = 0 step = 0 while True: step += 1 # Span for each LLM call with tracer.start_as_current_span(f"llm.call.step_{step}") as llm_span: response = client.messages.create( model="claude-opus-4-5", max_tokens=1024, tools=tools, messages=messages ) # Capture token metadata llm_span.set_attribute("llm.input_tokens", response.usage.input_tokens) llm_span.set_attribute("llm.output_tokens", response.usage.output_tokens) llm_span.set_attribute("llm.stop_reason", response.stop_reason) llm_span.set_attribute("llm.step", step) total_input_tokens += response.usage.input_tokens total_output_tokens += response.usage.output_tokens if response.stop_reason == "end_turn": root_span.set_attribute("agent.total_input_tokens", total_input_tokens) root_span.set_attribute("agent.total_output_tokens", total_output_tokens) root_span.set_attribute("agent.steps", step) # Estimate cost (Claude claude-opus-4-5 pricing as example) cost = (total_input_tokens * 0.000015) + (total_output_tokens * 0.000075) root_span.set_attribute("agent.estimated_cost_usd", round(cost, 6)) return response.content[0].text # Process tool calls messages.append({"role": "assistant", "content": response.content}) tool_results = [] for content_block in response.content: if content_block.type == "tool_use": with tracer.start_as_current_span(f"tool.{content_block.name}") as tool_span: tool_span.set_attribute("tool.name", content_block.name) tool_span.set_attribute("tool.input", json.dumps(content_block.input)) try: result = execute_tool(content_block.name, content_block.input) tool_span.set_attribute("tool.result_length", len(str(result))) tool_span.set_attribute("tool.status", "success") except Exception as e: tool_span.set_attribute("tool.status", "error") tool_span.set_attribute("tool.error", str(e)) tool_span.add_event("tool.error.occurred") result = {"error": type(e).__name__, "message": str(e)} tool_results.append({ "type": "tool_result", "tool_use_id": content_block.id, "content": json.dumps(result) }) messages.append({"role": "user", "content": tool_results})

Dimension	Why It Matters
Per tool call	Which tool is the most expensive to call? (Often: the LLM calls that process large tool results)
Per agent type	Is the research agent 5× more expensive than the execution agent?
Per user session	Which users generate the most cost? (Rate limiting candidates)
Per task category	Is "flight booking" 10× more expensive than "FAQ lookup"? (Pricing decisions)

Metric	What to Measure	Alert Threshold
Success rate	% of runs that complete without error	Alert if drops below 95%
P95 latency	95th percentile end-to-end run time	Alert if exceeds SLO (e.g., 30s)
Cost per run	Average tokens × price across last 100 runs	Alert if spikes 3× baseline
Token budget hit rate	% of runs that hit the token budget limit	Alert if above 5% (tuning signal)
Retry rate	Average retry events per run	Alert if above 0.5 (tool schema quality signal)
Context overflow rate	% of runs where context exceeded 80% of max	Alert if above 10% (chunking needed)
Tool error rate	% of tool calls that return errors	Alert if any tool exceeds 5% error rate

Agent Observability: Tracing, Debugging, and Monitoring in Production

#The Crash That Woke No One Up

Your Reflection

Continue Learning

Discussion

#The Observability Stack for Agents

#Traces: The Execution Tree

#Spans: Individual Operations

#Metadata: The Numbers That Matter

#Events: Notable Moments

#OpenTelemetry for Agents

#Manual Instrumentation

#Auto-Instrumentation

#LangSmith and LangFuse: Trace UIs

#What a Trace Looks Like in the UI

#Adding Evaluators to Traces

#Filtering Traces

#Debugging a Real Agent Failure

#The Run That Failed

#Step 1 — LLM Decides to Search

#Step 2 — Wrong Argument Name on Book

#Step 3 — Retry Spiral Begins

#Step 4 — Context Overflow

#The Fix

#Distributed Tracing for Multi-Agent Systems

#Propagating Trace Context

#The trace_id: Your Single Key

#Cost Monitoring

#Per-Run Cost Estimation

#Token Budget Alerts

#Cost Attribution

#Latency SLOs for Agents

#Measuring Agent Latency

#Slow Tool Identification

#Production Dashboard Pattern

#Putting It All Together

#Key Takeaways

#Quick Check

Deep Dive