Function Calling: How Tool Use Works Under the Hood
"Function calling" isn't a special mode — the LLM just generates JSON tokens, and your code parses them and runs the function. When you understand that, three things stop being mysterious: why arguments hallucinate, why parallel tool calls double your cost, and why Pydantic at the boundary saves your weekend.
Learning Objectives
After this lesson, you will be able to:
Understand that a tool call is just the AI generating JSON text token by token -- there is no magic 'function calling mode' -- and why this matters for debugging
Handle tricky situations: parallel tool calls (multiple tools at once), partial outputs, and malformed arguments that break your JSON parser
Know the differences between Anthropic and OpenAI tool call formats so you can work with both
Add Pydantic validation to catch bad tool arguments before execution, returning clear error messages the AI can learn from
Apply tool result truncation strategies so large tool outputs do not blow up your context window
Lesson scope note: This lesson covers the under-the-hood mechanics — constrained decoding, parallel tool execution with asyncio, and Pydantic validation. For the conceptual 7-step flow and tool schema design, see Tool Use & Function Calling (Lesson 2).
This lesson pulls back the curtain on how tool calls actually work under the hood. It is more technical than the previous tool-use lesson, but understanding these internals is what separates developers who can debug agent failures from those who just stare at error logs. You will feel empowered after this one.
When developers first learn function calling, they picture it as magic: the model "knows" to call a function. But there is no magic. The model generates token sequences. Some token sequences happen to look like function calls because the model was trained on millions of examples of function calls. The provider then parses those tokens into a structured object your code can execute.
Once you understand the generation mechanics, debugging becomes simple. "JSONDecodeError"? The model generated malformed JSON tokens. "Missing required field"? The model assigned near-zero probability to the field name token. "Wrong argument type"? The model saw similar training examples with a string where your schema requires an integer.
The LLM does not have a special "function calling mode." It generates tokens. The structured tool call output is just a particular shape of token sequence that the provider has trained the model to produce when it decides to use a tool.
Here is what happens at inference time:
Step 1: The tool schema enters the context. Before the model generates any tokens, the tool definitions are injected into the context window — either in the system prompt or as a special format the API handles automatically. The model reads these definitions the same way it reads any other text.
Step 2: The model generates a response. For most messages, the model generates natural language. For messages that require a tool, the model generates a structured response instead. Providers like Anthropic use special token sequences to signal the start of a tool use block.
Step 3: Grammar constraints apply. This is the key mechanism. When the model starts generating a tool call argument, the inference server restricts the token vocabulary at each step to only tokens that maintain valid JSON. This is called constrained decoding or grammar-constrained sampling.
Step 4: Your code parses the output. The provider's API parses the structured token sequence into a usable object: tool name (string), tool use ID (string), and arguments (parsed JSON). You receive this as a structured response, not raw text.
The user asks: "What is the weather in Tokyo?" The API call includes the tool schema for get_weather(location: string, units: string). The model receives: the conversation history, the system prompt, and the tool definitions — all as one combined context.
At the first output token, the model has two paths: generate natural language ("The weather in Tokyo is...") or generate a tool use signal. Because the model has been fine-tuned on examples of tool use, it assigns high probability to the tool use signal tokens when the question requires real-time data it does not have.
On Anthropic, the model generates a response with stop_reason: "tool_use" and a content block of type: "tool_use". The content block contains: id (a unique identifier), name (the tool name), and input (the arguments as JSON). These come back as structured objects, not raw text — the provider parses them before returning the API response.
For the input field, the model generates tokens one by one: {, ", l, o, c, a, t, i, o, n, ", :, , ", T, o, k, , , , . The grammar constraint at each step restricts the vocabulary: after , only or are valid JSON; after , only is valid; after , only , , or are valid. This is why JSON arguments are usually well-formed even without explicit JSON mode.
The API returns the tool use block. Your code extracts the name (get_weather) and input ({"location": "Tokyo"}). Your code — not the model — actually calls the weather API. The model has stopped generating. It is waiting. Your code fetches the weather data and sends it back as a tool result in the next message.
With the tool result in context, the model generates a final natural language response: "The current weather in Tokyo is 18°C and partly cloudy, with a chance of rain in the afternoon." The tool call never appears in the user-facing response — only the synthesized answer does.
Normal LLM sampling works by assigning probabilities across the entire vocabulary (~100,000 tokens) and sampling from that distribution. Constrained decoding restricts this vocabulary at each step based on the current parse state.
At each token position, a parser tracks the current JSON parse state. The state determines which tokens are valid next:
After {: valid next tokens are " (start a key) or } (close object)
After "key": : valid next tokens are " (string value), [ (array), { (object), or digits (number)
After "key": "val": valid next tokens are , (another field) or } (close object)
Tokens outside these valid sets are assigned probability zero before sampling. The model cannot generate invalid JSON — the probability mass is redistributed across the valid tokens at each step.
Token step 14:
Current parse state: inside object, after key "location", expecting value
Valid next tokens: " (string start), [ (array start), { (object start), 0-9 (number start)
Model's raw probability distribution:
"T" -> 0.42 (part of "Tokyo")
"S" -> 0.18 (could be "San Francisco")
"N" -> 0.11 (could be "New York")
...
"]" -> 0.003 (invalid here — zeroed out)
"}" -> 0.002 (invalid here — zeroed out)
Wait — "T" is not in the valid set either (expecting the opening quote first).
After zeroing invalid tokens and renormalizing, the only options starting here
are the quote character. The model generates: "
Try it! Send a tool call request to the Anthropic API with a tool that has a required "city" parameter of type "string" and an optional "units" parameter of type "string" (enum: ["celsius", "fahrenheit"]). Ask "What is the weather?" without specifying a city. Watch how the model still generates valid JSON but might guess a city. That is constrained decoding in action -- valid structure, potentially wrong content.
#Why the Model Can Still Hallucinate Within Valid JSON
Constrained decoding ensures valid JSON structure. It does not ensure correct values. The model can generate:
json
{
"location": "Tokio", // Misspelled city — valid JSON, wrong value
"units": "kelvin", // Not in the enum — valid JSON, unsupported value
"date": "yesterday" // Not a valid date format — valid JSON, wrong format
}
Grammar constraints enforce syntax. Your Pydantic validation enforces semantics.
#The Outlines Library for Custom Constrained Generation
For open-source models and custom deployments, the outlines library implements constrained decoding with arbitrary JSON schemas:
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import outlines
from pydantic import BaseModel
from typing import Literal
class WeatherArgs(BaseModel):
location: str
units: Literal["fahrenheit", "celsius"] = "celsius"
model = outlines.models.transformers("mistralai/Mistral-7B-v0.1")
generator = outlines.generate.json(model, WeatherArgs)
# The generator will ONLY produce output that parses to WeatherArgs
result = generator("What is the weather in Tokyo?")
# result is guaranteed to be a valid WeatherArgs instance
The two dominant providers use different formats for the same concept. This matters when switching providers or using a library like LiteLLM that abstracts both.
LiteLLM normalizes these differences with a single unified interface:
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import litellm
import json
# Same call works for both providers
response = litellm.completion(
model="anthropic/claude-opus-4-5", # or "gpt-4o" — same code
messages=[{"role": "user", "content": "What is 847 * 293?"}],
tools=openai_tools, # LiteLLM uses OpenAI format, converts for Anthropic
)
# Same response format regardless of provider
tool_calls = response.choices[0].message.tool_calls
for tc in tool_calls:
name = tc.function.name
args = json.loads(tc.function.arguments)
print(f"Tool: {name}, Args: {args}")
Try it: Tool Use MechanicsInteractive
Loading visualization...
Explore this: Write a prompt and watch the model decide which tool to call. Notice how the model generates a JSON object with the function name and arguments — this is not magic, it's just the model outputting structured text. Try ambiguous prompts: the model sometimes picks the wrong tool, showing why tool descriptions matter. Observe how parallel tool requests appear when one query needs multiple tools.
⚡ Playground:Tool Use → — write a prompt and watch the model decide which tool to call and with what arguments.
Modern LLM APIs allow the model to request multiple tools in a single response. This is a major latency optimization for tasks that require independent information.
User: "What is the weather in New York and San Francisco, and what is 847 * 293?"
Without parallel tool calls:
Step 1: get_weather(New York) — wait 800ms
Step 2: get_weather(San Francisco) — wait 800ms
Step 3: calculate(847 * 293) — wait 50ms
Total: ~1,650ms + 3 LLM calls
With parallel tool calls:
Step 1: [get_weather(New York), get_weather(San Francisco), calculate(847 * 293)] — all at once
Total: ~800ms (slowest tool) + 2 LLM calls
import asyncio
import anthropic
import json
from typing import Any
client = anthropic.Anthropic()
async def execute_tool_async(tool_name: str, tool_input: dict) -> Any:
"""Execute a single tool call asynchronously."""
# Route to the appropriate tool function
if tool_name == "get_weather":
return await get_weather_async(tool_input["location"])
elif tool_name == "calculate":
return await calculate_async(tool_input["expression"])
elif tool_name == "search_web":
return await search_web_async(tool_input["query"])
else:
return {"error": "UnknownTool", "message": f"Tool '{tool_name}' is not registered"}
async def handle_parallel_tool_calls(response) -> list[dict]:
"""Execute all tool calls in a single response in parallel."""
tool_use_blocks = [
block for block in response.content
if block.type == "tool_use"
]
if not tool_use_blocks:
return []
# Launch all tool executions simultaneously
tasks = [
execute_tool_async(block.name, block.input)
for block in tool_use_blocks
]
# Wait for all to complete (or fail)
results = await asyncio.gather(*tasks, return_exceptions=True)
# Build tool result messages
tool_results = []
for block, result in zip(tool_use_blocks, results):
if isinstance(result, Exception):
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": json.dumps({
"error": type(result).__name__,
"message": str(result)
}),
"is_error": True
})
else:
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": json.dumps(result)
})
return tool_results
The model returns one response with multiple tool_use blocks. You execute all of them with asyncio.gather. You must return all results together in a single user message before the model continues — you cannot return them one at a time.
Even with grammar constraints, models generate wrong arguments. Missing required fields, wrong types, and invalid values all happen in production. The correct response is always: validate first, return a structured error, let the model self-correct.
When a tool returns a ValidationError, pass it back to the model in the tool result. The model almost always corrects itself on the next attempt:
Attempt 1 — Model generates:
{"tool": "calculate", "input": {"expression": 847, "precision": 2}}
# expression is an int, should be a string
Validation catches this, returns:
{"error": "ValidationError", "field": "expression",
"error": "Input should be a valid string", "received": 847}
Attempt 2 — Model self-corrects:
{"tool": "calculate", "input": {"expression": "847 * 293", "precision": 2}}
# Correct — model learned from the structured error
The key is making the error message informative. A raw Python traceback tells the model nothing useful. A structured error with the field name, what was received, and what was expected gives the model exactly what it needs to correct the argument.
Tool results can be massive. A web search might return 50,000 tokens of HTML. A database query might return 1,000 rows. Adding these directly to the context window is expensive and often degrades reasoning quality (more noise, less signal).
What Do You Think?
Your tool returns a 40,000 token document. Your agent has a 200K context window. After 3 similar tool calls, you are at 120K tokens. After 5 more steps, you hit 200K. What breaks?
Context overflow is a hard failure. The API either returns an error (if you exceed the hard limit) or silently truncates the oldest messages (if your framework does truncation). Either way, the agent loses access to the tool results from early in the conversation — the ones that might contain the most important information.
Three strategies for keeping tool results manageable:
pythonreference · read-only
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import json
from typing import Any
def truncate_tool_result(
result: Any,
max_tokens: int = 2000,
strategy: str = "tail"
) -> str:
"""Truncate a tool result to fit within a token budget."""
result_str = json.dumps(result) if not isinstance(result, str) else result
# Rough approximation: 4 chars ≈ 1 token
max_chars = max_tokens * 4
if len(result_str) <= max_chars:
return result_str
if strategy == "tail":
# Keep the end — often more relevant for search results
truncated = result_str[-max_chars:]
return f"[Truncated {len(result_str) - max_chars} characters from start]\n{truncated}"
elif strategy == "head":
# Keep the beginning
truncated = result_str[:max_chars]
return f"{truncated}\n[Truncated {len(result_str) - max_chars} characters from end]"
elif strategy == "summary":
# Use an LLM to summarize — most expensive but highest quality
summary = summarize_with_llm(result_str, target_tokens=max_tokens)
return f"[Summary of {len(result_str)//4} token result]\n{summary}"
elif strategy == "paginate":
# Return page 1 with instructions to request more
page = result_str[:max_chars]
remaining_pages = len(result_str) // max_chars
return f"{page}\n[Result continues — {remaining_pages} more pages available. Call tool with page=2 to continue.]"
return result_str[:max_chars]
When to use each strategy
Strategy
Use When
head
Tool results are front-loaded (SQL query results, ordered lists)
tail
Tool results get more specific toward the end (log files, search results ranked by relevance)
summary
Result contains essential information spread throughout (documents, code files)
paginate
The tool supports pagination and the agent needs specific pages
A tool call is token generation with grammar constraints. The model generates JSON token by token; constrained decoding restricts the vocabulary at each step to maintain valid JSON structure; this is why you get well-formed JSON but not necessarily correct values
Parallel tool calls require one round-trip, not N. When the model requests multiple tools in a single response, execute all of them with asyncio.gather and return all results together in one message; this cuts latency from N× to 1× the slowest tool
Pydantic validation before execution prevents ghost failures. Validate every tool argument against a typed schema before touching the tool logic; return structured errors with field names and expected types so the model can self-correct; never pass raw tracebacks
Truncate tool results before they inflate context. A 40,000 token tool result that gets added 5 times will overflow most context windows; budget 2,000 tokens per result, choose a truncation strategy based on where the signal lives in the result