Tool Calling Internals: How LLMs Decide to Call a Function

#What is tool calling, really?

A pretrained language model only ever does one thing: given a prefix of tokens, predict the next token. Everything else (chat formatting, system prompts, RAG, agents) is a convention layered on top of that single capability. Tool calling is one such convention. It works like this:

1. The orchestrator (your code) sends the model a prompt that includes descriptions of available tools, usually in JSON-Schema form.
2. The model generates text that includes a structured payload like {"name": "get_weather", "arguments": {"city": "Paris"}} wrapped in special tokens or a designated field.
3. The orchestrator parses that payload, recognizes it as a tool call, and executes the real function get_weather(city="Paris") in your runtime.
4. The orchestrator appends the tool's result back into the conversation as a new message and asks the model to continue.
5. The model generates a final natural-language response using the result.

The crucial point: the model never executes anything. It only emits a string the orchestrator agrees to interpret as a command. The "tool" is whatever real code your orchestrator decides to run when it sees the structured payload.

#Why this matters

Three things shifted in the year before this lesson was written:

1. Every production agent is built on tool calling. Anthropic's computer-use API, Claude Code, the MCP ecosystem, Cursor, Devin, ChatGPT Operator, Replit Ghostwriter, GitHub Copilot Workspace — all of them are tool-calling loops with different prompts. There is no separate "agent model"; the agent is the loop around the tool call.
2. Models are now trained for it. Llama 3.1 / 3.2, Claude 3.5+, GPT-4o, Mistral Large 2, Gemini 1.5+, Granite Code 8B, Qwen 2.5 — all of these ship with tool calling baked into the base model. You do not need to fine-tune a model to use tools any more.
3. The protocol layer is consolidating. OpenAI's tool-call API, Anthropic's tool blocks, and the open Model Context Protocol (MCP) are converging on the same shape: JSON-Schema in, JSON tool-call out, JSON-RPC for the server side.

If you want to build anything that reaches outside the chat window, tool calling is the only bridge. Everything else is plumbing.

#The text-level mechanics

What does a tool call actually look like to the model? It depends on which format the model was trained on. Here are the three you will meet in practice.

#Format 1: Hermes / ChatML tags

<|im_start|>system
You are a helpful assistant with access to functions. Call them
in JSON inside <tool_call> tags.
<tools>
[{"name": "get_weather", "description": "Current weather for a city",
  "parameters": {"type": "object",
                 "properties": {"city": {"type": "string"}},
                 "required": ["city"]}}]
</tools>
<|im_end|>
<|im_start|>user
What's the weather in Paris?
<|im_end|>
<|im_start|>assistant
<tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call>
<|im_end|>

#Format 2: Llama 3 python_tag syntax

<|start_header_id|>assistant<|end_header_id|>

<|python_tag|>get_weather.call(city="Paris")<|eom_id|>

#Format 3: OpenAI / Anthropic API fields

The frontier API providers hide the special tokens from you entirely. You send a request like:

{
  "model": "claude-sonnet-4-7",
  "messages": [{"role": "user", "content": "Weather in Paris?"}],
  "tools": [{
    "name": "get_weather",
    "description": "Current weather for a city",
    "input_schema": {
      "type": "object",
      "properties": {"city": {"type": "string"}},
      "required": ["city"]
    }
  }]
}

And get back a structured response:

{
  "content": [
    {"type": "tool_use", "id": "toolu_01abc",
     "name": "get_weather", "input": {"city": "Paris"}}
  ],
  "stop_reason": "tool_use"
}

#How tool calling is trained into the model

A base model does not know what a tool call is. The behavior is taught during supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO). The recipe, simplified:

1. Synthetic conversation generation. Hire annotators or use a stronger model to generate thousands of dialogues where the right answer is "call a tool". Each dialogue includes a tool catalog in the system prompt, a user question, and an assistant turn that emits a correct tool call in the target format.
2. SFT on (prompt, correct tool call) pairs. The model learns the surface pattern: when the system prompt declares tools and the user asks something that maps to one, emit the tagged JSON.
3. Schema-conditioned generation. Examples include diverse JSON Schemas (nested objects, enums, optional fields) so the model generalizes to schemas it has not seen.
4. DPO on (good_call, bad_call) preference pairs. The "bad" examples include malformed JSON, hallucinated parameters, wrong tool selection, and calls when a tool was not needed. The preference signal pushes the model toward well-formed calls and away from these specific failure modes.
5. Reward shaping for argument correctness. Some pipelines explicitly check argument types against the schema and only count an output as "good" if it round-trips through a validator.

The Toolformer paper (Schick et al. 2023) showed that you can do most of this in a self-supervised way: insert tool calls into a corpus, keep the ones that improve next-token loss on the surrounding text, train on the kept examples. The Gorilla paper (Patil et al. 2023) extended this to a massive catalog of real APIs, training a Llama variant to invoke 1,600+ ML model APIs by name and signature.

#Function description format: what the model actually sees

The format you use to describe a tool dramatically affects whether the model picks the right one. The de facto standard is JSON Schema:

{
  "name": "get_weather",
  "description": "Get the current weather conditions for a city. Returns temperature in Celsius, humidity, wind speed, and a short condition string. Use this when the user asks about weather, temperature, rain, or conditions in a specific location.",
  "parameters": {
    "type": "object",
    "properties": {
      "city": {
        "type": "string",
        "description": "The city name, e.g. 'Paris' or 'San Francisco'. Country optional but recommended for disambiguation."
      },
      "units": {
        "type": "string",
        "enum": ["metric", "imperial"],
        "description": "Units for the temperature. Default 'metric'."
      }
    },
    "required": ["city"]
  }
}

Four pieces matter, and three of them are prose:

• name: a stable identifier the model emits verbatim. Snake_case or camelCase are both fine; the model will copy whatever you give it.
• description at the top level: the most important field. This is the prose the model uses to decide whether to call this tool at all. A description that is vague ("gets weather data") will lose to a description that gives the model concrete trigger phrases ("Use this when the user asks about weather, temperature, rain, or conditions"). Treat this like prompt engineering — because it is.
• parameters schema: structural constraints. Used by both the model (to format arguments correctly) and the validator (to reject malformed output).
• description on each parameter: tells the model how to fill the slot. Concrete examples beat type annotations. "e.g. 'Paris' or 'San Francisco'" is worth ten words of theory.

#Parallel tool calls

Modern frontier models can emit multiple tool calls in a single assistant turn. This is critical for latency: if a user asks "what's the weather in Paris and the stock price of AAPL?", a sequential agent makes two round-trips (call weather, wait, call stock, wait). A parallel-capable model emits both calls at once, the orchestrator runs them in parallel, and the user sees one combined response.

Raw token sequence in the Hermes format:

<|im_start|>assistant
<tool_call>{"name": "get_weather", "arguments": {"city": "Paris"}}</tool_call>
<tool_call>{"name": "get_stock", "arguments": {"ticker": "AAPL"}}</tool_call>
<|im_end|>

#Structured generation: making bad output literally unsamplable

Training the model to emit valid tool calls works most of the time. But "most of the time" is not good enough when an agent makes 100 tool calls per task and one malformed call breaks the whole chain. The robust fix is to enforce structure at decode time — to never let the model sample an invalid token in the first place.

The mechanism is called grammar-constrained decoding, and it works at the logit level:

1. Compile the JSON Schema (or any context-free grammar) into a finite-state machine.
2. At each decode step, ask the FSM: "given the tokens I have generated so far, which token IDs are legal next?"
3. Mask the model's logits so all illegal tokens are pushed to -inf before the softmax.
4. Sample from the remaining valid distribution.

The major implementations:

• Outlines (dottxt-ai): converts Pydantic models, JSON Schema, or regex into FSMs. Compiles once, masks logits at each step. Works on top of any HuggingFace model.
• xgrammar (mlc-ai): high-performance C++ engine that handles JSON Schema and EBNF grammars. Used inside vLLM and SGLang for structured-output serving at scale.
• lm-format-enforcer: similar approach, integrates with vLLM, AutoGPTQ, llama-cpp-python.
• Guidance / Instructor: developer-facing wrappers — Instructor takes a Pydantic model, generates the schema, validates the output, retries on failure. Less rigorous than logit masking (it retries instead of constraining), but very ergonomic.

Run that and watch each failure mode get caught by exactly the layer responsible for it: bad JSON by the parser, missing fields by the schema validator, unknown tools by the registry, well-formed calls by the dispatcher. This is the entire backend of a tool-calling system in 80 lines of Python. Everything you see in MCP, in the OpenAI SDK, in Claude's runtime, is a more elaborate version of the same loop.

#Common production patterns

Tool calling is a primitive. Real agents compose it into one of three shapes.

#Single-turn tool call

The simplest loop:

user → model → tool_call → orchestrator runs tool → tool_result → model → final answer

Two model invocations, one tool execution. This is what a typical "look up something for the user" assistant does. The orchestrator code is roughly 30 lines.

#Chain-of-tools (ReAct loop)

The agent loop:

while not done:
    response = model.complete(messages)
    if response.tool_calls:
        for call in response.tool_calls:
            result = run(call)
            messages.append(tool_result(call.id, result))
    else:
        return response.content  # final answer

#MCP server farms

#Tool execution pitfalls and defenses

Four failure modes show up in production at high enough frequency to deserve names.

#1. Argument hallucination

#2. Wrong tool selection

#3. Infinite loops

The model keeps calling the same tool with slight argument variations, never producing a final answer. Common when the tool result is ambiguous or when the model thinks the user is still waiting for more detail.

#4. Capability surface mismatch

#Tool-augmented training

A short tour of the research that built today's tool-calling models.

• Toolformer (Schick et al., Meta, Feb 2023) — self-supervised tool insertion. Take a text corpus, run a teacher model to propose tool calls at every position, keep only the calls whose result lowers cross-entropy on the next ~10 tokens. Train on the augmented corpus. The model learns when a tool helps, without human annotation.
• Gorilla (Patil et al., Berkeley, May 2023) — large-scale API training. Built a corpus of 1,600+ ML APIs (HuggingFace, TorchHub, TensorFlow Hub), generated training pairs of (user_intent, correct API call), fine-tuned Llama. Showed that with retrieval-augmented training, models can correctly call APIs they have never seen during training, just from their documentation.
• ToolLLM / ToolBench (Qin et al., Tsinghua, Jun 2023) — 16,000 real-world REST APIs, multi-step task generation, DFS-based annotation of correct trajectories.
• Llama 3.1 / 3.2 (Meta, Jul/Sep 2024) — first major open base model with tool calling baked in via the <|python_tag|> token. Supports parallel calls, built-in tools (brave_search, wolfram_alpha, code_interpreter), and custom user-defined tools.
• Granite Code 8B Instruct + Tools (IBM, 2024). Instruction-tuned code model with native tool calling for enterprise integrations.
• Claude 3.x / 4.x (Anthropic). Tool use as a first-class content block (tool_use / tool_result), parallel calls supported since 3.5, computer use (clicking, typing, screen reading) released October 2024.

#Modern API patterns: a head-to-head

The two big providers expose tool calling with nearly identical semantics under slightly different field names.

Functionally the same; cosmetically different. Both convert the JSON Schema into an in-context prompt under the hood. Both run constrained decoding on the tool-call output to guarantee well-formed JSON. Both ship parallel calls. The interesting differences are:

• Anthropic uses content blocks, so a single assistant message can interleave prose and multiple tool calls in one ordered list. OpenAI keeps prose in message.content and tool calls in a parallel tool_calls array.
• OpenAI lets the model emit parallel_tool_calls: false as a request-level flag if you want to force serial calling. Anthropic does not have a native off-switch (you would set tool_choice per round instead).
• Anthropic ships cache_control on tool definitions so a long tool catalog can sit in the prompt cache.

If you are starting from scratch in 2026, write your orchestrator against an abstraction layer (LiteLLM, Anthropic SDK with its OpenAI-compat shim, or your own thin wrapper) and treat the field-name differences as a serialization concern.

#A tiny JSON-mode FSM, by hand

The state set is tiny and the alphabet is small, but everything important about Outlines, xgrammar, lm-format-enforcer, and OpenAI's JSON-mode is in those 40 lines. Real implementations:

• operate at the token level (LLaMA tokenizer has ~128k tokens; the FSM precomputes which tokens are legal in each state),
• compile JSON Schema into a much larger FSM (one state per $ref boundary, one alphabet per regex pattern, etc.),
• use bit-vectors so the per-step legality check is O(1) instead of O(vocab),
• handle the BPE merge ambiguity where one token might cross a state boundary.

None of those are conceptual changes. They are engineering optimizations on the same idea: at every decode step, mask out tokens the grammar forbids, then sample.

#A worked end-to-end, in token order

Putting everything together — here is what actually happens when a user asks Claude "what's the weather in Paris?" with one tool available:

1. Your orchestrator sends the API a messages list with the user message plus a tools list containing the get_weather schema.
2. Anthropic's server converts tools into a system-prompt insertion (<tools>...</tools>) using the format Claude was trained on.
3. The model decodes tokens. At some point its logits over the next-token distribution favor the <tool_use_start> special token. A constrained decoder confirms valid emission and the model continues into the structured payload.
4. The decoder is now generating inside the schema for get_weather. JSON-mode is on; the only sampleable tokens are those legal at the current FSM state.
5. At the end of the JSON, the model emits a stop token. Anthropic's server packages the tool call into a tool_use content block and stops generation with stop_reason: "tool_use".
6. Your orchestrator parses the response, validates input against the schema once more for safety, runs get_weather(city="Paris"), and gets back {"temp_c": 14.2, ...}.
7. Your orchestrator sends a new request, this time with the assistant's tool_use block appended and a new user message containing a matching tool_result block keyed by tool_use_id.
8. The model decodes again, now with both the original question, its own tool call, and the tool's result in context. It emits a normal assistant message, no tool tags this time, that uses the result: "It's 14.2°C and cloudy in Paris right now."
9. Your orchestrator returns that text to the user.

Two model invocations, one tool call. Everything between is plumbing. This is the loop. Once you have written it once, every agent framework on the planet looks the same.

#Forward reference

catalog (JSON Schema) ──▶ model ──▶ structured call ──▶ runtime ──▶ result ──▶ model
                                       (constrained                              ↓
                                        decoding)                          (loop or stop)

Memorize that pipeline; everything else is decoration.