How an AI Agent Decides Which Tool to Use

Quick answer

An AI model does not have a separate "decision module" or a testing environment where it tries tools out before answering.

Tool selection is a specialized form of next-token prediction:

1. Tools are described in plain text: Every tool available to the agent is formatted as a structured JSON Schema inside the prompt. This includes the tool's name, description, and required parameters.
2. The model reads the tool docs on every request: The language model evaluates your query alongside the tool descriptions. Because the model was fine-tuned on thousands of examples of tool-use pairs, it recognizes when a query matches a tool's described purpose.
3. The model outputs a special format: Instead of generating regular conversational words, the model outputs special structured tokens (such as <tool_call>) followed by the exact name of the function and its input arguments.
4. The runtime validates the choice: A host program parses the model's text output against the original schema. If the model chose an existing tool and provided valid arguments, the host runs it. If the arguments are malformed, the host rejects the call and tells the model to try again.

When an agent has only 5 tools, all 5 descriptions are placed directly in the prompt. When an agent has hundreds of tools, a separate search engine or router selects the top 3 or 4 most relevant tools before the main language model ever sees them.

The tool selection pipeline

Here is the exact mechanical journey from a user's question to a selected tool:

Step 1: Tools are just text inside the context window

The most common misconception about AI tools is that the language model is directly connected to software libraries. In reality, the language model is simply looking at text.

When an agent developer defines a tool in Python or JavaScript, the agent framework translates that definition into a standardized specification called JSON Schema:

{
  "name": "lookup_stock_price",
  "description": "Retrieves the latest trading price and daily volume for a public equity ticker.",
  "parameters": {
    "type": "object",
    "properties": {
      "ticker": {
        "type": "string",
        "description": "The stock exchange ticker symbol, e.g., AAPL, RELIANCE."
      },
      "currency": {
        "type": "string",
        "enum": ["USD", "INR", "EUR"],
        "description": "The currency for the returned quote."
      }
    },
    "required": ["ticker"]
  }
}

When you type a message like "What is Apple trading at right now?", the agent runtime wraps your question with the tool schema above and sends the combined text to the model.

To the language model, the tool definition looks no different than a paragraph in a textbook. The model relies entirely on the description field to understand what the tool does and when it should be chosen.

Step 2: How the model scores and picks a tool

How does the model transition from reading a schema to choosing it?

Supervised Fine-Tuning (SFT)

Modern foundation models (GPT-4, Claude 3.5, Gemini 1.5, Llama 3) undergo extensive fine-tuning specifically for tool use. Training datasets contain hundreds of thousands of examples structured like this:

• Input: User asks for real-time data + tool schemas provided in context.
• Target Output: A formatted function call string rather than conversational text.

Through this training, the neural network builds strong internal associations between questions involving current facts, calculations, or external actions and the corresponding tool syntax.

Next-Token Probability

When the model evaluates "What is Apple trading at right now?", it processes the attention patterns across your sentence and the tool schemas.

At the end of the input, the probability of generating a regular word like "Apple" or "The" is suppressed by the model's training. Instead, the model assigns the highest statistical probability to a special delimiter token (such as <|start_tool_call|> or a opening curly brace {).

Next, the model predicts the characters l-o-o-k-u-p-_-s-t-o-c-k-_-p-r-i-c-e because those tokens have the strongest semantic resonance with "trading" and "price" in the context.

What happens when an agent has 100 tools?

Placing 5 tool definitions into a prompt works smoothly. But what happens in an enterprise system with 500 APIs?

If you paste 500 JSON schemas into the prompt:

1. Context Bloat: The schemas consume tens of thousands of tokens before the user has even spoken, costing significant money on every turn.
2. The "Needle in a Haystack" Confusion: As the number of schemas grows, tools with overlapping descriptions confuse the model. The model's selection accuracy drops precipitously.

Production architectures solve this with three distinct routing strategies:

Why agents pick the wrong tool (and how developers fix it)

Understanding how tool selection fails reveals how fragile LLM reasoning can be:

1. The Overlapping Description Trap

Consider two tools:

• search_customer_records(query: string)
• find_user_by_email(email: string)

If a user writes: "Look up john@example.com in our CRM", which tool will the agent choose? Both tools match the semantic concept of finding a customer. If the descriptions do not explicitly define the boundary—e.g., "Use find_user_by_email ONLY when an exact email string is provided; use search_customer_records for names and phone numbers"—the model will essentially flip a coin.

2. Hallucinating Optional Arguments

If a tool takes an optional parameter like date_range, models frequently fabricate values (e.g., assuming last_30_days even when the user didn't specify a timeframe) because the training data favored fully populated JSON objects.

3. Missing Negative Guidance

Models often suffer from eager execution: if you provide a tool called send_slack_message, the agent may call it prematurely while still researching a topic. Developers must add negative constraints directly into the description: "Do NOT call this tool until the user has explicitly confirmed the draft text."

The Expert Layer: Grammar-Constrained Decoding

In mission-critical applications, relying on the model's raw probability to output valid JSON is a liability. A single missing quotation mark or trailing comma causes a parser crash.

Modern runtimes employ Grammar-Constrained Decoding (using tools like Outlines, Guidance, or llama.cpp grammars):

• During generation, the runtime forces the model's vocabulary distribution to adhere to a formal Context-Free Grammar (CFG) derived from the JSON Schema.
• If the schema expects a boolean, the decoding engine mathematically masks out every token in the model's vocabulary except true and false.
• This guarantees 100% syntactically valid tool calls at the token level, eliminating argument syntax failures entirely.

Related explainers in this series

• How AI Agents Work — The complete architectural loop: models, runtimes, memory, and failure modes.
• How Large Language Models Generate Text — How attention and token probability distributions function under the hood.
• How UPI Works — How deterministic financial protocols compare to probabilistic agent decision loops.

Authoritative sources & references

1. OpenAI Technical Documentation (2024). Structured Outputs and JSON Schema Enforcement. platform.openai.com/docs/guides/structured-outputs.
2. Willard, B. T., & Louf, R. (2023). Efficient Guided Generation for Large Language Models. arXiv:2307.09702. (Grammar-constrained token sampling).
3. Patil, S. G., et al. (2023). Gorilla: Large Language Model Connected with Massive APIs. UC Berkeley & Microsoft Research. arXiv:2305.15334.
4. Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. Meta AI Research.