Tool use: the model asks, the program acts

Code in:JavaC#PythonJavaScript

In short: Tool use lets the model ask the program to run a function, such as an order status lookup, and answer from the result instead of from memory.

Problem

The model does not know your orders

The customer asks: “When will order 1042 arrive?” The assistant asks the model without giving it the data:

const client = new FakeClient();
const question = "When will order 1042 arrive?";
const messages = [{ role: "user", content: [{ type: "text", text: question }] }];
const reply = client.create(SYSTEM, messages, []);
console.log(`model: ${textOf(reply)}`);
const order = ORDERS["1042"];
console.log(`shop: ${order.status}, delivery ${order.eta}`);

Output

model: Order 1042 will arrive by Friday.
shop: shipped, delivery 3 November

The model says “by Friday”, and the shop says 3 November. It has not seen your database. It has only the text you sent it, and it makes up something plausible.

In the lesson about the prompt the order data went into the template. That works when you know in advance which data is needed. But the customer may ask about another order, about a return, or about two things at once. You cannot put the whole shop into the prompt: the context is limited, and every word costs money.

The idea

The model asks, the program acts

You give the model a list of tools. Each has a name, a description and a schema of its input: the same kind of schema as in the lesson about structured output. Now the model can answer in two ways:

• With text. stop_reason is end_turn and the conversation is over.
• With a request. stop_reason is tool_use, and the answer has a tool_use block: an id, the tool’s name and an input, for example {"order_id": "1042"}.

On a request the program:

1. Adds the model’s answer to the history, with the role assistant.
2. Runs the tool against its own data.
3. Returns the result as a new user message with a tool_result block. In it tool_use_id is the same id, and on an error is_error is true.
4. Asks the model again, with the whole history.

This is the tool loop (tool use). The model runs nothing and does not see the database. It only says what it needs, and the program decides whether and how to do it. That is why the assistant’s tools only read: the status of an order, the deadline for a return.

The tool’s input comes from the model, that is, from outside. The program checks it the way it checks a form.

See it

The tool and the blocks

The tool the model can ask for. The description is for the model: it uses it to decide when to call the tool.

const TOOLS = [
  {
    name: "get_order_status",
    description: "Status and delivery date of an order by its number.",
    input_schema: {
      type: "object",
      properties: { order_id: { type: "string" } },
      required: ["order_id"],
    },
  },
];

The blocks are objects with the API’s fields: type says the kind, and tool_use has id, name and input. In Python and JavaScript the tool above is in exactly the shape the API accepts.

The program runs what was asked. First it checks that there is such a tool and that the input has the required fields, then it looks up the order:

function runTool(use) {
  const tool = TOOLS.find((t) => t.name === use.name);
  if (tool === undefined) return error(use, `no tool ${use.name}`);
  for (const field of tool.input_schema.required) {
    if (!Object.hasOwn(use.input, field)) return error(use, `missing ${field}`);
  }
  const orderId = use.input.order_id;
  const order = ORDERS[orderId];
  if (order === undefined) return error(use, `no order ${orderId}`);
  const text = `${orderId} — ${order.status}, delivery ${order.eta}`;
  return { type: "tool_result", tool_use_id: use.id, content: text };
}

function error(use, text) {
  return { type: "tool_result", tool_use_id: use.id, content: text, is_error: true };
}

Every error is a result with is_error, not an exception. The model has to understand what happened and decide what to tell the customer.

See it

The loop

function answer(client, question) {
  console.log(`question: ${question}`);
  const messages = [
    { role: "user", content: [{ type: "text", text: question }] },
  ];
  for (let call = 1; call <= MAX_CALLS; call++) {
    const reply = client.create(SYSTEM, messages, TOOLS);
    console.log(`  call ${call} (~${reply.inputTokens} tokens):`,
      reply.stopReason);
    if (reply.stopReason !== "tool_use") {
      console.log(`  answer: ${textOf(reply)}`);
      return;
    }
    messages.push({ role: "assistant", content: reply.content });
    const results = [];
    for (const block of reply.content) {
      if (block.type === "tool_use") {
        const result = runTool(block);
        const shown = (result.is_error ? "✗ " : "")
          + result.content;
        console.log(`    ${block.id} ${describe(block)} → ${shown}`);
        results.push(result);
      }
    }
    messages.push({ role: "user", content: results });
  }
  console.log(`  stopping after ${MAX_CALLS} calls → to a person`);
}

Step by step, in words

1. The question is the first message. It goes to the model together with the list of tools (~41 tokens).
2. The model does not know the status. It answers with stop_reason tool_use: “run get_order_status with order_id 1042”.
3. The program keeps the model’s answer in the history: the result must have a request to answer.
4. The program runs the tool against its own data. The model never touches the shop: it only asked.
5. The result goes back as a user message: tool_result with the same id, call_1.
6. Call 2 sends the whole history again: ~52 tokens instead of ~41.
7. Now the model has the data and answers with end_turn. Two calls to the model for one question.

The whole loop. Two helper functions say what the model asked for and what the answer’s text is:

function describe(use) {
  return `${use.name}(${use.input.order_id ?? ""})`;
}

function textOf(reply) {
  return reply.content.find((b) => b.type === "text")?.text ?? "";
}

function answer(client, question) {
  console.log(`question: ${question}`);
  const messages = [
    { role: "user", content: [{ type: "text", text: question }] },
  ];
  for (let call = 1; call <= MAX_CALLS; call++) {
    const reply = client.create(SYSTEM, messages, TOOLS);
    console.log(`  call ${call} (~${reply.inputTokens} tokens):`,
      reply.stopReason);
    if (reply.stopReason !== "tool_use") {
      console.log(`  answer: ${textOf(reply)}`);
      return;
    }
    messages.push({ role: "assistant", content: reply.content });
    const results = [];
    for (const block of reply.content) {
      if (block.type === "tool_use") {
        const result = runTool(block);
        const shown = (result.is_error ? "✗ " : "")
          + result.content;
        console.log(`    ${block.id} ${describe(block)} → ${shown}`);
        results.push(result);
      }
    }
    messages.push({ role: "user", content: results });
  }
  console.log(`  stopping after ${MAX_CALLS} calls → to a person`);
}

const client = new FakeClient();
answer(client, "When will order 1042 arrive?");

Output

question: When will order 1042 arrive?
  call 1 (~41 tokens): tool_use
    call_1 get_order_status(1042) → 1042 — shipped, delivery 3 November
  call 2 (~52 tokens): end_turn
  answer: I checked: 1042 — shipped, delivery 3 November.

One question from the customer makes two calls to the model. The second sends the whole history and is bigger: ~52 tokens instead of ~41.

The model here is a fake client, as in the earlier lessons: from the words of the question it asks for the tools it has, and after the results it builds an answer from them. Its rule is in “More detail”.

Try it

Fill in the blank

When does the loop stop and print the answer? What is missing in the check?

if (reply.stopReason !== "") {

Check

Try it

Predict the output

The customer only says thanks. What does answer print?

const client = new FakeClient();
answer(client, "Thanks, that is all.");

Answer

question: Thanks, that is all.
  call 1 (~40 tokens): end_turn
  answer: You are welcome! Write to us if you have another question.

question: Thanks, that is all.
  call 1 (~40 tokens): end_turn
  call 2 (~49 tokens): end_turn
  answer: You are welcome! Write to us if you have another question.

question: Thanks, that is all.
  call 1 (~40 tokens): tool_use
  stopping after 4 calls → to a person

Check

The idea

When a tool fails

Tools fail: the number does not exist, a field is missing, the database does not answer. The program does not throw an exception and does not hide the error. It returns it to the model as a tool_result with is_error, and the model decides what to say:

const client = new FakeClient();
answer(client, "Where is order 9999?");
answer(client, "Where is my parcel?");

Output

question: Where is order 9999?
  call 1 (~40 tokens): tool_use
    call_1 get_order_status(9999) → ✗ no order 9999
  call 2 (~47 tokens): end_turn
  answer: I cannot find that order. Please check the number.
question: Where is my parcel?
  call 1 (~40 tokens): tool_use
    call_2 get_order_status() → ✗ missing order_id
  call 2 (~45 tokens): end_turn
  answer: What is your order number?

• Order 9999 does not exist. The model says so to the customer instead of inventing a status.
• The question has no number. The model still asked for the tool, but without order_id. The check catches the missing field, and the model asks the customer for the number.

Some models do not stop. This one guesses the number and asks for one after another:

const client = new FakeClient();
answer(client, "I forgot the number, it was around 1050.");

Output

question: I forgot the number, it was around 1050.
  call 1 (~45 tokens): tool_use
    call_1 get_order_status(1050) → ✗ no order 1050
  call 2 (~52 tokens): tool_use
    call_2 get_order_status(1051) → ✗ no order 1051
  call 3 (~59 tokens): tool_use
    call_3 get_order_status(1052) → ✗ no order 1052
  call 4 (~66 tokens): tool_use
    call_4 get_order_status(1053) → ✗ no order 1053
  stopping after 4 calls → to a person

That is why the loop has a limit: at most 4 calls, then the conversation goes to a person. Without the limit the program spins until the money runs out.

Try it

Find the bug

A colleague writes a shorter runTool. The program sends the result, but the API refuses the second call: the tool_use has no answer. Which line has the bug?

Pick the line with the bug.

1function runTool(use) {2 const orderId = use.input.order_id;3 const order = ORDERS[orderId];4 if (order === undefined) return error(use, `no order ${orderId}`);5 const text = `${orderId} — ${order.status}`;6 return { type: "tool_result", tool_use_id: use.name, content: text };7}

Check

The idea

Run it at your place

This program is not part of the lesson’s check: you need an account with a model provider, a key and money for every call. The key is in LLM_API_KEY, never in the code. Providers ship official SDKs for Java, C#, Python and JavaScript with the same shape (messages, roles, tokens, tools); the exact class and method names are in the provider’s documentation. Here is what travels over the network for the question about 1042. The first request carries the tools, and the model name is a placeholder:

{
  "model": "<model name>",
  "max_tokens": 1024,
  "system": "You are the support assistant of an online shop. Ask the tools about orders.",
  "tools": [
    {
      "name": "get_order_status",
      "description": "Status and delivery date of an order by its number.",
      "input_schema": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"]
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "When will order 1042 arrive?"
    }
  ]
}

The answer is a request for a tool:

{
  "content": [
    {
      "type": "tool_use",
      "id": "call_1",
      "name": "get_order_status",
      "input": {"order_id": "1042"}
    }
  ],
  "stop_reason": "tool_use",
  "usage": {"input_tokens": 41, "output_tokens": 3}
}

The second request carries the whole history: the question, the model’s tool_use and the program’s tool_result with the same id:

{
  "model": "<model name>",
  "max_tokens": 1024,
  "system": "You are the support assistant of an online shop. Ask the tools about orders.",
  "tools": [
    {
      "name": "get_order_status",
      "description": "Status and delivery date of an order by its number.",
      "input_schema": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"]
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "When will order 1042 arrive?"
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "tool_use",
          "id": "call_1",
          "name": "get_order_status",
          "input": {"order_id": "1042"}
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "tool_result",
          "tool_use_id": "call_1",
          "content": "1042 — shipped, delivery 3 November"
        }
      ]
    }
  ]
}

{
  "content": [
    {
      "type": "text",
      "text": "I checked: 1042 — shipped, delivery 3 November."
    }
  ],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 52, "output_tokens": 11}
}

On an error the result has is_error:

{
  "type": "tool_result",
  "tool_use_id": "call_1",
  "content": "no order 9999",
  "is_error": true
}

Each request goes out with the language’s standard HTTP client, and the loop around it is the same as in answer. The address is a placeholder:

// requestJson is the request JSON above.
// The exact key header is in the provider’s documentation.
const response = await fetch("https://<provider>/messages", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LLM_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: requestJson,
});
console.log(await response.text());

Real ids are long random strings. The fake client counts them: call_1, call_2.

The idea

When it hurts

• A tool that acts instead of reading. “Cancel the order” or “refund the money” is not given to the model without a person who confirms. The model may get the order wrong, and here a mistake costs money.
• Too many tools. Every tool is a description sent with every call. Twenty tools are hundreds of tokens and a model that picks the wrong one more often.
• Vague descriptions. The model chooses by the description. “Returns data” says nothing. “Status and delivery date of an order by its number” says when to use it.
• Trusting the input. order_id comes from the model. The program checks that the order exists and belongs to this customer before it shows anything. If a tool reads someone else’s text (an email, a web page), that text may contain instructions to the model. That attack (prompt injection) has its own lesson in the agents module.
• A loop without a limit. Every call sends the whole history and costs more than the one before. How much exactly is in the lesson about cost and speed. Whether the assistant answers a hundred questions right, not just one, is measured with a test set.

More precisely, if you are curious

Why the model’s answer goes into the history. The API has no memory. On the second call the model sees only what you send it. A tool_result without a tool_use before it is an answer without a question, and the API refuses it.

Two tools in one call. The model can ask for several tools at once. The program runs them all and returns all the results in one user message, each with its own tool_use_id. That way there are fewer calls to the model.

Forcing a choice. The API also has a tool_choice field. The default, auto, lets the model decide whether to use a tool. Newer models refuse to be forced into a tool, so this lesson does not do it.

The fake client inside. If there are no tools, it invents a date. If the question is “around 1050”, it guesses number after number. If the last message has results, it builds an answer from them, and on an error it asks for the number or says it cannot find the order. Otherwise it asks for the tool by the number in the question: 1042 and 9999 for the status, 1187 for a return. The tokens are words × 1.3, and the tool descriptions are counted on every call.

// Stands in for the real model: recorded answers,
// with no network and the same every time.
class FakeClient {
  static MADE_UP = "Order 1042 will arrive by Friday.";
  static ASK_ID = "What is your order number?";
  static NOT_FOUND = "I cannot find that order. Please check the number.";
  static THANKS = "You are welcome! Write to us if you have another question.";
  lastId = 0;

  create(system, messages, tools) {
    const question = messages[0].content[0].text;
    const last = messages[messages.length - 1].content;
    const content = [];
    if (tools.length === 0) {
      content.push({ type: "text", text: FakeClient.MADE_UP });
    } else if (question.includes("around")) {
      // Guesses the next number until someone stops it.
      const guess = 1050 + Math.floor(messages.length / 2);
      this.ask(content, tools, "get_order_status", String(guess));
    } else if (last[0].type === "tool_result") {
      content.push({ type: "text", text: FakeClient.afterResults(last) });
    } else {
      if (question.includes("1042")) this.ask(content, tools, "get_order_status", "1042");
      if (question.includes("9999")) this.ask(content, tools, "get_order_status", "9999");
      if (question.includes("1187")) this.ask(content, tools, "get_return_window", "1187");
      if (question.includes("parcel")) this.ask(content, tools, "get_order_status", null);
      if (content.length === 0) content.push({ type: "text", text: FakeClient.THANKS });
    }
    let inputTokens = tokens(system);
    for (const t of tools) inputTokens += tokens(t.name + " " + t.description);
    for (const m of messages) {
      for (const b of m.content) inputTokens += tokens(words(b));
    }
    let outputTokens = 0;
    for (const b of content) outputTokens += tokens(words(b));
    const stopReason = content[0].type === "tool_use" ? "tool_use" : "end_turn";
    return { content, stopReason, inputTokens, outputTokens };
  }

  ask(content, tools, name, orderId) {
    if (!tools.some((t) => t.name === name)) return;
    const input = orderId === null ? {} : { order_id: orderId };
    this.lastId++;
    content.push({ type: "tool_use", id: `call_${this.lastId}`, name, input });
  }

  // A real model retells them; this one only glues the results.
  static afterResults(results) {
    const parts = [];
    for (const r of results) {
      if (r.is_error) {
        return r.content.includes("order_id") ? FakeClient.ASK_ID : FakeClient.NOT_FOUND;
      }
      parts.push(r.content);
    }
    return "I checked: " + parts.join("; ") + ".";
  }
}

function words(block) {
  if (block.type === "text") return block.text;
  if (block.type === "tool_use") {
    return block.name + " " + Object.values(block.input).join(" ");
  }
  return block.content;
}

// Roughly: words × 1.3, rounded up.
function tokens(text) {
  const t = text.trim();
  return Math.floor(((t ? t.split(/\s+/).length : 0) * 13 + 9) / 10);
}

Exercise

The return deadline

Solve it in your editor.

The customer asks: “When does 1042 arrive and until when can I return 1187?” Now the model has only get_order_status and answers half of it:

Output

question: When does 1042 arrive and until when can I return 1187?
  call 1 (~49 tokens): tool_use
    call_1 get_order_status(1042) → 1042 — shipped, delivery 3 November
  call 2 (~60 tokens): end_turn
  answer: I checked: 1042 — shipped, delivery 3 November.

Add a second tool get_return_window with the description “The last day an order can be returned, by its number.” and the input order_id. In runTool return 1187 — return by … with the order’s return field. The model will ask for both tools in one call:

Output

question: When does 1042 arrive and until when can I return 1187?
  call 1 (~65 tokens): tool_use
    call_1 get_order_status(1042) → 1042 — shipped, delivery 3 November
    call_2 get_return_window(1187) → 1187 — return by 9 November
  call 2 (~87 tokens): end_turn
  answer: I checked: 1042 — shipped, delivery 3 November; 1187 — return by 9 November.

More tasks

• Someone else’s order. Add the customer it belongs to to each order, and the customer who asks to answer. runTool returns an error for another customer’s order. What does the model say then?
• A token budget. Stop the loop also when the sum of the input tokens passes 100. On which call does the question “around 1050” stop?

Test

Test yourself

1. The model answered with stop_reason tool_use and a get_order_status block with order_id 1042. Who looks the order up in the database?

The model: since it knows the tool’s name, it runs itThe program: it runs the tool and returns the result to the modelThe model provider, on its own servers

2. One answer has two tool_use blocks. How does the model know which result belongs to which request?

By tool_use_id: every tool_result carries the id of its tool_useBy the tool’s nameBy the order: the first result is for the first request

3. The customer asks about two things and the model asks for both tools in one answer. How many calls to the model does answer() make?

Two: one that asks for the tools and one with the resultsOneThree: one per tool and one for the answer

4. Why does the loop have a limit, MAX_CALLS?

The API does not allow more than four callsThe model may ask for tools forever, and every call costs tokens and timeSo that the database does not fill up with orders

Check

Start the lesson over

Related lessons

• Builds on: The limits of the model · Calling a model through an API · The prompt is code · Structured output: JSON by a schema
• Continues in: The agent loop: plan, action, observation · Prompt injection: the email that gives orders · Guardrails: checks before and after the model · The support assistant
• Tasks on this topic: Apps with LLMs