Skip to content
wannacode

Tool use: the model asks, the program acts

In short: Tool use lets the model ask the program to run a function, such as an order status lookup, and answer from the result instead of from memory.

Problem

The model does not know your orders

The customer asks: “When will order 1042 arrive?” The assistant asks the model without giving it the data:

FakeClient client = new FakeClient();
String question = "When will order 1042 arrive?";
List<Message> messages =
        List.of(new Message("user", List.of(new TextBlock(question))));
Reply reply = client.create(SYSTEM, messages, List.of());
System.out.println("model: " + textOf(reply));
Order order = ORDERS.get("1042");
System.out.println("shop: " + order.status() + ", delivery " + order.eta());
Output
model: Order 1042 will arrive by Friday.
shop: shipped, delivery 3 November

The model says “by Friday”, and the shop says 3 November. It has not seen your database. It has only the text you sent it, and it makes up something plausible.

In the lesson about the prompt the order data went into the template. That works when you know in advance which data is needed. But the customer may ask about another order, about a return, or about two things at once. You cannot put the whole shop into the prompt: the context is limited, and every word costs money.

The idea

The model asks, the program acts

You give the model a list of tools. Each has a name, a description and a schema of its input: the same kind of schema as in the lesson about structured output. Now the model can answer in two ways:

  • With text. stop_reason is end_turn and the conversation is over.
  • With a request. stop_reason is tool_use, and the answer has a tool_use block: an id, the tool’s name and an input, for example {"order_id": "1042"}.

On a request the program:

  1. Adds the model’s answer to the history, with the role assistant.
  2. Runs the tool against its own data.
  3. Returns the result as a new user message with a tool_result block. In it tool_use_id is the same id, and on an error is_error is true.
  4. Asks the model again, with the whole history.

This is the tool loop (tool use). The model runs nothing and does not see the database. It only says what it needs, and the program decides whether and how to do it. That is why the assistant’s tools only read: the status of an order, the deadline for a return.

The tool’s input comes from the model, that is, from outside. The program checks it the way it checks a form.

See it

The tool and the blocks

The tool the model can ask for. The description is for the model: it uses it to decide when to call the tool.

static final List<Tool> TOOLS = List.of(
        new Tool("get_order_status",
                "Status and delivery date of an order by its number.",
                List.of("order_id")));
A message’s content is a list of blocks. A block is one of three kinds, and sealed says exactly which:
sealed interface Block permits TextBlock, ToolUse, ToolResult {}


record TextBlock(String text) implements Block {}


record ToolUse(String id, String name, Map<String, String> input) implements Block {}


record ToolResult(String toolUseId, String content, boolean isError) implements Block {}


record Message(String role, List<Block> content) {}


record Reply(List<Block> content, String stopReason, int inputTokens, int outputTokens) {}


record Tool(String name, String description, List<String> required) {}


record Order(String status, String eta, String returnBy) {}

The program runs what was asked. First it checks that there is such a tool and that the input has the required fields, then it looks up the order:

static ToolResult runTool(ToolUse use) {
    Tool tool = TOOLS.stream()
            .filter(t -> t.name().equals(use.name()))
            .findFirst().orElse(null);
    if (tool == null) {
        return new ToolResult(use.id(), "no tool " + use.name(), true);
    }
    for (String field : tool.required()) {
        if (!use.input().containsKey(field)) {
            return new ToolResult(use.id(), "missing " + field, true);
        }
    }
    String orderId = use.input().get("order_id");
    Order order = ORDERS.get(orderId);
    if (order == null) {
        return new ToolResult(use.id(), "no order " + orderId, true);
    }
    String text = orderId + " — " + order.status() + ", delivery " + order.eta();
    return new ToolResult(use.id(), text, false);
}

Every error is a result with is_error, not an exception. The model has to understand what happened and decide what to tell the customer.

See it

The loop

static void answer(FakeClient client, String question) {
    System.out.println("question: " + question);
    List<Message> messages = new ArrayList<>();
    messages.add(new Message("user",
            List.of(new TextBlock(question))));
    for (int call = 1; call <= MAX_CALLS; call++) {
        Reply reply = client.create(SYSTEM, messages, TOOLS);
        System.out.printf("  call %d (~%d tokens): %s%n",
                call, reply.inputTokens(), reply.stopReason());
        if (!reply.stopReason().equals("tool_use")) {
            System.out.println("  answer: " + textOf(reply));
            return;
        }
        messages.add(new Message("assistant", reply.content()));
        List<Block> results = new ArrayList<>();
        for (Block block : reply.content()) {
            if (block instanceof ToolUse use) {
                ToolResult result = runTool(use);
                String shown = (result.isError() ? "✗ " : "")
                        + result.content();
                System.out.printf("    %s %s → %s%n",
                        use.id(), describe(use), shown);
                results.add(result);
            }
        }
        messages.add(new Message("user", results));
    }
    System.out.println("  stopping after " + MAX_CALLS + " calls → to a person");
}
Step by step, in words
  1. The question is the first message. It goes to the model together with the list of tools (~41 tokens).
  2. The model does not know the status. It answers with stop_reason tool_use: “run get_order_status with order_id 1042”.
  3. The program keeps the model’s answer in the history: the result must have a request to answer.
  4. The program runs the tool against its own data. The model never touches the shop: it only asked.
  5. The result goes back as a user message: tool_result with the same id, call_1.
  6. Call 2 sends the whole history again: ~52 tokens instead of ~41.
  7. Now the model has the data and answers with end_turn. Two calls to the model for one question.

The whole loop. Two helper functions say what the model asked for and what the answer’s text is:

static String describe(ToolUse use) {
    return use.name() + "(" + use.input().getOrDefault("order_id", "") + ")";
}


static String textOf(Reply reply) {
    for (Block block : reply.content()) {
        if (block instanceof TextBlock t) {
            return t.text();
        }
    }
    return "";
}
static void answer(FakeClient client, String question) {
    System.out.println("question: " + question);
    List<Message> messages = new ArrayList<>();
    messages.add(new Message("user",
            List.of(new TextBlock(question))));
    for (int call = 1; call <= MAX_CALLS; call++) {
        Reply reply = client.create(SYSTEM, messages, TOOLS);
        System.out.printf("  call %d (~%d tokens): %s%n",
                call, reply.inputTokens(), reply.stopReason());
        if (!reply.stopReason().equals("tool_use")) {
            System.out.println("  answer: " + textOf(reply));
            return;
        }
        messages.add(new Message("assistant", reply.content()));
        List<Block> results = new ArrayList<>();
        for (Block block : reply.content()) {
            if (block instanceof ToolUse use) {
                ToolResult result = runTool(use);
                String shown = (result.isError() ? "✗ " : "")
                        + result.content();
                System.out.printf("    %s %s → %s%n",
                        use.id(), describe(use), shown);
                results.add(result);
            }
        }
        messages.add(new Message("user", results));
    }
    System.out.println("  stopping after " + MAX_CALLS + " calls → to a person");
}
FakeClient client = new FakeClient();
answer(client, "When will order 1042 arrive?");
Output
question: When will order 1042 arrive?
  call 1 (~41 tokens): tool_use
    call_1 get_order_status(1042) → 1042 — shipped, delivery 3 November
  call 2 (~52 tokens): end_turn
  answer: I checked: 1042 — shipped, delivery 3 November.

One question from the customer makes two calls to the model. The second sends the whole history and is bigger: ~52 tokens instead of ~41.

The model here is a fake client, as in the earlier lessons: from the words of the question it asks for the tools it has, and after the results it builds an answer from them. Its rule is in “More detail”.

Try it

Fill in the blank

When does the loop stop and print the answer? What is missing in the check?

if (!reply.stopReason().equals("")) {

Try it

Predict the output

The customer only says thanks. What does answer print?

FakeClient client = new FakeClient();
answer(client, "Thanks, that is all.");
Answer

The idea

When a tool fails

Tools fail: the number does not exist, a field is missing, the database does not answer. The program does not throw an exception and does not hide the error. It returns it to the model as a tool_result with is_error, and the model decides what to say:

FakeClient client = new FakeClient();
answer(client, "Where is order 9999?");
answer(client, "Where is my parcel?");
Output
question: Where is order 9999?
  call 1 (~40 tokens): tool_use
    call_1 get_order_status(9999) → ✗ no order 9999
  call 2 (~47 tokens): end_turn
  answer: I cannot find that order. Please check the number.
question: Where is my parcel?
  call 1 (~40 tokens): tool_use
    call_2 get_order_status() → ✗ missing order_id
  call 2 (~45 tokens): end_turn
  answer: What is your order number?
  • Order 9999 does not exist. The model says so to the customer instead of inventing a status.
  • The question has no number. The model still asked for the tool, but without order_id. The check catches the missing field, and the model asks the customer for the number.

Some models do not stop. This one guesses the number and asks for one after another:

FakeClient client = new FakeClient();
answer(client, "I forgot the number, it was around 1050.");
Output
question: I forgot the number, it was around 1050.
  call 1 (~45 tokens): tool_use
    call_1 get_order_status(1050) → ✗ no order 1050
  call 2 (~52 tokens): tool_use
    call_2 get_order_status(1051) → ✗ no order 1051
  call 3 (~59 tokens): tool_use
    call_3 get_order_status(1052) → ✗ no order 1052
  call 4 (~66 tokens): tool_use
    call_4 get_order_status(1053) → ✗ no order 1053
  stopping after 4 calls → to a person

That is why the loop has a limit: at most 4 calls, then the conversation goes to a person. Without the limit the program spins until the money runs out.

Try it

Find the bug

A colleague writes a shorter runTool. The program sends the result, but the API refuses the second call: the tool_use has no answer. Which line has the bug?

Pick the line with the bug.

The idea

Run it at your place

This program is not part of the lesson’s check: you need an account with a model provider, a key and money for every call. The key is in LLM_API_KEY, never in the code. Providers ship official SDKs for Java, C#, Python and JavaScript with the same shape (messages, roles, tokens, tools); the exact class and method names are in the provider’s documentation. Here is what travels over the network for the question about 1042. The first request carries the tools, and the model name is a placeholder:

{
  "model": "<model name>",
  "max_tokens": 1024,
  "system": "You are the support assistant of an online shop. Ask the tools about orders.",
  "tools": [
    {
      "name": "get_order_status",
      "description": "Status and delivery date of an order by its number.",
      "input_schema": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"]
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "When will order 1042 arrive?"
    }
  ]
}

The answer is a request for a tool:

{
  "content": [
    {
      "type": "tool_use",
      "id": "call_1",
      "name": "get_order_status",
      "input": {"order_id": "1042"}
    }
  ],
  "stop_reason": "tool_use",
  "usage": {"input_tokens": 41, "output_tokens": 3}
}

The second request carries the whole history: the question, the model’s tool_use and the program’s tool_result with the same id:

{
  "model": "<model name>",
  "max_tokens": 1024,
  "system": "You are the support assistant of an online shop. Ask the tools about orders.",
  "tools": [
    {
      "name": "get_order_status",
      "description": "Status and delivery date of an order by its number.",
      "input_schema": {
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"]
      }
    }
  ],
  "messages": [
    {
      "role": "user",
      "content": "When will order 1042 arrive?"
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "tool_use",
          "id": "call_1",
          "name": "get_order_status",
          "input": {"order_id": "1042"}
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "tool_result",
          "tool_use_id": "call_1",
          "content": "1042 — shipped, delivery 3 November"
        }
      ]
    }
  ]
}
{
  "content": [
    {
      "type": "text",
      "text": "I checked: 1042 — shipped, delivery 3 November."
    }
  ],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 52, "output_tokens": 11}
}

On an error the result has is_error:

{
  "type": "tool_result",
  "tool_use_id": "call_1",
  "content": "no order 9999",
  "is_error": true
}

Each request goes out with the language’s standard HTTP client, and the loop around it is the same as in answer. The address is a placeholder:

// requestJson is the request JSON above.
// The exact key header is in the provider’s documentation.
HttpClient http = HttpClient.newHttpClient();
HttpRequest request = HttpRequest.newBuilder(URI.create("https://<provider>/messages"))
        .header("Authorization", "Bearer " + System.getenv("LLM_API_KEY"))
        .header("Content-Type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(requestJson))
        .build();
HttpResponse<String> response =
        http.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.body());

Real ids are long random strings. The fake client counts them: call_1, call_2.

The idea

When it hurts

  • A tool that acts instead of reading. “Cancel the order” or “refund the money” is not given to the model without a person who confirms. The model may get the order wrong, and here a mistake costs money.
  • Too many tools. Every tool is a description sent with every call. Twenty tools are hundreds of tokens and a model that picks the wrong one more often.
  • Vague descriptions. The model chooses by the description. “Returns data” says nothing. “Status and delivery date of an order by its number” says when to use it.
  • Trusting the input. order_id comes from the model. The program checks that the order exists and belongs to this customer before it shows anything. If a tool reads someone else’s text (an email, a web page), that text may contain instructions to the model. That attack (prompt injection) has its own lesson in the agents module.
  • A loop without a limit. Every call sends the whole history and costs more than the one before. How much exactly is in the lesson about cost and speed. Whether the assistant answers a hundred questions right, not just one, is measured with a test set.
More precisely, if you are curious

Why the model’s answer goes into the history. The API has no memory. On the second call the model sees only what you send it. A tool_result without a tool_use before it is an answer without a question, and the API refuses it.

Two tools in one call. The model can ask for several tools at once. The program runs them all and returns all the results in one user message, each with its own tool_use_id. That way there are fewer calls to the model.

Forcing a choice. The API also has a tool_choice field. The default, auto, lets the model decide whether to use a tool. Newer models refuse to be forced into a tool, so this lesson does not do it.

The fake client inside. If there are no tools, it invents a date. If the question is “around 1050”, it guesses number after number. If the last message has results, it builds an answer from them, and on an error it asks for the number or says it cannot find the order. Otherwise it asks for the tool by the number in the question: 1042 and 9999 for the status, 1187 for a return. The tokens are words × 1.3, and the tool descriptions are counted on every call.

// Stands in for the real model: recorded answers,
// with no network and the same every time.
class FakeClient {
    static final String MADE_UP = "Order 1042 will arrive by Friday.";
    static final String ASK_ID = "What is your order number?";
    static final String NOT_FOUND = "I cannot find that order. Please check the number.";
    static final String THANKS = "You are welcome! Write to us if you have another question.";
    private int lastId = 0;


    Reply create(String system, List<Message> messages, List<Tool> tools) {
        String question = ((TextBlock) messages.get(0).content().get(0)).text();
        List<Block> last = messages.get(messages.size() - 1).content();
        List<Block> content = new ArrayList<>();
        if (tools.isEmpty()) {
            content.add(new TextBlock(MADE_UP));
        } else if (question.contains("around")) {
            // Guesses the next number until someone stops it.
            int guess = 1050 + messages.size() / 2;
            ask(content, tools, "get_order_status", String.valueOf(guess));
        } else if (last.get(0) instanceof ToolResult) {
            content.add(new TextBlock(afterResults(last)));
        } else {
            if (question.contains("1042")) {
                ask(content, tools, "get_order_status", "1042");
            }
            if (question.contains("9999")) {
                ask(content, tools, "get_order_status", "9999");
            }
            if (question.contains("1187")) {
                ask(content, tools, "get_return_window", "1187");
            }
            if (question.contains("parcel")) {
                ask(content, tools, "get_order_status", null);
            }
            if (content.isEmpty()) {
                content.add(new TextBlock(THANKS));
            }
        }
        int input = tokens(system);
        for (Tool t : tools) {
            input += tokens(t.name() + " " + t.description());
        }
        for (Message m : messages) {
            for (Block b : m.content()) {
                input += tokens(words(b));
            }
        }
        int output = 0;
        for (Block b : content) {
            output += tokens(words(b));
        }
        String stop = content.get(0) instanceof ToolUse ? "tool_use" : "end_turn";
        return new Reply(content, stop, input, output);
    }


    private void ask(List<Block> content, List<Tool> tools, String name, String orderId) {
        if (tools.stream().noneMatch(t -> t.name().equals(name))) {
            return;
        }
        Map<String, String> input = orderId == null ? Map.of() : Map.of("order_id", orderId);
        lastId++;
        content.add(new ToolUse("call_" + lastId, name, input));
    }


    // A real model retells them; this one only glues the results.
    private static String afterResults(List<Block> results) {
        List<String> parts = new ArrayList<>();
        for (Block b : results) {
            ToolResult r = (ToolResult) b;
            if (r.isError()) {
                return r.content().contains("order_id") ? ASK_ID : NOT_FOUND;
            }
            parts.add(r.content());
        }
        return "I checked: " + String.join("; ", parts) + ".";
    }


    static String words(Block block) {
        return switch (block) {
            case TextBlock t -> t.text();
            case ToolUse u -> u.name() + " " + String.join(" ", u.input().values());
            case ToolResult r -> r.content();
        };
    }


    // Roughly: words × 1.3, rounded up.
    static int tokens(String text) {
        String t = text.strip();
        int words = t.isEmpty() ? 0 : t.split("\\s+").length;
        return (words * 13 + 9) / 10;
    }
}
The records inside. sealed interface Block permits … tells the compiler that there are exactly three blocks. That is why the switch in words has no default: if you add a fourth kind of block, the code will not compile until you handle it. instanceof ToolUse use checks the kind and gives the name use in one go.

Exercise

The return deadline

Solve it in your editor.

The customer asks: “When does 1042 arrive and until when can I return 1187?” Now the model has only get_order_status and answers half of it:

Output
question: When does 1042 arrive and until when can I return 1187?
  call 1 (~49 tokens): tool_use
    call_1 get_order_status(1042) → 1042 — shipped, delivery 3 November
  call 2 (~60 tokens): end_turn
  answer: I checked: 1042 — shipped, delivery 3 November.

Add a second tool get_return_window with the description “The last day an order can be returned, by its number.” and the input order_id. In runTool return 1187 — return by … with the order’s return field. The model will ask for both tools in one call:

Output
question: When does 1042 arrive and until when can I return 1187?
  call 1 (~65 tokens): tool_use
    call_1 get_order_status(1042) → 1042 — shipped, delivery 3 November
    call_2 get_return_window(1187) → 1187 — return by 9 November
  call 2 (~87 tokens): end_turn
  answer: I checked: 1042 — shipped, delivery 3 November; 1187 — return by 9 November.

More tasks

  • Someone else’s order. Add the customer it belongs to to each order, and the customer who asks to answer. runTool returns an error for another customer’s order. What does the model say then?
  • A token budget. Stop the loop also when the sum of the input tokens passes 100. On which call does the question “around 1050” stop?

Test

Test yourself

1. The model answered with stop_reason tool_use and a get_order_status block with order_id 1042. Who looks the order up in the database?
2. One answer has two tool_use blocks. How does the model know which result belongs to which request?
3. The customer asks about two things and the model asks for both tools in one answer. How many calls to the model does answer() make?
4. Why does the loop have a limit, MAX_CALLS?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·