Skip to content
wannacode

Calling a model through an API

In short: Calling a language model through an API is a request with a model, a system prompt and messages with roles; the response carries the text, the stop reason and the token count.

Problem

The assistant goes in front of customers

So far the models were small and lived inside the program. The support assistant has to answer real customers, and that takes a large model at a provider. The program asks it through an API: it sends a request over the network and waits for an answer.

On the first day of the sale customers see "Error 429" instead of an answer. Another customer is asked by the assistant for the order number he has just given.

What does a request look like, what does the model return, and what do you do when the answer is an error?

The idea

Request and response

Calling a model is an HTTP request with JSON. The request carries:

  • model: which model;
  • max_tokens: at most how many tokens the answer may be;
  • system: the system prompt, the instruction for the role and the tone;
  • messages: the messages with their roles, user (the customer) and assistant (the model's earlier answers).

The response carries:

  • content: a list of blocks with a type; the text is in a text block;
  • stop_reason: why the model stopped, end_turn (it finished) or max_tokens (it hit the ceiling);
  • usage: how many tokens went in (input_tokens) and how many came out (output_tokens). You pay by them.

The API has no memory. Every request stands on its own and carries everything the model has to see, as in the lesson on the context window. The APIs of different providers have the same shape: messages with roles, tokens, tools.

See it

A fake client with recorded answers

The programs in the lesson do not call a real model: their check has no network, and the real answer is different every time (the lesson on limits). So instead of the provider's client there is a stand-in with the same shape: it takes model, max_tokens, system and messages and returns content, stop_reason and usage. The answers are recorded in advance and picked by a word in the last message. The tokens are counted roughly, 4 per 3 words.

record Message(string Role, string Content);


record Block(string Type, string Text);


record Usage(int InputTokens, int OutputTokens);


record Response(List<Block> Content, string StopReason, Usage Usage);


class ApiException(int status, string message) : Exception(message)
{
    public int Status { get; } = status;
}
// a stand-in for the real client: recorded answers, no network
class FakeClient(List<int> script)
{
    readonly Queue<int> failures = new(script);


    public Response Create(string model, int maxTokens, string system,
                           List<Message> messages)
    {
        if (failures.Count > 0)
        {
            var status = failures.Dequeue();
            throw new ApiException(status, Reason(status));
        }
        var text = Answer(messages);
        var stopReason = "end_turn";
        if (Tokens(text) > maxTokens)
        {
            // max_tokens: the answer stops in the middle
            var words = text.Split(' ');
            text = string.Join(" ", words.Take(maxTokens * 3 / 4));
            stopReason = "max_tokens";
        }
        var input = Tokens(system) + messages.Sum(m => Tokens(m.Content));
        var usage = new Usage(input, Tokens(text));
        return new Response([new Block("text", text)], stopReason, usage);
    }


    // the recorded answer: picked by a word in the last message
    static string Answer(List<Message> messages)
    {
        var last = messages[^1].Content;
        var hasOrder = messages.Any(m => m.Content.Contains("1042"));
        if (last.Contains("return")) return "You can return the item within 14 days of delivery.";
        if (last.Contains("arrive"))
        {
            return hasOrder ? "Order 1042 is with the courier and arrives tomorrow by 6 pm." : "What is your order number?";
        }
        return "How can I help: order status, delivery or returns?";
    }


    // the error text for its code
    static string Reason(int status) => status switch
    {
        400 => "bad request",
        401 => "invalid key",
        429 => "too many requests",
        _ => "server error",
    };


    // rough: 4 tokens per 3 words (the tokenizer gives the real count)
    public static int Tokens(string text)
    {
        var words = text.Split(' ', StringSplitOptions.RemoveEmptyEntries).Length;
        return (words * 4 + 2) / 3;
    }
}

One call, then the same with max_tokens 8:

// the text blocks, then why it stopped and how many tokens
static void Show(Response response)
{
    foreach (var block in response.Content)
    {
        if (block.Type == "text") Console.WriteLine(block.Text);
    }
    var u = response.Usage;
    Console.WriteLine($"stop_reason: {response.StopReason}, input_tokens: {u.InputTokens}, output_tokens: {u.OutputTokens}");
}
var client = new FakeClient([]);
var system = "You are a support assistant for an online shop. Answer briefly.";
List<Message> messages = [new Message("user", "When will order 1042 arrive?")];


var response = client.Create(Model, 1024, system, messages);
Show(response);


response = client.Create(Model, 8, system, messages);
Show(response);
Output
Order 1042 is with the courier and arrives tomorrow by 6 pm.
stop_reason: end_turn, input_tokens: 22, output_tokens: 16
Order 1042 is with the courier
stop_reason: max_tokens, input_tokens: 22, output_tokens: 8

With 8 tokens the answer stops in the middle of the sentence, and stop_reason is max_tokens. The program has to check it: a cut answer shown to a customer looks like a mistake of the shop.

See it

Run it at your place: the real call

The same call to a real model. It is not one of the checked programs and does not run here: you need an account at a provider, a key in the environment, and every call costs money.

What goes over the network is JSON. The request:

request
{
  "model": "<model name>",
  "max_tokens": 1024,
  "system": "You are a support assistant for an online shop. Answer briefly.",
  "messages": [
    {
      "role": "user",
      "content": "When will order 1042 arrive?"
    }
  ]
}

And the response (the numbers here are from the fake client):

response
{
  "content": [
    {
      "type": "text",
      "text": "Order 1042 is with the courier and arrives tomorrow by 6 pm."
    }
  ],
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 22,
    "output_tokens": 16
  }
}

With the standard HTTP client of the language this is one POST request. The address, the model name and the header with the key are placeholders: the real ones are in the provider's documentation. The key is read from the variable LLM_API_KEY.

using System.Net.Http.Json;


var key = Environment.GetEnvironmentVariable("LLM_API_KEY");
using var http = new HttpClient();
using var request = new HttpRequestMessage(HttpMethod.Post, "https://<provider>/messages");
request.Headers.Add("Authorization", $"Bearer {key}");
request.Content = JsonContent.Create(new
{
    model = "<model name>",
    max_tokens = 1024,
    system = "You are a support assistant for an online shop. Answer briefly.",
    messages = new[] { new { role = "user", content = "When will order 1042 arrive?" } },
});
var response = await http.SendAsync(request);
Console.WriteLine((int)response.StatusCode);
Console.WriteLine(await response.Content.ReadAsStringAsync()); // JSON: content, stop_reason, usage

Providers also ship official libraries (SDKs) for Java, C#, Python and JavaScript with the same shape: messages with roles, tokens, tools. The exact class and method names are in their documentation. await: the call goes over the network and is asynchronous.

The idea

The key is in the environment, not in the code

The API key is like a password for the account: whoever has it spends from it. In the code it reaches git, every colleague and sometimes a public repository. That is why it is read from an environment variable, here LLM_API_KEY.

var key = Environment.GetEnvironmentVariable("LLM_API_KEY");
if (string.IsNullOrWhiteSpace(key))
{
    Console.WriteLine("No key: set LLM_API_KEY in the environment.");
}
else
{
    Console.WriteLine("The key is loaded from the environment. Do not print it or write it in the code.");
}
Output
No key: set LLM_API_KEY in the environment.

The program says whether the key is there, but never prints it. The file with local secrets (for example .env) is in .gitignore.

How secrets are kept in a real project is in the lesson on secrets.

See it

An API with no memory

ask adds the question to the history, sends the whole history and keeps the answer with the role assistant. At the end the same question, sent alone:

// adds the question, sends the whole history and keeps the answer
static string Ask(FakeClient client, string system,
                  List<Message> history, string text)
{
    history.Add(new Message("user", text));
    var r = client.Create(Model, 1024, system, history);
    var reply = r.Content[0].Text;
    history.Add(new Message("assistant", reply));
    return reply;
}
var client = new FakeClient([]);
var system = "You are a support assistant for an online shop. Answer briefly.";
var history = new List<Message>();
foreach (var text in new[] { "Hello, I have order 1042.", "When will it arrive?" })
{
    Console.WriteLine($"customer: {text}");
    Console.WriteLine($"assistant: {Ask(client, system, history, text)}");
}


var alone = client.Create(Model, 1024, system,
    [new Message("user", "When will it arrive?")]);
Console.WriteLine($"the question alone, without the history: {alone.Content[0].Text}");
Output
customer: Hello, I have order 1042.
assistant: How can I help: order status, delivery or returns?
customer: When will it arrive?
assistant: Order 1042 is with the courier and arrives tomorrow by 6 pm.
the question alone, without the history: What is your order number?

With the history the model sees number 1042 from the first message. Without it, it asks for it. The memory of the conversation is in the program, not in the model, and the history grows and is paid for on every turn.

Try it

Find the bug

A colleague writes ask. On the second question the assistant asks for the order number again. Which line has the bug?

Pick the line with the bug.

Try it

Fill in

With which role does the model's answer go into the history?

static string Ask(FakeClient client, string system,
                  List<Message> history, string text)
{
    history.Add(new Message("user", text));
    var r = client.Create(Model, 1024, system, history);
    var reply = r.Content[0].Text;
    history.Add(new Message(, reply));
    return reply;
}

The idea

The errors of the API

The network and the provider's server fail like any outside service. The errors come with an HTTP code and fall into two groups:

  • Temporary: 429 (too many requests, a limit reached), 500 and the other 5xx (the server has a problem or is overloaded), 408 and 409. A moment later the same request may pass. These are worth retrying.
  • Permanent: 400 (bad request), 401 (invalid key), 404. A retry gives the same. The program reports these at once.

A retry waits between attempts, and the pause doubles: 1, 2, 4 seconds. A short problem passes quickly, and an overloaded server does not get a stream of new requests. This is exponential backoff. After a set number of attempts the program gives up.

See it

Retries with a pause that doubles

// retries the temporary errors, the pause doubles
static Response CreateWithRetry(FakeClient client,
    string system, List<Message> messages)
{
    var maxAttempts = 4;
    var delay = 1;
    for (var attempt = 1; ; attempt++)
    {
        try
        {
            return client.Create(Model, 1024,
                system, messages);
        }
        catch (ApiException e)
        {
            var last = attempt == maxAttempts;
            if (last || !Retryable(e.Status)) throw;
            Console.WriteLine($"  attempt {attempt}: " +
                $"{e.Status}, waiting {delay} s");
            // a real program waits here:
            // Thread.Sleep(delay * 1000)
            delay *= 2;
        }
    }
}


// temporary: 408, 409, 429 and 5xx
static bool Retryable(int status) =>
    status == 408 || status == 409
    || status == 429 || status >= 500;
Step by step, in words
  1. The request: the model, the system prompt and the messages with their roles. Here there is one, from the customer (user).
  2. The first attempt returns 429: "too many requests". A temporary error.
  3. 429 is in the temporary list, so the program waits 1 s and tries again.
  4. The second attempt: 500, "server error". Also temporary.
  5. The pause doubles: 2 s. An overloaded server gets room to breathe.
  6. The third attempt goes through.
  7. The answer: a text block, stop_reason end_turn and usage: 22 input and 16 output tokens.
  8. 401 and 400 are not retried: waiting will not fix a wrong key. After 4 attempts the program gives up.
// retries the temporary errors, the pause doubles
static Response CreateWithRetry(FakeClient client,
    string system, List<Message> messages)
{
    var maxAttempts = 4;
    var delay = 1;
    for (var attempt = 1; ; attempt++)
    {
        try
        {
            return client.Create(Model, 1024,
                system, messages);
        }
        catch (ApiException e)
        {
            var last = attempt == maxAttempts;
            if (last || !Retryable(e.Status)) throw;
            Console.WriteLine($"  attempt {attempt}: " +
                $"{e.Status}, waiting {delay} s");
            // a real program waits here:
            // Thread.Sleep(delay * 1000)
            delay *= 2;
        }
    }
}


// temporary: 408, 409, 429 and 5xx
static bool Retryable(int status) =>
    status == 408 || status == 409
    || status == 429 || status >= 500;

Three scenarios: 429 and 500, then an answer; 401, no retry; four times 429, giving up after the fourth attempt. The pause is only printed, so that the program runs at once.

var system = "You are a support assistant for an online shop. Answer briefly.";
List<Message> messages = [new Message("user", "When will order 1042 arrive?")];
List<List<int>> scenarios = [[429, 500], [401], [429, 429, 429, 429]];
foreach (var failures in scenarios)
{
    Console.WriteLine($"scenario [{string.Join(", ", failures)}]:");
    try
    {
        var r = CreateWithRetry(new FakeClient(failures), system, messages);
        Console.WriteLine($"  answer: {r.Content[0].Text}");
    }
    catch (ApiException e)
    {
        Console.WriteLine($"  gave up: {e.Status} {e.Message}");
    }
}
Output
scenario [429, 500]:
  attempt 1: 429, waiting 1 s
  attempt 2: 500, waiting 2 s
  answer: Order 1042 is with the courier and arrives tomorrow by 6 pm.
scenario [401]:
  gave up: 401 invalid key
scenario [429, 429, 429, 429]:
  attempt 1: 429, waiting 1 s
  attempt 2: 429, waiting 2 s
  attempt 3: 429, waiting 4 s
  gave up: 429 too many requests

The providers' official SDKs usually do this inside: they retry the temporary errors themselves, with a growing pause. Check the documentation and do not put your own loop on top: the attempts multiply. You write your own loop when you call the API directly over HTTP, as in the "run it at your place" block.

Timeouts, retries with jitter and the circuit breaker are in the lesson on resilience.

Try it

Predict the output

The fake client returns 429 first, then 400. system and messages are as above. What will the program print?

var client = new FakeClient([429, 400]);
try
{
    var r = CreateWithRetry(client, system, messages);
    Console.WriteLine($"answer: {r.Content[0].Text}");
}
catch (ApiException e)
{
    Console.WriteLine($"gave up: {e.Status}");
}
Answer

The idea

What the customer sees

Retries do not always help. When the last attempt fails too, the customer must not see "ApiException: 503". The program catches the error and says something polite and true: "I cannot answer right now. I will connect you with a colleague." The error goes into the log for the team.

More precisely, if you are curious

Jitter. When thousands of programs fail together and wait exactly 1, 2, 4 seconds, they try again together. A random part (jitter) is added to the pause to spread them out.

Timeout. The request may also hang, so every call over the network has a ceiling on the time for an answer. A long answer is slow, so max_tokens guards both time and money. More on slowness and cost is in the lesson on cost and latency.

Limits. 429 means you reached the limit of requests or tokens per minute. A retry helps at a peak. If you reach it all the time, the fix is fewer requests, shorter requests or a higher limit, not more retries.

What comes next. The system prompt is the first step towards the prompt as code. An answer the program reads, not only shows, is structured output, and a model that asks the program for data itself is tool use. Text from a user or a document that tries to change the instructions (prompt injection) is in the module on agents later in the course.

Exercise

An answer or a polite refusal

Solve it in your editor.

Write askSupport(client, system, question): it calls createWithRetry and returns the text of the answer, and if the last attempt fails too (or the error is permanent), it returns "I cannot answer right now. I will connect you with a colleague.". Run it with the question "Can I return the jacket?" and four scenarios of the fake client: no errors; four times 503; 401; one 429.

Output
scenario []:
  You can return the item within 14 days of delivery.
scenario [503, 503, 503, 503]:
  attempt 1: 503, waiting 1 s
  attempt 2: 503, waiting 2 s
  attempt 3: 503, waiting 4 s
  I cannot answer right now. I will connect you with a colleague.
scenario [401]:
  I cannot answer right now. I will connect you with a colleague.
scenario [429]:
  attempt 1: 429, waiting 1 s
  You can return the item within 14 days of delivery.

More tasks

  • A log for the team. On a refusal print a separate line with the code and the text of the error before the polite answer. The customer sees only the polite one.
  • The cut answer. Add a parameter for max_tokens to createWithRetry. If stop_reason is max_tokens, return the answer with "…" at the end and log a warning. Try it with 8.

Test

Check yourself

1. The customer writes "Hello, I have order 1042." and then "When will it arrive?". For the second request the program sends only the new question. What happens?
2. The response comes with stop_reason: max_tokens. What does it mean?
3. Which errors are worth retrying?
4. Why does the pause between attempts double (1, 2, 4 s)?
5. Where does the API key belong?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·