Skip to content
wannacode

Cost and latency: budget, cache, a smaller model

In short: The cost of a language model is tokens × price, separately for input and output; a token budget, an answer cache and a smaller model cut the bill, and the cache and the smaller model also cut the response time.

Problem

The bill at the end of the month

The support assistant has been working for a month. It answers correctly, and the set from the lesson on evaluation passes. Then two complaints arrive: the bill for the model is higher than expected, and customers wait about a second for an answer.

Here is one call as the assistant makes it. The token prices are examples, in euro per million tokens:

question = "How much does delivery cost?"
r = CLIENT.create(model=LARGE, max_tokens=1024, system=PROMPT + " " + ALL_DOCS, messages=[{"role": "user", "content": question}])
m = MODELS[LARGE]
euro = cost(m, r.usage)
input_euro = r.usage.input_tokens * m.in_price / 1_000_000
print(f"input: {r.usage.input_tokens} tokens, output: {r.usage.output_tokens} tokens")
print(f"cost: €{euro:.5f} (the input is {100 * input_euro / euro:.0f}%)")
print(f"example time: {latency(m, r.usage)} ms")
Output
input: 358 tokens, output: 28 tokens
cost: €0.00188 (the input is 76%)
example time: 995 ms

The question “How much does delivery cost?” is 10 tokens, and the input is 358. Part of the bill is the answer, and the larger part is the input: the system prompt and all the shop documents travel with every question. And in the log half the questions are the same: how much is delivery, when is support open.

The idea

Where the money and the time go

A model is paid in tokens (the lesson on tokens), separately for the input and for the output:

  • the input is everything you send: the system prompt, the documents, the history of the conversation, the question. The API has no memory, so all of it is sent every time (the lesson on context);
  • the output is the answer. An output token usually costs several times more than an input token, here 4 times.

Time is similar, with a different balance. The model reads the input fast, and writes the answer token by token. So a long answer is slow, and a long input is mostly expensive.

That gives the three fixes the lesson measures with a program:

  1. a token budget: less input (only what is needed) and a ceiling on the output;
  2. a cache: the same question does not reach the model;
  3. a smaller model for the easy questions: a cheaper and faster token.

A token budget is an upper limit on the tokens of one call, planned in advance, the way you plan the memory or the response time of one request.

See it

The assistant in code

The fake client is the same as in the lesson on evaluation: the shape of the real API, recorded answers and rough tokens (every word is 2 tokens). The answers are recorded by a word in the question:

# a fake client: the shape of the real API, the answers are recorded
@dataclass(frozen=True)
class Usage:
    input_tokens: int
    output_tokens: int




@dataclass(frozen=True)
class Response:
    text: str
    stop_reason: str
    usage: Usage




# a recorded answer: model ("" = any), a word in the question, text the prompt needs
@dataclass(frozen=True)
class Recorded:
    model: str
    keyword: str
    needs: str
    text: str




# roughly: every word is 2 tokens
def tokens(text):
    return 2 * len(text.split())




# max_tokens cuts the answer
def reply(text, max_tokens, input_tokens):
    words = text.split(" ")
    if 2 * len(words) <= max_tokens:
        return Response(text, "end_turn", Usage(input_tokens, 2 * len(words)))
    kept = max_tokens // 2
    return Response(" ".join(words[:kept]), "max_tokens", Usage(input_tokens, 2 * kept))




class FakeLlm:
    def __init__(self, recorded):
        self.recorded = recorded


    def create(self, model, max_tokens, system, messages):
        question = messages[-1]["content"]
        input_tokens = tokens(system) + sum(tokens(m["content"]) for m in messages)
        for r in self.recorded:
            if r.model in ("", model) and r.keyword in question and r.needs in system:
                return reply(r.text, max_tokens, input_tokens)
        return reply("(no recorded answer)", max_tokens, input_tokens)
RECORDED = [
    Recorded("", "cost", "", "Delivery to an address is €4.90, to an office €3.50, and free over €50."),
    Recorded("", "cash", "", "Yes, you can pay cash on delivery. The fee is €1.50."),
    Recorded("", "return", "", "You can return the goods within 14 days of delivery if they are unused and have the label."),
    Recorded("", "open", "", "Support is open Monday to Friday, 9 am to 6 pm."),
    Recorded("", "1042", "", "Order 1042 has shipped and arrives on Thursday."),
    Recorded("", "1077", "", "Order 1077 is being packed and leaves tomorrow."),
    Recorded("", "damaged", "", "I am sorry! Take a photo of the parcel and the label and write to us with the order number. We will send a new one or refund you within 5 working days."),
    Recorded("", "days", "", "Parcels arrive in 2 to 4 working days."),
    Recorded("", "wholesale", "", "I do not know, I will connect you with a colleague."),
    Recorded("", "Mastercard", "", "Yes, we take both Visa and Mastercard."),
]

The shop documents are five sections. Each has words that identify it:

# a section of the documents and the words that identify it
@dataclass(frozen=True)
class Doc:
    triggers: list
    text: str




PROMPT = "You are the support assistant of an online shop. Answer briefly and only from the documents. If the answer is not there, say: “I do not know, I will connect you with a colleague.”"
DOCS = [
    Doc(["delivery", "parcel"], "Delivery: parcels arrive in 2 to 4 working days. Delivery to an address costs €4.90, to a courier office €3.50, and it is free over €50. A parcel is tracked by its number."),
    Doc(["return", "damaged"], "Returns: goods can be returned within 14 days of delivery if they are unused and have the label. A damaged parcel is photographed and reported to support. The money goes back to the card within 5 working days."),
    Doc(["pay", "card", "cash"], "Payment: we take Visa and Mastercard. Cash on delivery has a fee of €1.50. Card payment is protected, and the amount is charged when the order is confirmed."),
    Doc(["order"], "Orders: the status of an order is shown in the profile. Support checks an order only by its number and never asks for card details."),
    Doc(["open", "hours"], "Support: we work Monday to Friday, 9 am to 6 pm. Write to us at help@shop.example."),
]
ALL_DOCS = " ".join(doc.text for doc in DOCS)

The models, the prices and the times. All the numbers here are examples, and the time comes from a formula, it is not measured:

# example prices in € per 1,000,000 tokens and example time (not measured)
@dataclass(frozen=True)
class ModelInfo:
    in_price: float
    out_price: float
    base_ms: int
    input_per_ms: int
    ms_per_token: int




LARGE, SMALL = "model-large", "model-small"
MODELS = {
    LARGE: ModelInfo(4.00, 16.00, 400, 10, 20),
    SMALL: ModelInfo(0.40, 1.60, 150, 40, 5),
}
CACHE_MS = 5
# the cost of one call in €
def cost(m, u):
    return (u.input_tokens * m.in_price + u.output_tokens * m.out_price) / 1_000_000




# example time in ms: start + reading the input + generating
def latency(m, u):
    return m.base_ms + u.input_tokens // m.input_per_ms + u.output_tokens * m.ms_per_token

The answer for order 1042 is recorded. In the real assistant the status comes from a tool (the lesson on tools).

1_000_000 is the same as 1000000. // is integer division, like / between integers in Java and C#.

Try it

Fill in

The cost is the input tokens times the input price plus the output tokens times…?

return (u.input_tokens * m.in_price + u.output_tokens * ) / 1_000_000

See it

One day

A day in support is a sample of 20 questions. Some repeat, sometimes in lower case or with an extra space. Three are about a specific order:

questions = [
    "How much does delivery cost?",
    "When is support open?",
    "Where is order 1042?",
    "how much does delivery cost?",
    "Can I pay cash on delivery?",
    "Until when can I return the goods?",
    "How much does delivery cost?",
    "My parcel came damaged, what do I do?",
    "When is support open?",
    "Where is order 1077?",
    "Do you take Mastercard?",
    "How much does  delivery cost? ",
    "How many days does a parcel to Burgas take?",
    "Can I pay cash on delivery?",
    "Do you sell wholesale?",
    "Until when can I return the goods?",
    "Where is order 1042?",
    "When is support open?",
    "How much does delivery to an office cost?",
    "How do I return the goods?",
]

day runs the questions and adds up the cost and the time. The three flags switch on the three fixes, and for now they are off:

def day(questions, budget, cache, route):
    answers = {}
    euro, ms, hits, on_small = 0.0, 0, 0, 0
    for question in questions:
        k = key(question)
        # a hit: no model
        if cache and k in answers:
            hits += 1
            ms += CACHE_MS
            continue
        model = pick_model(question) if route else LARGE
        docs = relevant(question) if budget else ALL_DOCS
        max_tokens = 200 if budget else 1024
        msg = {"role": "user", "content": question}
        r = CLIENT.create(model=model,
                          max_tokens=max_tokens,
                          system=PROMPT + " " + docs,
                          messages=[msg])
        m = MODELS[model]
        euro += cost(m, r.usage)
        ms += latency(m, r.usage)
        if model == SMALL:
            on_small += 1
        # an order belongs to one customer: no cache
        if cache and "order" not in k:
            answers[k] = r.text
    return Totals(euro, ms, hits, on_small)
def row(name, t, n):
    print(f"{name}: €{t.euro / n * 1000:.2f} per 1000 questions, {t.ms // n} ms on average")




def month(t, n):
    print(f"a month at 3000 questions a day: €{t.euro / n * 3000 * 30:.2f}")
all = day(questions, False, False, False)
row("large model, all documents", all, len(questions))
month(all, len(questions))
Output
large model, all documents: €1.85 per 1000 questions, 959 ms on average
a month at 3000 questions a day: €166.93

The cost per 1000 questions is under €2. At 3000 questions a day the month is about €167, and a question waits almost a second on average.

The idea

A token budget

The first fix is the input. The question about delivery needs only the delivery section. relevant takes the sections whose words are in the question, and if there is none, it sends them all (the question about wholesale has to see everything to say “I do not know”):

# budget: only the sections whose words are in the question
def relevant(question):
    low = question.lower()
    parts = [doc.text for doc in DOCS if any(word in low for word in doc.triggers)]
    return " ".join(parts) if parts else ALL_DOCS

The second ceiling is max_tokens: at most 200 tokens for an answer. It does not make the answers shorter. It says how much one call can cost at most.

all = day(questions, False, False, False)
row("large model, all documents", all, len(questions))
budget = day(questions, True, False, False)
row("+ token budget", budget, len(questions))
Output
large model, all documents: €1.85 per 1000 questions, 959 ms on average
+ token budget: €1.05 per 1000 questions, 939 ms on average

The cost drops by almost half. The time barely moves: the input is read fast, and the answer is the same.

Here the sections are picked by words. How to pick them by meaning, with vectors, is in the module on semantic search (RAG).

any(word in low for word in doc.triggers) is true if at least one word of the section is in the question.

Try it

Predict the output

max_tokens is 10, and the question is about a damaged parcel. What will it print?

r = CLIENT.create(model=LARGE, max_tokens=10, system=PROMPT, messages=[{"role": "user", "content": "My parcel came damaged, what do I do?"}])
print(r.text)
print(f"{r.stop_reason}, {r.usage.output_tokens}")
Answer

The idea

An answer cache

The second fix is repetition. If the same question was asked a moment ago, the answer is the same: there is no need to ask the model again. This is cache-aside from the lesson "Caching": you look in the cache, and on a miss you ask the model and keep a copy.

The key is the question in lower case with one space each, so “how much does delivery cost? ” is the same as “How much does delivery cost?”:

# the cache key: lower case, one space each
def key(question):
    return " ".join(question.lower().split())

In day these are the lines with answers: a hit returns the stored answer in 5 ms and for free. A question about an order is not cached. The answer belongs to one customer, and the status changes.

all = day(questions, False, False, False)
row("large model, all documents", all, len(questions))
budget = day(questions, True, False, False)
row("+ token budget", budget, len(questions))
cached = day(questions, True, True, False)
row("+ cache", cached, len(questions))
print(f"from the cache: {cached.hits} of {len(questions)}")
Output
large model, all documents: €1.85 per 1000 questions, 959 ms on average
+ token budget: €1.05 per 1000 questions, 939 ms on average
+ cache: €0.70 per 1000 questions, 610 ms on average
from the cache: 7 of 20

7 of 20 questions come from the cache. Now the time drops too, because a hit does not wait for the model.

Try it

Find the bug

A colleague simplifies the cache. The cost tests pass, and a day later a customer complains that the assistant told them the status of someone else's order. Which line has the bug?

Pick the line with the bug.

The idea

A smaller model for the easy ones

The third fix is the model. The small model is 10 times cheaper per token and writes faster. In the lesson on evaluation the set showed where it does fine: the delivery price, payment, returns. And where it does not: the delivery time and the question outside the documents. So it gets only the approved topics:

# the easy topics: the test set showed the small model handles them
EASY = ["cost", "cash", "return", "Mastercard"]




def pick_model(question):
    return SMALL if any(word in question for word in EASY) else LARGE
all = day(questions, False, False, False)
row("large model, all documents", all, len(questions))
budget = day(questions, True, False, False)
row("+ token budget", budget, len(questions))
cached = day(questions, True, True, False)
row("+ cache", cached, len(questions))
routed = day(questions, True, True, True)
row("+ small model for the easy ones", routed, len(questions))
print(f"on the small model: {routed.on_small} of {len(questions)}")
month(all, len(questions))
month(routed, len(questions))
Output
large model, all documents: €1.85 per 1000 questions, 959 ms on average
+ token budget: €1.05 per 1000 questions, 939 ms on average
+ cache: €0.70 per 1000 questions, 610 ms on average
+ small model for the easy ones: €0.42 per 1000 questions, 408 ms on average
on the small model: 6 of 20
a month at 3000 questions a day: €166.93
a month at 3000 questions a day: €37.36

The month drops from about €167 to about €37, and the average time from 959 to 408 ms. The numbers come from example prices and an example formula, but the order of the fixes is real.

See it

The bill, fix by fix

def day(questions, budget, cache, route):
    answers = {}
    euro, ms, hits, on_small = 0.0, 0, 0, 0
    for question in questions:
        k = key(question)
        # a hit: no model
        if cache and k in answers:
            hits += 1
            ms += CACHE_MS
            continue
        model = pick_model(question) if route else LARGE
        docs = relevant(question) if budget else ALL_DOCS
        max_tokens = 200 if budget else 1024
        msg = {"role": "user", "content": question}
        r = CLIENT.create(model=model,
                          max_tokens=max_tokens,
                          system=PROMPT + " " + docs,
                          messages=[msg])
        m = MODELS[model]
        euro += cost(m, r.usage)
        ms += latency(m, r.usage)
        if model == SMALL:
            on_small += 1
        # an order belongs to one customer: no cache
        if cache and "order" not in k:
            answers[k] = r.text
    return Totals(euro, ms, hits, on_small)
Step by step, in words
  1. One call: the prompt, all the documents and the question go in (358 tokens), the answer comes out (28). The documents are most of the bill: €0.00188.
  2. The token budget: only the section the question asks about, and a ceiling on the answer. 144 tokens in, €0.00102 for the same answer.
  3. A day of 20 questions, per 1000 questions: €1.85 with all the documents, €1.05 with the budget. The time barely moves: it is mostly the answer.
  4. The cache: 7 of 20 questions are a repeat and do not reach the model. Questions about an order are never cached.
  5. The small model takes the easy topics the test set approved: 6 of 20 questions. It is cheaper and faster.
  6. From €1.85 to €0.42 per 1000 questions (78% less) and from 959 to 408 ms. The prices and times are examples.

The idea

Run it at your place

The real reply says how many tokens it used. Here is the request with the budget for the delivery question, and the reply. The text and the counts come from the fake client:

Request
{
  "model": "<model name>",
  "max_tokens": 200,
  "system": "<the system prompt>",
  "messages": [
    {
      "role": "user",
      "content": "How much does delivery cost?"
    }
  ]
}
Reply
{
  "content": [
    {
      "type": "text",
      "text": "Delivery to an address is €4.90, to an office €3.50, and free over €50."
    }
  ],
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 144,
    "output_tokens": 28
  }
}

The fields are usage.input_tokens and usage.output_tokens, and stop_reason says whether the answer is whole (end_turn) or the ceiling cut it (max_tokens). With them the program computes the same bill as here, only with real counts.

Providers ship official SDKs for Java, C#, Python and JavaScript with the same shape: messages, roles, tokens, tools. The exact class and method names are in the provider's documentation. Here the request goes through the language's standard HTTP client, without an SDK. The address and the model are placeholders:

import json
import os
import urllib.request


# display only: the address and the key header are in the provider docs
ENDPOINT = "https://<provider>/..."




def ask(system, question):
    body = {
        "model": "<model name>",
        "max_tokens": 200,
        "system": system,
        "messages": [{"role": "user", "content": question}],
    }
    request = urllib.request.Request(
        ENDPOINT,
        data=json.dumps(body).encode("utf-8"),
        headers={
            "content-type": "application/json",
            "authorization": "Bearer " + os.environ["LLM_API_KEY"],
        },
    )
    with urllib.request.urlopen(request) as response:
        return json.load(response)  # JSON: content, stop_reason, usage




reply = ask(PROMPT + " " + relevant(question), question)
usage = reply["usage"]
print(reply["stop_reason"], usage["input_tokens"], usage["output_tokens"])

This code does not run in the site's code check. The key is in the environment variable LLM_API_KEY, not in the code (the lesson "Secrets in code and in git"), you need an account with a provider, and every run costs money.

urllib.request and json are in the standard library, and json.dumps turns the dictionary into JSON.

The idea

When it hurts

  • A budget that cuts what is needed. If relevant misses the section with the answer, the model says “I do not know” or, worse, invents. A max_tokens that is too low gives half an answer. Every change to the budget goes through the test set.
  • A cache that lies. The delivery price changes, and the cache gives the old one. A cache has an expiry (TTL) and is cleared when the documents change. Answers that depend on the customer or on the time do not go into it.
  • A small model without a set. “It looks simple” is not an argument. The small model takes a topic only after the set has passed it, and the set runs again on every change.
  • Saving that is not worth it. If the assistant gets 50 questions a day, the bill is a few euro a month. The complexity of the cache and the router may cost more than that. Measure first.
More precisely, if you are curious

Caching the start of the prompt. Many providers keep the repeated start of the prompt (the system prompt and the documents) and charge less for reading it again. This is a different cache: the model still answers, but the input is cheaper. It works if the start is identical to the last character, so the changing parts (the question, the customer’s name) go at the end.

A cache by meaning. “How do I return the goods?” and “Until when can I return the goods?” are different keys. A cache with vectors (the lesson on vectors) treats them as one question. It finds more hits, and wrong ones too: “Can I return shoes?” and “Can I return a gift?” are close, and the answers may differ.

A streamed answer. The answer can arrive word by word (streaming). The total time is the same, but the customer sees the first words after about a hundred ms instead of after a second. This is speed for the person, not for the bill.

Batches. Work that no customer waits for (summarising 10,000 reviews overnight) can go through a batch mode: slower and cheaper.

Time in production. The average hides the tail: 95% of the answers may be under a second, and 5% over 5 seconds. Measure percentiles, as in the lesson "Non-functional requirements you can measure" from “Software engineering”. Watching cost and time in production is part of MLOps and is taught in ZeroToOps.

" ".join(question.lower().split()) splits on any whitespace and glues with one space.

Exercise

Which fix saves the most

Solve it in your editor.

The program applies the fixes cumulatively, so you cannot see how much each gives alone. Run day three times, each time with only one flag, and print the rows next to the one without fixes:

Output
large model, all documents: €1.85 per 1000 questions, 959 ms on average
budget only: €1.05 per 1000 questions, 939 ms on average
cache only: €1.20 per 1000 questions, 622 ms on average
small model only: €0.92 per 1000 questions, 577 ms on average

On its own the small model is the cheapest, and the budget comes close after it. The budget hardly saves time, and the cache and the small model cut it the most, because they do not wait for the large model to write. The three save in different places, so together they give the most.

More tasks

  • One more easy topic. Add “open” to the easy topics. How much does the cost drop? What do you have to do before you ship it? (Hint: the lesson on evaluation.)
  • A ceiling for the day. Stop asking the large model once the cost for the day passes €0.01, and answer “I will connect you with a colleague”. How many questions are left without an answer from the model?

Test

Check yourself

1. The question is 10 tokens, and the bill for the call is for 358 input tokens. Where do the rest come from?
2. The token budget cut the cost by almost half, and the average time barely moved. Why?
3. Which question must not go into the answer cache?
4. How do you choose which questions the small model takes?
5. max_tokens is 10, and the answer needs 30 tokens. What do you get?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·