The context window in Python

Problem

The assistant forgot the number

The support assistant is talking to a customer. In the first message the customer gives the order number: 1042. Then they ask about a courier office, a discount code, swapping a jacket. On the fifth question ("when exactly will the parcel come?") the assistant answers: "What is your order number?"

The customer is annoyed: they said it. And at the end of the month the bill for the model is bigger than the number of conversations explains.

Both problems have one cause. It is not in the model but in what the program sends to it.

The idea

The model has no memory

A language model remembers nothing between two requests. It gets text, writes a continuation and forgets. So the "memory" of the conversation lives in your program: on every turn it sends the system prompt and the whole history again, all messages so far, plus the new question.

The model sees only what is in the request, and only up to a certain number of tokens. That limit is called the context window. Everything goes into it: the system prompt, the history, the new question and the answer the model is about to write.

How big the window is depends on the model. As an example: from a few tens of thousands to about a million tokens. It sounds like a lot, but tokens run out faster than words, and every token in the request is paid for.

See it

The conversation in code

A message is a dictionary with a role (user is the customer, assistant is the model) and text, exactly as the real SDKs want it:

Tokens are counted roughly so that the numbers are visible. The model's tokenizer gives the real count. The request is the system prompt plus every message:

# rough: 4 tokens per 3 words (the tokenizer gives the real count)
def tokens(text):
    words = len(text.split())
    return (words * 4 + 2) // 3


# the request: the system prompt and the whole history
def request_tokens(system, history):
    total = tokens(system)
    for message in history:
        total += tokens(message["content"])
    return total

The conversation from the problem: five questions from the customer and four answers.

system = "You are the support assistant of the shop. Answer briefly and politely."
customer = [
    "Hello, order 1042 has not arrived. I ordered it five days ago.",
    "Can you leave it at a courier office instead of my address?",
    "I have a discount code that did not work at checkout. What now?",
    "Thanks. The jacket from the same order is too small. Can I swap it?",
    "Last thing: when exactly will the parcel come?",
]
assistant = [
    "Sorry for the delay. Order 1042 is with the courier, arriving tomorrow by 18:00.",
    "Yes, the parcel will wait at the courier office on Vitosha 12.",
    "The code works above 50 euro. We refund the difference in 5 days.",
    "Yes, swaps are free within 14 days. Write to us once it arrives.",
]

On every turn the question goes into the history and the whole history is sent. The price is an example: €3 per million input tokens.

history = []
total = 0
print("turn request total")
for i, text in enumerate(customer):
    history.append({"role": "user", "content": text})
    sent = request_tokens(system, history)
    total += sent
    print(f"{i + 1:3d} {sent:7d} {total:6d}")
    if i < len(assistant):
        history.append({"role": "assistant", "content": assistant[i]})
price = 3.0  # € per million input tokens (an example price)
one = total * price / 1_000_000
print(f"one conversation: {one:.4f} €, 20000 conversations: {one * 20000:.2f} €")

turn request total
  1      32     32
  2      67     99
  3     101    200
  4     138    338
  5     167    505
one conversation: 0.0015 €, 20000 conversations: 30.30 €

The fifth request is 167 tokens, although the new question is only 11. The whole conversation is 167 tokens, but 505 were sent: the first message was sent five times. €0.0015 per conversation is not much. But the cost grows with every turn, not only with every new conversation, and 20,000 conversations a month are already €30.30.

The idea

When the window is full

The conversation grows, the window does not. When the request does not fit, the provider rejects it with an error or cuts it itself. It is better that your program decides what falls out.

The most common rule: always keep the system prompt and the newest messages, as many as fit. The oldest fall out. The function goes from the newest message back and stops at the first one that does not fit:

# from the newest back, while it fits
def fit_window(history, budget):
    kept = []
    used = 0
    for message in reversed(history):
        t = tokens(message["content"])
        if used + t > budget:
            break
        kept.insert(0, message)
        used += t
    return kept

The window in the lesson is 150 tokens, small on purpose so it fits on a screen. The program cuts the conversation from the problem twice: once with the rule only and once with a note the program keeps separately, next to the system prompt.

history = []
for i, text in enumerate(customer):
    history.append({"role": "user", "content": text})
    if i < len(assistant):
        history.append({"role": "assistant", "content": assistant[i]})

limit = 150
print(f"the request: {request_tokens(system, history)} tokens, the window: {limit}")

window = fit_window(history, limit - tokens(system))
used = request_tokens(system, window)
seen = any("1042" in m["content"] for m in window)
print(f"without a note: {len(window)} of {len(history)} messages, {used} tokens")
print(f"  1042 in the window: {'yes' if seen else 'no'}")

note = "Note: order 1042."
window = fit_window(history, limit - tokens(system) - tokens(note))
used = request_tokens(system, window) + tokens(note)
seen = "1042" in note or any("1042" in m["content"] for m in window)
print(f"with a note: {len(window)} of {len(history)} messages, {used} tokens")
print(f"  1042 in the window: {'yes' if seen else 'no'}")

the request: 167 tokens, the window: 150
without a note: 7 of 9 messages, 132 tokens
  1042 in the window: no
with a note: 7 of 9 messages, 136 tokens
  1042 in the window: yes

Without the note the two oldest messages fall out, and with them the number 1042. The model cannot use text it does not see, so it asks again. The note is 4 tokens and brings it back.

See it

The window fills up

# from the newest back, while it fits
def fit_window(history, budget):
    kept = []
    used = 0
    for message in reversed(history):
        t = tokens(message["content"])
        if used + t > budget:
            break
        kept.insert(0, message)
        used += t
    return kept

Step by step, in words

The window holds 150 tokens. The system prompt is always first: 16 tokens.
Turn 1: the customer gives order number 1042. The request is 32 tokens.
Every request carries the whole history: 67, 101, 138 tokens. Sent so far: 338.
Turn 5: the request is 167 tokens and the window is 150. It does not fit.
fitWindow goes from the newest back and stops at the first message that does not fit. The two oldest fall out: 132 tokens are left.
Order number 1042 was in them. The model no longer sees it and will ask for it again.
The fix: the program keeps a note "order 1042" next to the system prompt. 136 tokens, and the number is visible.
The conversation is 167 tokens, but 505 were sent: the history is paid for again on every turn.

Whole messages fall out, from the oldest. The system prompt and the note never fall out: the program puts them in separately, before the history.

Try it

Find the bug

A colleague writes fitWindow themselves. When the conversation gets longer, the assistant starts answering questions from ten minutes ago and seems not to hear the last question. Which line has the bug?

Pick the line with the bug.

Try it

Predict the output

Four short messages and a budget of 25 tokens. The counting is the same: 4 tokens per 3 words, rounded up. What will the program print?

history = [
    {"role": "user", "content": "I ordered two jackets and a cap last Friday morning."},
    {"role": "user", "content": "Only one jacket arrived."},
    {"role": "assistant", "content": "The cap and the second jacket ship in another parcel."},
    {"role": "user", "content": "When then?"},
]
window = fit_window(history, 25)
print(len(window), window[0]["content"])

Answer

2 I ordered two jackets and a cap last Friday morning.

2 The cap and the second jacket ship in another parcel.

3 Only one jacket arrived.

Try it

Fill in

Every request carries the system prompt. Finish the counting.

def request_tokens(system, history):
    total = 
    for message in history:
        total += tokens(message["content"])
    return total

The idea

What to keep and when a long context hurts

Ways to fit a conversation into the window:

The last n messages (a sliding window). The simplest and the most common. It loses the start.
A note with the facts the program keeps itself: an order number, an address, what has already been promised. Cheap and exact, but you have to know which facts matter.
A summary of the old turns, written by the model. It keeps more, but costs another request and may drop a detail.
Search instead of everything. The shop's documents do not go into the request in full: the program finds the pieces that answer the question. That is semantic search, the basis of RAG, and it has its own module later in the course.

When a long context hurts:

Cost and slowness. Every token in the request is paid for and read on every turn. A history twice as long is a request twice as expensive and a slower answer.
Lost in the middle. Models use the start and the end of a long context better than the middle. A fact buried in the middle of 50 pages may be missed.
The window is not a database. Whatever must be remembered tomorrow (orders, preferences) is kept in the shop's database. Only what this answer needs goes into the request.
Foreign text in the context. Everything in the window affects the answer, including text you did not write. Why that is dangerous is the lesson on prompt injection in the module on agents.

More precisely, if you are curious

Why what you send grows so fast. Let the system prompt be S tokens and let every turn add m tokens of history. The request on turn k is about S + k·m. The sum over n turns is n·S + m·(1 + 2 + … + n) = n·S + m·n·(n + 1) / 2. In words: a conversation twice as long costs about four times more tokens sent. That is why long chats are cut or summarised.

The answer is in the window too. If the window is 200,000 tokens (an example) and the request is 199,000, 1,000 are left for the answer. That is why the request has a limit for the answer (max_tokens, in the lesson on the API), and it has to fit together with the history.

The bigram model has a window of one token. The model from the lesson on the next token looks only at the last word. The number 1042 from the start of the sentence never reaches it. Large models look back thousands of tokens, but still only up to a limit.

Why the window is limited. Attention, the idea from the lesson on vectors, compares every token with every token before it. With twice as many tokens there are about four times as many comparisons. That is why a long context is expensive for the provider and slow for you.

Prompt caching. Providers can keep the repeated start of a request (the system prompt, the documents) and charge less for it. How to use it is in the lesson on cost and latency.

Real counters. Here tokens are words × 4 / 3. A real tokenizer gives about 4 characters per token in English, and often two or three times more tokens for the same meaning in Bulgarian. The API's answer tells you the exact count (usage.input_tokens).

Exercise

A long conversation

Solve it in your editor.

The assistant holds a conversation of 20 turns. The system prompt is 200 tokens, every customer question is 32 tokens, every answer is 45. Write a program that works out the request on every turn (the system prompt, all past questions and answers and the new question) and the total sent. It prints every fifth turn and at the end the cost at €3 per million input tokens (an example) for one conversation and for 10,000.

turn  5: request  540, total  1930
turn 10: request  925, total  5785
turn 15: request 1310, total 11565
turn 20: request 1695, total 19270
one conversation: 0.0578 €, 10000 conversations: 578.10 €

The twentieth request is 1,695 tokens, and 19,270 are sent in total, more than eleven times the last one.

More tasks

A window of 1,000 tokens. When the request goes over 1,000, drop the oldest turn (question and answer). How many are sent in total now?
A note instead of the history. Instead of the whole history send the system prompt, a note of 60 tokens and the last 2 turns. Compare the cost.

Test

Check yourself

1. The customer writes a second question to the assistant. What does the program send to the model?Only the new question: the model remembers the previous oneOnly the history, without the system promptThe system prompt, the whole history so far and the new question

2. What is the context window?The most tokens the model sees at once: prompt, history and answerThe chat window in the browserHow many messages the model remembers from last week

3. The conversation is 5 turns and the last request is 167 tokens. Why are 505 sent in total?The model answers are counted three timesThe system prompt is 505 tokensEvery request carries all earlier messages, so they are paid for again on every turn

4. fitWindow cuts the history. Why does it go from the newest message back?So that the question you are answering stays, with the last turns before itBecause it is faster that wayBecause the newest messages are the shortest

5. Which one does NOT help when an important fact from the start of the conversation falls out of the window?A note with the fact that the program keeps next to the system promptA short summary of the old turns instead of the turns themselvesA higher temperature

The context window

Cookies