Skip to content
wannacode

The limits of the model

In short: A language model hallucinates: it confidently writes plausible but false things, because it predicts likely text instead of checking the truth; so the answer is checked against the source.

Problem

Thirty days that do not exist

The support assistant answers a customer: "You can return the jacket within 30 days of delivery." The answer is polite, smooth and confident. The shop's policy is 14 days.

On day twenty the customer sends the jacket back. The warehouse returns it, and the customer writes a one-star review quoting the assistant.

Nobody programmed "30 days". Where did the number come from, and how do you guard against the next one?

The idea

The model writes likely text, not checked facts

A language model is trained to continue text with a likely next word. It keeps no table of facts and does not check its answer. If in the text it learned from "within" is often followed by a number, it will put a number there. Which number is decided by probability, not by the shop's policy.

A smooth, confident answer that is wrong or made up is called a hallucination. It is not a bug that one patch will fix. It comes from the very way the model writes: for it, the true and the false sentence are just more likely and less likely text.

See it

A small model that promises the wrong thing

The model from the lesson on the next token, in its smallest form: for every word it keeps the words that followed it in the text. The repeats stay, so a word that comes more often is drawn more often.

# for every word: the words that followed it in the text
def train(corpus):
    follows = {}
    for sentence in corpus:
        words = (sentence + " .").split(" ")
        for word, after in zip(words, words[1:]):
            follows.setdefault(word, []).append(after)
    return follows

The text is ten sentences from the shop's support:

corpus = [
    "a return is within 14 days of delivery",
    "delivery to sofia is within 2 days",
    "delivery to plovdiv is within 2 days",
    "delivery to varna is within 3 days",
    "refunds are paid within 5 working days",
    "the warranty is 24 months",
    "card payment is safe",
    "the courier calls before delivery",
    "an order over 50 euro is shipped for free",
    "a refused parcel goes back within 7 days",
]

The most likely word every time:

# the most common one (on a tie, the first in the text)
def most_common(options):
    return max(options, key=options.count)




def greedy(follows, start):
    words = start.split(" ")
    while len(words) < 12:
        last = words[-1]
        word = most_common(follows[last])
        if word == ".":
            break
        words.append(word)
    return " ".join(words)

And picked by probability, with a seeded generator so the output is the same on every run:

def sample(follows, start, rng):
    words = start.split(" ")
    while len(words) < 12:
        options = follows[words[-1]]
        word = options[int(rng.next() * len(options))]
        if word == ".":
            break
        words.append(word)
    return " ".join(words)
follows = train(corpus)
print("the most likely word every time:")
print("  " + greedy(follows, "a return"))
print("six runs, the word is drawn by probability:")
rng = Rng(3876)
for i in range(1, 7):
    answer = sample(follows, "a return", rng)
    mark = "in the text" if answer in corpus else "not in the text"
    print(f"  {i}. {answer} ({mark})")
Output
the most likely word every time:
  a return is within 2 days
six runs, the word is drawn by probability:
  1. a return is 24 months (not in the text)
  2. a return is within 14 days of delivery (in the text)
  3. a return is safe (not in the text)
  4. a return is within 5 working days (not in the text)
  5. a return is within 2 days (not in the text)
  6. a return is within 3 days (not in the text)

The most likely answer is "a return is within 2 days", and the text says 14. Of the six drawn answers only one is a sentence from the text. The rest are smooth and wrong: "a return is 24 months" is taken from the warranty, and "within 5 working days" from refunds.

See it

Where the number came from

# the most common one (on a tie, the first in the text)
def most_common(options):
    return max(options, key=options.count)




def greedy(follows, start):
    words = start.split(" ")
    while len(words) < 12:
        last = words[-1]
        word = most_common(follows[last])
        if word == ".":
            break
        words.append(word)
    return " ".join(words)
Step by step, in words
  1. The model continues "a return" with the most common next word each time.
  2. "is within" is in 4 sentences, one of them about returns. So far it is true.
  3. After "within" the most common word is "2": it comes from the two sentences about delivery.
  4. Then the full stop. A smooth sentence that is not in the text: "a return is within 2 days".
  5. The check takes the numbers from the answer and from the returns policy.
  6. 2 is not in the policy. It says 14 days.
  7. Every pair of words is from the shop’s text. The whole sentence is false, and it reads as confidently as a true one.

Every pair of words in the answer is true somewhere in the text. The whole sentence is true nowhere. Large models look much more than one word back and are wrong far less often, but the principle is the same: they write what is likely, not what is checked.

Try it

Predict the output

The model is trained on three sentences. What will it write with the most likely word every time, if it starts from the word for delivery?

corpus = [
    "delivery to sofia is 2 days",
    "delivery to varna is 3 days",
    "a size swap is 3 days",
]
print(greedy(train(corpus), "delivery"))
Answer

The idea

The same question, a different answer

When the word is drawn by probability, every run may give a different answer. In the program the six runs gave five different sentences. This is non-determinism: the same input, a different output. Tests, logs and customer complaints get harder, because "it works for me" no longer means anything.

Temperature 0 (the most likely word every time) makes the answer repeatable, but not right. The program gave "within 2 days" on every run. With real providers even temperature 0 does not promise a completely identical answer: the computations on the server do not always run in the same order.

The idea

The knowledge cutoff

The model only knows the text it was trained on. That text was collected up to a certain date, the knowledge cutoff. Whatever happened after it, the model does not know: the new returns policy, the new product, today's price.

And it knows nothing that was never in its text: the status of order 1042 is in the shop's database, not in the model. By default the model does not say "I don't know". It writes a likely answer, and that is where "arrives tomorrow", which nobody checked, comes from.

That is why facts go into the request: the policy as text in the prompt, the order status from the program. How the model asks for the status itself is in the lesson on tools.

The idea

Check the answer against the source

When the answer must rest on a document, the program can check at least part of it. The roughest check: every number in the answer must be in the source.

# every number in the answer must be in the source
def check(answer, source):
    known = numbers(source)
    claimed = numbers(answer)
    if not claimed:
        return "? no number to check"
    for n in claimed:
        if n not in known:
            return f"✗ {n} is not in the source"
    return "✓ the numbers are in the source"




def numbers(text):
    words = [w.strip(".,") for w in text.split(" ")]
    return [w for w in words if w.isdigit()]

The six answers from the program above against the returns policy:

source = "Returns are accepted within 14 days of delivery. Refunds are paid within 5 working days."
answers = [
    "a return is 24 months",
    "a return is within 14 days of delivery",
    "a return is safe",
    "a return is within 5 working days",
    "a return is within 2 days",
    "a return is within 3 days",
]
for a in answers:
    print(f"{a} → {check(a, source)}")
Output
a return is 24 months → ✗ 24 is not in the source
a return is within 14 days of delivery → ✓ the numbers are in the source
a return is safe → ? no number to check
a return is within 5 working days → ✓ the numbers are in the source
a return is within 2 days → ✗ 2 is not in the source
a return is within 3 days → ✗ 3 is not in the source

The check caught three wrong answers. But it let "a return is within 5 working days" pass: 5 is in the policy, only it is for refunds. And it cannot say anything about "a return is safe", because there is no number in it. A rough check lowers the risk without removing it. That is why real systems have several checks, and where a mistake is expensive, a person looks.

Try it

Find the bug

A colleague simplifies the check. Now the answer "a return is within 4 days" passes as correct, although the policy says 14. Which line has the bug?

Pick the line with the bug.

Try it

Fill in

The model ends the sentence when the most likely next word is the end. What does the end look like in the training text?

def greedy(follows, start):
    words = start.split(" ")
    while len(words) < 12:
        last = words[-1]
        word = most_common(follows[last])
        if word == :
            break
        words.append(word)
    return " ".join(words)

The idea

When a language model is the wrong tool

  • There is an exact answer that can be computed. The order total, VAT, an account balance. One line of code does not get it wrong; the model may get a digit wrong.
  • The data is live. Order status, stock, today's price. They are in the database. The model may ask for them through a tool, but must not "remember" them.
  • The rule is firm. "Returns within 14 days" is an if, not a prompt. Whatever must always be the same is written in code.
  • The decision must be explained and checked later. A refused payment or loan needs a reason that repeats. For such decisions there are models like logistic regression, and rules are written out explicitly.
  • A mistake costs more than silence. Medical, legal, financial answers to a customer. Then the model writes a draft and a person decides.

Where the model is strong: drafts, summaries, translation, sorting free text (which department the email is for), extracting data from text, answers from a document that get checked. Tasks where a small inaccuracy is easy to catch and does not cost much.

More precisely, if you are curious

Why large models hallucinate too. They are trained on a huge text, and then a little more to follow instructions and to say "I don't know". That lowers the mistakes but does not remove them: the model still picks likely text. Rare facts are especially dangerous (the number of an article of law, a quote, a function name in a small library), where the training text has few examples.

What helps, in layers. Facts in the request (the policy as text; the module on semantic search and RAG later in the course does exactly that). Output to a schema and a check in code (structured output). A test set of questions with correct answers, run on every change (evaluation). An "I don't know, passing you to a colleague" answer when the check fails. And a person where a mistake is expensive.

How small the model here is. The bigram model remembers one word back and is trained on ten sentences. A large model looks thousands of tokens back and is trained on trillions. That is why it rarely glues "a return is within 2 days" together from two sentences. But when the context holds no answer, it still writes the most likely continuation, not "I don't know".

Measuring hallucinations. The share of wrong answers depends on the task and the data. Numbers from other people's tests say nothing about your assistant. You measure them yourself, on your questions, as in the lesson on evaluation.

Exercise

The delivery answers

Solve it in your editor.

The assistant answers about delivery, and the source is: "Delivery to Sofia takes up to 2 working days. To Varna it takes up to 3 working days. Delivery is free over 50 euro." Check five answers with the same rule (every number in the answer must be in the source) and print the verdict for each:

  • Delivery to Sofia takes up to 2 days.
  • Delivery to Varna takes up to 4 days.
  • Delivery is free over 30 euro.
  • Delivery to Plovdiv takes up to 1 day.
  • Delivery is free for all orders.
Output
Delivery to Sofia takes up to 2 days. → ✓ the numbers are in the source
Delivery to Varna takes up to 4 days. → ✗ 4 is not in the source
Delivery is free over 30 euro. → ✗ 30 is not in the source
Delivery to Plovdiv takes up to 1 day. → ✗ 1 is not in the source
Delivery is free for all orders. → ? no number to check

The last answer is wrong, and the check cannot catch it: there is no number in it.

More tasks

  • The number in its place. Check not only that the number is there, but also the word before it: "up to 2" for Sofia. Which answers does it catch now?
  • Refusal. When the check fails, the program returns "I will connect you with a colleague." instead of the model's answer.

Test

Check yourself

1. What is a hallucination of a language model?
2. You set the temperature to 0 and the assistant gives the same answer every time. Does that mean the answer is right?
3. The returns policy changed yesterday. Why does the model still state the old one?
4. The check "are the numbers of the answer in the source" let "a return is within 5 working days" pass. Why?
5. For which task is a language model the wrong tool?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·