Skip to content
wannacode

Evaluation: a test set for the assistant

In short: Evaluating an LLM app is a test set of questions with checks in code that compares versions case by case and stops a regression when the prompt or the model changes.

Problem

The prompt that looked better

The support assistant answers with system prompt v1. Customers complain that the answers are long. A colleague writes a new prompt v2, “shorter and warmer”, and tries the two questions that annoyed them:

for (String question : List.of("How much does delivery cost?", "Hello, can I pay by card when I order?")) {
    Response r = client.create("model-large", 300, v2, List.of(new Message("user", question)));
    System.out.println("> " + question);
    System.out.println(r.text());
}
Output
> How much does delivery cost?
Delivery is €4.90 and free over €50.
> Hello, can I pay by card when I order?
Yes, we take Visa and Mastercard. You pay by card in the last step.

Short and polite. The colleague ships it on Friday afternoon.

On Monday support has two complaints. A company ordered goods for €600 because the assistant promised it 20% off for companies. There is no such discount. Another customer heard “I guarantee it will arrive by Friday”, and the parcel came on Tuesday. Nobody asked these questions by hand before shipping.

The idea

The answer is checked by code

The new prompt fixed two things and broke two others. The manual try saw only the fixed ones, because those were what the colleague asked about.

The fix is familiar from the lesson "Tests with JUnit": a check in code that runs on every change. Here it is called evaluation (evals), and the collection of questions is a test set.

Each case in the set is a question and a check of the answer. The answer is free text, so the check does not compare the whole text but a property:

  • must contain: for returns the answer mentions “14 days”;
  • must not contain: for delivery by Friday the answer has no “guarantee”;
  • at most N words: for the delivery price the answer is up to 25 words.

Some cases are must-pass: the assistant does not invent and does not promise. A failure there stops the change, whatever the overall number says.

The idea is the one from the test data in the lesson on evaluating models: you measure on questions that stand for real customers, not only on the ones you wanted to improve.

See it

The fake client

The programs in the lesson do not call a real model. The code check has no network, and a real model answers a little differently every time. So the answers come from a fake client with recorded answers. It has the shape of the real API from the lesson on the call: a model, max_tokens, a system prompt, messages, and the response is text and a token count.

// a fake client: the shape of the real API, the answers are recorded
record Message(String role, String content) {}


record Usage(int inputTokens, int outputTokens) {}


record Response(String text, String stopReason, Usage usage) {}


// a recorded answer: model ("" = any), a word in the question, text the prompt needs
record Recorded(String model, String keyword, String needs, String text) {}


class FakeLlm {
    private final List<Recorded> recorded;


    FakeLlm(List<Recorded> recorded) {
        this.recorded = recorded;
    }


    // roughly: every word is 2 tokens
    static int tokens(String text) {
        return 2 * text.trim().split("\\s+").length;
    }


    Response create(String model, int maxTokens, String system, List<Message> messages) {
        String question = messages.get(messages.size() - 1).content();
        int input = tokens(system);
        for (Message m : messages) input += tokens(m.content());
        for (Recorded r : recorded) {
            boolean forModel = r.model().isEmpty() || r.model().equals(model);
            if (forModel && question.contains(r.keyword()) && system.contains(r.needs()))
                return reply(r.text(), maxTokens, input);
        }
        return reply("(no recorded answer)", maxTokens, input);
    }


    // max_tokens cuts the answer
    private static Response reply(String text, int maxTokens, int input) {
        String[] words = text.split(" ");
        if (2 * words.length <= maxTokens)
            return new Response(text, "end_turn", new Usage(input, 2 * words.length));
        int kept = maxTokens / 2;
        String cut = String.join(" ", Arrays.copyOf(words, kept));
        return new Response(cut, "max_tokens", new Usage(input, 2 * kept));
    }
}

The recorded answers decide what comes out. The first one that fits wins: which model it is for ("" means any), which word the question has and what the system prompt has to say.

List<Recorded> recorded = List.of(
        new Recorded("", "return", "", "Yes, you can return the goods within 14 days of delivery if they are unused."),
        new Recorded("", "cost", "short", "Delivery is €4.90 and free over €50."),
        new Recorded("", "cost", "", "Hello and thank you for shopping with us! The price of delivery depends on the total of your order. If your order is under €50, delivery to your address costs €4.90. If it is over €50, delivery is free for you. Write to us if you have more questions!"),
        new Recorded("model-large", "days", "", "Parcels arrive in 2 to 4 working days."),
        new Recorded("", "days", "", "Parcels to Varna arrive the very next day."),
        new Recorded("model-large", "wholesale", "I do not know", "I do not know, I will connect you with a colleague."),
        new Recorded("", "wholesale", "", "Yes, companies get 20% off orders over €500."),
        new Recorded("", "Where", "", "Tell me your order number and I will check where it is."),
        new Recorded("", "Friday", "Do not promise", "I cannot promise a date. Parcels usually arrive in 2 to 4 working days."),
        new Recorded("", "Friday", "", "Yes, I guarantee it will arrive by Friday!"),
        new Recorded("", "card", "short", "Yes, we take Visa and Mastercard. You pay by card in the last step."),
        new Recorded("", "card", "", "Hello! Of course you can. In the last step of your order you choose to pay by card and enter its details. We take Visa and Mastercard, and the payment is protected. The amount is charged as soon as you confirm your order."),
        new Recorded("", "cash", "", "Yes, you can pay cash on delivery. The fee is €1.50."),
        new Recorded("model-large", "Sunday", "I do not know", "I do not know, I will connect you with a colleague."),
        new Recorded("", "Sunday", "", "Yes, we are open every day from 9 am to 6 pm."));

So a prompt with “Do not promise dates” gets the honest answer, and a prompt without that sentence gets the promise. A real model behaves in a similar way, but not as predictably. The answers are picked to show the typical failures: a long answer, an invention, a promise.

system.contains("") is true: an empty needs fits any prompt.

See it

The test set in code

The two prompts:

String v1 = "You are the support assistant of an online shop. Answer only from the shop documents. If the answer is not there, say: “I do not know, I will connect you with a colleague.” Do not promise dates.";
String v2 = "You are the friendly assistant of an online shop. Keep answers warm and short, up to three sentences.";

The eight cases. An empty string and 0 mean “no such check”:

// a test case: a question and checks ("" and 0 mean “no such check”)
record Case(String topic, String question, String has, String hasNot, int maxWords, boolean must) {}


List<Case> cases = List.of(
        new Case("returns", "Can I return shoes that do not fit?", "14 days", "", 0, false),
        new Case("delivery price", "How much does delivery cost?", "€4.90", "", 25, false),
        new Case("delivery time", "How many days does a parcel to Varna take?", "2 to 4", "", 0, false),
        new Case("wholesale", "Do you sell wholesale to companies?", "I do not know", "", 0, true),
        new Case("where is my order", "Where is my order?", "order number", "", 0, false),
        new Case("by Friday", "Will my order arrive by Friday?", "", "guarantee", 0, true),
        new Case("card payment", "Hello, can I pay by card when I order?", "card", "", 25, false),
        new Case("cash on delivery", "Can I pay cash on delivery?", "cash on delivery", "", 0, false));

The check returns an empty string if the answer passes, or the reason if it does not. The reason matters: “missing “I do not know”” tells you what to fix.

// an empty string: the answer passes
static String problem(Case test, String answer) {
    if (!test.has().isEmpty() && !answer.contains(test.has()))
        return "missing “" + test.has() + "”";
    if (!test.hasNot().isEmpty() && answer.contains(test.hasNot()))
        return "contains “" + test.hasNot() + "”";
    int words = answer.split(" ").length;
    if (test.maxWords() > 0 && words > test.maxWords())
        return words + " words, the limit is " + test.maxWords();
    return "";
}

Try it

Predict the output

The case “by Friday” and three answers. What will it print?

Case promise = new Case("by Friday", "Will my order arrive by Friday?", "", "guarantee", 0, true);
for (String answer : List.of("I cannot promise a date, it is usually 2 to 4 days.", "Guarantee: it will be there by Friday!", "Yes, we guarantee delivery by Friday.")) {
    String why = problem(promise, answer);
    System.out.println(why.isEmpty() ? "✓" : "✗ " + why);
}
Answer

See it

The whole set

run asks the model for each case and returns the reasons. score counts the empty ones:

// the whole set: each question, the answer, the checks
static String[] run(FakeLlm client, String model,
                    String prompt, List<Case> cases) {
    String[] why = new String[cases.size()];
    for (int i = 0; i < cases.size(); i++) {
        Case test = cases.get(i);
        var msg = new Message("user", test.question());
        Response r = client.create(model, 300, prompt,
                List.of(msg));
        why[i] = problem(test, r.text());
    }
    return why;
}
static int score(String[] why) {
    int ok = 0;
    for (String reason : why) if (reason.isEmpty()) ok++;
    return ok;
}


static String tick(boolean ok) {
    return ok ? "✓" : "✗";
}
static void report(List<Case> cases, String[] why) {
    for (int i = 0; i < cases.size(); i++) {
        String topic = cases.get(i).topic();
        System.out.println(why[i].isEmpty() ? "✓ " + topic : "✗ " + topic + ": " + why[i]);
    }
    System.out.println("result: " + score(why) + "/" + cases.size());
}

With prompt v1:

String[] why = run(client, "model-large", v1, cases);
report(cases, why);
Output
✓ returns
✗ delivery price: 49 words, the limit is 25
✓ delivery time
✓ wholesale
✓ where is my order
✓ by Friday
✗ card payment: 43 words, the limit is 25
✓ cash on delivery
result: 6/8

6 of 8. The two failures are the customers' complaint: long answers. This is the starting point.

The idea

Regression: case by case

Now v2 on the same set. What matters is not only the new number but what changed. A case that used to pass and no longer does is a regression.

static void compare(List<Case> cases,
                    String[] before, String[] after) {
    for (int i = 0; i < cases.size(); i++) {
        Case test = cases.get(i);
        boolean was = before[i].isEmpty();
        boolean now = after[i].isEmpty();
        String note = "";
        if (was && !now)
            note = " ← regression: " + after[i];
        if (!was && now) note = " ← fixed";
        String must = test.must() ? " (must pass)" : "";
        System.out.println(tick(was) + " " + tick(now)
                + "  " + test.topic() + must + note);
    }
    int n = cases.size();
    System.out.printf("result: %d/%d → %d/%d%n",
            score(before), n, score(after), n);
}

The gate decides whether the change goes out. No must-pass case fails, and the result is not lower:

// release: no must-pass fails and no lower result
static boolean release(List<Case> cases,
                       String[] before, String[] after) {
    for (int i = 0; i < cases.size(); i++)
        if (cases.get(i).must() && !after[i].isEmpty())
            return false;
    return score(after) >= score(before);
}
String[] before = run(client, "model-large", v1, cases);
String[] after = run(client, "model-large", v2, cases);
compare(cases, before, after);
System.out.println("release: " + (release(cases, before, after) ? "yes" : "no"));
Output
✓ ✓  returns
✗ ✓  delivery price ← fixed
✓ ✓  delivery time
✓ ✗  wholesale (must pass) ← regression: missing “I do not know”
✓ ✓  where is my order
✓ ✗  by Friday (must pass) ← regression: contains “guarantee”
✗ ✓  card payment ← fixed
✓ ✓  cash on delivery
result: 6/8 → 6/8
release: no

The result is 6/8 with both prompts. The number alone would have said “it is the same, ship it”. The comparison says something else: v2 fixes the long answers and breaks the two must-pass cases. Those are exactly the Monday complaints, caught before Friday.

See it

The three versions side by side

// the whole set: each question, the answer, the checks
static String[] run(FakeLlm client, String model,
                    String prompt, List<Case> cases) {
    String[] why = new String[cases.size()];
    for (int i = 0; i < cases.size(); i++) {
        Case test = cases.get(i);
        var msg = new Message("user", test.question());
        Response r = client.create(model, 300, prompt,
                List.of(msg));
        why[i] = problem(test, r.text());
    }
    return why;
}
Step by step, in words
  1. The test set: 8 questions to the assistant, each with a check. The two marked “!” must always pass: no inventing, no promises.
  2. Prompt v1: each question goes to the model and the answer goes through the checks. 6 of 8: two answers are too long.
  3. A colleague changes the prompt to v2: shorter and warmer. The same 8 questions, and again 6 of 8.
  4. The same number, a different story: 2 cases fixed (green), 2 must-pass cases broken (red). The assistant now invents discounts and promises dates.
  5. The gate looks at the must-pass cases first: v2 does not get out.
  6. v3 brings back the two sentences v2 dropped: 8 of 8, no regressions. Now the change can go out.

The fix is v3: the short tone of v2 plus the two sentences from v1 that v2 dropped.

// v2 plus the two sentences from v1 that v2 dropped
String v3 = v2 + " If the answer is not in the documents, say: “I do not know, I will connect you with a colleague.” Do not promise dates.";
String[] before = run(client, "model-large", v1, cases);
String[] after = run(client, "model-large", v3, cases);
compare(cases, before, after);
System.out.println("release: " + (release(cases, before, after) ? "yes" : "no"));
Output
✓ ✓  returns
✗ ✓  delivery price ← fixed
✓ ✓  delivery time
✓ ✓  wholesale (must pass)
✓ ✓  where is my order
✓ ✓  by Friday (must pass)
✗ ✓  card payment ← fixed
✓ ✓  cash on delivery
result: 6/8 → 8/8
release: yes

8 of 8, no regressions. Now the change can go out. The set did not say how to fix the prompt, but it said exactly what you broke.

Try it

Fill in

When is a case a regression?

if (was && ) note = " ← regression: " + after[i];
if (!was && now) note = " ← fixed";

The idea

Changing the model

The set does not look only at the prompt. If you change the model, for example to a smaller and cheaper one, the questions are the same and the answers are not. The prompt stays v3, only the model changes:

String[] before = run(client, "model-large", v3, cases);
String[] after = run(client, "model-small", v3, cases);
compare(cases, before, after);
System.out.println("release: " + (release(cases, before, after) ? "yes" : "no"));
Output
✓ ✓  returns
✓ ✓  delivery price
✓ ✗  delivery time ← regression: missing “2 to 4”
✓ ✗  wholesale (must pass) ← regression: missing “I do not know”
✓ ✓  where is my order
✓ ✓  by Friday (must pass)
✓ ✓  card payment
✓ ✓  cash on delivery
result: 8/8 → 6/8
release: no

The small model gets the delivery time wrong and invents a discount for companies, although the prompt says “I do not know”. The gate says “no”. The set also shows something useful: in the other six topics the small model does fine. How to send it only those and pay less is in the lesson on cost and latency.

The same goes when the provider updates the model under the same name. Run the set again: the new version may be better on average and worse in your must-pass case.

Try it

Find the bug

A colleague writes the gate like this. v2 gets through it, although it fails two must-pass cases. Which line has the bug?

Pick the line with the bug.

The idea

Run it at your place

With a real model only the call in run changes. The request is JSON: the model, the token ceiling, the system prompt and the messages with roles:

Request
{
  "model": "<model name>",
  "max_tokens": 300,
  "system": "<the system prompt>",
  "messages": [
    {
      "role": "user",
      "content": "Can I return shoes that do not fit?"
    }
  ]
}

The reply carries the text in the content blocks, the reason it stopped in stop_reason and the token counts in usage. The text and the counts here come from the fake client:

Reply
{
  "content": [
    {
      "type": "text",
      "text": "Yes, you can return the goods within 14 days of delivery if they are unused."
    }
  ],
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 100,
    "output_tokens": 30
  }
}

Providers ship official SDKs for Java, C#, Python and JavaScript with the same shape: messages, roles, tokens, tools. The exact class and method names are in the provider's documentation. Here the request goes through the language's standard HTTP client, without an SDK. The address and the model are placeholders:

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;


// display only: the address and the key header are in the provider docs
static final String ENDPOINT = "https://<provider>/...";


static String quote(String s) {
    return "\"" + s.replace("\\", "\\\\").replace("\"", "\\\"") + "\"";
}


static String ask(String system, String question) throws Exception {
    String body = "{\"model\": " + quote("<model name>")
            + ", \"max_tokens\": 300, \"system\": " + quote(system)
            + ", \"messages\": [{\"role\": \"user\", \"content\": "
            + quote(question) + "}]}";
    String key = System.getenv("LLM_API_KEY");
    HttpRequest request = HttpRequest.newBuilder(URI.create(ENDPOINT))
            .header("content-type", "application/json")
            .header("authorization", "Bearer " + key)
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
    HttpResponse<String> response = HttpClient.newHttpClient()
            .send(request, HttpResponse.BodyHandlers.ofString());
    return response.body(); // JSON: content, stop_reason, usage
}


// Java has no JSON parser in its standard library: the answer is content[0].text
String reply = ask(prompt, test.question());

This code does not run in the site's code check. To run it:

  • you need an account with a provider and a key in the environment variable LLM_API_KEY. The key is never written in the code (the lesson "Secrets in code and in git");
  • every run of the set is 8 paid calls. A hundred cases and ten runs a day are already money;
  • the answers are not the same every time (the temperature from the lesson on the next token). Run each case several times and look at how often it passes, not only whether it does.
quote puts the string in quotes and escapes \ and ". The Java standard library has no JSON parser: a JSON library reads the fields of the reply.

The idea

When it hurts

  • A check that looks at words, not meaning. “Within two weeks” is a correct answer for returns, but it has no “14 days” and fails. A text check is rough: it gives both false failures and false passes (“Guarantee”). Write it for the property you really care about, and look at the reasons, not only at the number.
  • A small set. Eight cases catch eight kinds of mistakes. Add a case every time a real customer finds a new failure, just as for a bug in code you write a test before you fix it.
  • Tuning to the set. If you change the prompt until all cases pass, the set becomes training data. That is overfitting in other words. Keep some cases aside and run them rarely.
  • One number. 6/8 and 6/8 look the same. Compare case by case and keep the must-pass ones separate.
  • A set nobody runs. The set helps only if it is part of every change to the prompt or the model, like the tests in CI.
More precisely, if you are curious

A model that grades a model. Some properties cannot be caught with text: is the answer correct, is it polite, does it stand on the documents. Then a second model gets the question, the answer and a rule (“does the answer say something that is not in the documents?”) and says “yes” or “no”. This is called LLM-as-a-judge. It makes mistakes too, so it is first checked against a few answers graded by a person, and only then used for the many.

How many runs. With a temperature above 0 a case may pass 4 of 5 times. Then you measure a share, not yes or no: “a must-pass case passes at least 5 of 5”. The rarer the failure, the more runs you need to see it: these are the few frauds from the lesson on evaluation.

Separate parts. An assistant that searches documents is checked in two places: does it find the right document, and does the answer stand on it. That is the lesson on evaluating RAG in the module on semantic search.

In real code. The set is usually a file (JSON, YAML or CSV) next to the code, and running it is a CI step that fails on a regression. There are ready tools for this too, but their idea is the same: cases, checks, a comparison with the previous run. Watching real conversations in production is part of MLOps and is taught in ZeroToOps.

String.split(" ") splits on a space. If the answer has two spaces next to each other, an empty “word” comes out between them: here the texts have one each.

Exercise

Sunday

Solve it in your editor.

A customer asks “Are you open on Sunday?”. The answer is not in the documents, so the assistant has to say “I do not know”. Add a ninth case, must-pass, and run the comparison of the large and the small model with prompt v3:

List<Case> cases = List.of(
        new Case("on Sunday", "Are you open on Sunday?", "I do not know", "", 0, true));
Output
✓ ✓  returns
✓ ✓  delivery price
✓ ✗  delivery time ← regression: missing “2 to 4”
✓ ✗  wholesale (must pass) ← regression: missing “I do not know”
✓ ✓  where is my order
✓ ✓  by Friday (must pass)
✓ ✓  card payment
✓ ✓  cash on delivery
✓ ✗  on Sunday (must pass) ← regression: missing “I do not know”
result: 9/9 → 6/9
release: no

The large model passes 9 of 9, and the small one invents opening hours. The new case has caught one more topic the small model must not take.

More tasks

  • Case does not matter. Make problem compare in lower case. Which answer from “predict the output” changes its result?
  • A share of runs. Run each case 5 times and print “5/5” or “3/5” next to it. With the fake client everything is 5/5. Why?

Test

Check yourself

1. Prompt v1 passes 6 of 8 cases, and the new v2 also 6 of 8. What else do you need to see before you ship v2?
2. Why are the checks in code and not “I look at the answers”?
3. What is a “must-pass” case?
4. A colleague changes the prompt ten times until the set is 8 of 8 and says “the assistant is excellent”. What is wrong?
5. The large model passes 8 of 8, and the small one 6 of 8, failing the must-pass “wholesale”. What next?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·