Skip to content
wannacode

Evaluating RAG: does it find it, does it stand on it

In short: Evaluating RAG checks two things: whether the search finds the right piece (recall@k), and whether the answer stands only on what was found.

Problem

Cheaper, and it “works”

The assistant from the previous lesson puts up to 3 pieces in the prompt. A colleague notices that the pieces are most of the tokens. They lower k to 1 and try two questions:

for question in ["how much is delivery to an office", "the bank declined my card"]:
    print(question + "\n  → " + answer(question, 1))
Output
how much is delivery to an office
  → Delivery to a courier office costs 3 €. [delivery-1]
the bank declined my card
  → Try the payment with another card. [payment-2]

Both answers are right, with a source. The colleague ships the change.

The next day the assistant tells a customer that the money comes back within 14 days. The documents say 5. The two questions tried by hand could not catch it: for both of them the right piece is first by closeness.

The idea

Two separate questions

RAG can go wrong in two places, and they are measured separately:

  1. Does it find it? The search has to bring the right piece among the first k. Its place among the found pieces is the rank. The share of questions where the rank is from 1 to k is recall@k. This is the same recall as in the lesson on evaluating a model: of everything you had to find, how much you found.
  2. Does it stand on it? The answer has to come from the pieces the model has seen: every source has to be from the prompt, and every number has to be in the cited piece.

The third one, “is it right”, is the “must have” check from the lesson on evaluation. The three together tell you where the mistake is: in the search, in the model or in the documents.

For this you need a test set: customer questions, and for each the piece that answers it. The set is written by hand, once, by someone who knows the documents.

See it

The test set and the rank

# a case of the test set: topic, question, the right piece ("" = none) and what the answer must have
@dataclass(frozen=True)
class Case:
    topic: str
    question: str
    expect: str
    has: str
# the test set
CASES = [
    Case("delivery price", "how much is delivery to an office", "delivery-1", "3 €"),
    Case("delivery time", "when will i get my parcel", "delivery-2", "two days"),
    Case("returns", "i do not want the item", "returns-1", "14 days"),
    Case("money back", "when will i get the money", "returns-2", "5 days"),
    Case("bank transfer", "can i pay by bank transfer", "payment-1", "bank transfer"),
    Case("declined card", "the bank declined my card", "payment-2", "another card"),
    Case("warranty", "what is the warranty on the item", "warranty-1", "2 years"),
    Case("sunday", "are you open on sunday", "", "I don't know"),
]


# an honest refusal starts like this
DECLINE = "i don't know"

The last question has no piece: the shop has written nothing about Sunday. The right answer is “I don't know”.

The rank is the place of the right piece among the found ones, and recall@k counts the questions where it is among the first k:

# the place of the right piece (from 1), 0 if missing
def rank(c):
    hits = search(c.question, len(PIECES))
    for i, h in enumerate(hits):
        if h.piece.id == c.expect:
            return i + 1
    return 0




# recall@k: how often it is among the first k
def recall(k):
    found = 0
    for c in CASES:
        r = rank(c)
        if r >= 1 and r <= k:
            found += 1
    return found
print(f"{'topic':<18}{'rank':<6}{'k=1':<5}k=3")
total = 0
for c in CASES:
    if not c.expect:
        continue
    total += 1
    r = rank(c)
    print(f"{c.topic:<18}{r:<6}{mark(1 <= r <= 1):<5}{mark(1 <= r <= 3)}")
for k in [1, 3]:
    print(f"recall@{k} = {recall(k)}/{total}")
Output
topic             rank  k=1  k=3
delivery price    1     ✓    ✓
delivery time     3     ✗    ✓
returns           4     ✗    ✗
money back        2     ✗    ✓
bank transfer     2     ✗    ✓
declined card     1     ✓    ✓
warranty          1     ✓    ✓
recall@1 = 3/7
recall@3 = 6/7

# the place of the right piece (from 1), 0 if missing
def rank(c):
    hits = search(c.question, len(PIECES))
    for i, h in enumerate(hits):
        if h.piece.id == c.expect:
            return i + 1
    return 0




# recall@k: how often it is among the first k
def recall(k):
    found = 0
    for c in CASES:
        r = rank(c)
        if r >= 1 and r <= k:
            found += 1
    return found
Step by step, in words
  1. The test set: 8 customer questions, and for each the piece that answers it. The last one has no piece in the documents.
  2. Rank: where the right piece is among the found ones. For “returns” it is only 4th: the small vector model misses.
  3. With k = 1 only the first piece goes into the prompt. The right one is there for 3/7 questions: recall@1 = 3/7.
  4. With k = 3 the right piece is in for 6/7: recall@3 = 6/7. Only “returns” is still left out.
  5. Does the answer stand on the pieces? Every source must be in the prompt and every number in the source: 7/8.
  6. Two different failures: the search did not find the piece (recall), and the model wrote a number that is not in its source (stands).

With k = 1 the right piece is in the prompt for only 3 of 7 questions. With k = 3 it is there for 6. “returns” has rank 4: the small vector model thinks “i do not want the item” is closer to the warranty.

See it

Does the answer stand on it

The check takes the answer and the pieces the model has seen. An honest refusal passes. Every source has to be among the pieces, and every number in the answer has to be a word in the cited pieces:

# does the answer stand on the pieces: "" or why not
def unsupported(answer, hits):
    if answer.lower().startswith(DECLINE):
        return ""
    ids = sources(answer)
    if not ids:
        return "no source"
    words = set()
    for source in ids:
        hit = next(
            (h for h in hits if h.piece.id == source), None)
        if hit is None:
            return "source not in the prompt: " + source
        words.update(hit.piece.text.split(" "))
    for word in answer.split(" "):
        if word.startswith("["):
            continue
        n = "".join(ch for ch in word if ch.isdigit())
        if n and n not in words:
            return f"number {n} not in the source"
    return ""

It returns an empty string when the answer stands, and the reason otherwise. The reason is more useful than ✗: it tells you where to look.

Try it

Predict the output

With k = 1, for “when will i get the money” only “returns-1” goes into the prompt: “you can return the item within 14 days in the original box”. What does the check print for this answer?

claim = "The money comes back within 14 days. [returns-1]"
why = unsupported(claim, search("when will i get the money", 1))
print(why or "stands")
Answer

See it

The whole set

report runs every question through RAG and prints the three marks: found, stands, right. With k = 3:

# the whole set with k pieces: found, stands, right
def report(k):
    found = total = stands = right = 0
    print(f"k = {k}")
    print("topic             found  stands  right")
    for c in CASES:
        hits = search(c.question, k)
        text = answer(c.question, k)
        why = unsupported(text, hits)
        f = "—"
        if c.expect:
            total += 1
            in_top = any(h.piece.id == c.expect for h in hits)
            found += in_top
            f = mark(in_top)
        stands += why == ""
        right += c.has in text
        row = f"{c.topic:<18}{f:<7}{mark(why == ''):<8}{mark(c.has in text)}"
        print(row if why == "" else row + "  " + why)
    n = len(CASES)
    print(f"recall@{k} {found}/{total}, stands {stands}/{n}, right {right}/{n}")
report(3)
Output
k = 3
topic             found  stands  right
delivery price    ✓      ✓       ✓
delivery time     ✓      ✓       ✓
returns           ✗      ✗       ✗  number 30 not in the source
money back        ✓      ✓       ✓
bank transfer     ✓      ✓       ✓
declined card     ✓      ✓       ✓
warranty          ✓      ✓       ✓
sunday            —      ✓       ✓
recall@3 6/7, stands 7/8, right 7/8

The only failure is “returns”: the right piece was not found, the model cited the piece about the money and wrote 30 days, while the piece says 5. The check catches the number.

Now the same with k = 1, the colleague's change:

report(1)
Output
k = 1
topic             found  stands  right
delivery price    ✓      ✓       ✓
delivery time     ✗      ✗       ✓  source not in the prompt: delivery-2
returns           ✗      ✗       ✗  source not in the prompt: returns-2
money back        ✗      ✓       ✗
bank transfer     ✗      ✓       ✗
declined card     ✓      ✓       ✓
warranty          ✓      ✓       ✓
sunday            —      ✓       ✓
recall@1 3/7, stands 6/8, right 5/8

All three numbers fall. The reasons differ: for “delivery time” the model cites a piece it has not seen (and is right by chance), for “money back” it stands on the wrong piece, and for “bank transfer” it honestly says “I don't know.”, because the piece was not found. The two questions tried by hand were among the three that pass.

Try it

Fill in

Among which pieces is every source of the answer looked up? What goes in place of ___?

# does the answer stand on the pieces: "" or why not
def unsupported(answer, hits):
    if answer.lower().startswith(DECLINE):
        return ""
    ids = sources(answer)
    if not ids:
        return "no source"
    words = set()
    for source in ids:
        hit = next(
            (h for h in  if h.piece.id == source), None)
        if hit is None:
            return "source not in the prompt: " + source
        words.update(hit.piece.text.split(" "))
    for word in answer.split(" "):
        if word.startswith("["):
            continue
        n = "".join(ch for ch in word if ch.isdigit())
        if n and n not in words:
            return f"number {n} not in the source"
    return ""

Try it

Find the bug

A colleague rewrites recall. Now recall@1 is 0/7 and recall@3 is 5/7. Which line has the bug?

Pick the line with the bug.

The idea

When it hurts

  • The set is small. Here one question is 14% of recall. A real set has hundreds of questions taken from real conversations, and it grows with every bug a customer finds.
  • The labels get old. The right piece is written down by its source. When the documents are cut again, the sources change, and the set has to be updated with them.
  • Stands does not mean right. The “14 days” answer stands on the wrong piece. That is why the three checks go together: found, stands, right.
  • The check is rough. It looks only at digits and only at whether the word is there. “two days” in words is not checked, and the “5” from “5 euro” passes even if the answer says 5 days. A second model checks more precisely (see “More”), but it makes mistakes too.
  • There can be more than one right piece. The answer may be in two pieces. Then the set keeps all of them, and “found” means at least one, or all of them if both are needed.
  • The set runs on every change. A new k, a new threshold, a new vector model, new chunking, a new prompt: each changes the numbers. That is why the set is code and runs like the tests, with a bar below which the change does not ship, as in the lesson on evaluation.
More precisely, if you are curious

A second model as a judge. Instead of looking for digits, you can ask a second model: “Does this answer follow from these pieces? Yes or no.” It catches more than the rough check, but it costs one more call, and it has to be checked itself on a set with known grades.

Other measures for the search. recall@k says only whether the piece is in. MRR (the mean of 1/rank) also says how high it is: rank 1 gives 1, rank 3 gives 0.33. precision@k says how many of the k pieces are about the question, that is, how many tokens go to waste.

The set in CI. How the set runs automatically on every change and stops a bad version before it reaches customers is a job for ZeroToOps.

set() keeps the words of the cited pieces. next((h for h in hits if ...), None) returns the first match or None.

Exercise

How big should k be

Solve it in your editor.

Write a function that runs the whole set for a given k and prints recall, stands, right and how many tokens went into the prompts (the tokens of the prompt and of the question, only when the model is called). Run it for k from 1 to 4:

Output
k = 1: recall 3/7, stands 6/8, right 5/8, 734 tokens
k = 2: recall 5/7, stands 6/8, right 7/8, 916 tokens
k = 3: recall 6/7, stands 7/8, right 7/8, 994 tokens
k = 4: recall 7/7, stands 8/8, right 8/8, 1064 tokens

Which k would you choose, and why? k = 4 fixes “returns”, and it costs about 7% more tokens than k = 3. Think also about how much of these numbers is just this small set.

More tasks

  • Threshold. Run the set with a threshold of 0.6 and 0.7 at k = 3. What happens to recall and to “sunday”?
  • MRR. Compute the mean of 1/rank for the seven questions with a piece (rank 0 gives 0).

Test

Check yourself

1. The right piece for a question is third by closeness. For which k does it go into the prompt?
2. recall@3 is 6/7. What does that mean?
3. The answer cites [delivery-2], but the prompt has only “returns-1”. What does the check say?
4. The answer stands on the piece that is in the prompt. Does that mean it is right?
5. With k = 4 all three numbers are full (7/7, 8/8, 8/8), and the tokens are about 7% more than with k = 3. What follows from that?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·