Skip to content
wannacode

Chunking documents

In short: Chunking splits a document into pieces by words or by sentences, with an overlap, so that each piece gets its own vector and the answer is not cut in half.

Problem

The whole page in one vector

The shop has a returns policy. It is one page with six sentences:

// the returns policy: six sentences
const POLICY = [
  "you can return the item within 14 days of delivery",
  "the item must be in the original box",
  "a courier collects it for free",
  "the money goes back to your card within five days",
  "items with your own print can not be returned",
  "for damage write to us within two days",
];


// the question of the customer
const QUESTION = "when do i get my money back";

The customer asks "when do i get my money back". The answer is the fourth sentence. If the whole page is one text in the index, it is one vector:

console.log(`words: ${words.length}`);
const doc = POLICY.join(" ");
console.log(`whole document: ${cosine(v, space.ofText(doc)).toFixed(3)}`);
console.log(`the answer sentence: ${cosine(v, space.ofText(POLICY[3])).toFixed(3)}`);
Output
words: 51
whole document: 0.694
the answer sentence: 0.959

The whole page is 0.694, and the answer sentence alone is 0.959. The vector of the page is a mix of six topics: deadline, box, courier, money, exceptions, damage. The longer the page, the more it blurs. In a base with many documents such a page loses to short texts that do not answer the question at all.

And if it is found after all, all 51 words go into the prompt because of 10. Real policies are thousands of words, and three such pages fill the context window and the bill.

The idea

Pieces

The document is cut into pieces (chunks) before it goes into the index. Every piece is a separate entry: its own text, its own vector and which document it is from. The search from the previous lesson finds the piece, not the whole page.

The simplest cut is by number of words: the first 10 words, the next 10, and so on to the end. For the vectors in the lesson we use the same model as in the lesson on vectors, with two differences: it also learns from the text of the policy (otherwise its words are unknown), and it has 16 axes instead of 8. All the pieces are about returns, and with 8 axes they differ only a little.

const space = new Space([...TEXT, ...POLICY], STOP, 16);
const words = POLICY.join(" ").split(" ");
const v = space.ofText(QUESTION);

See it

Pieces of 10 words

The function cuts the words into pieces. For now the overlap is 0: every piece starts where the one before ends.

// pieces of size words; the step is size - overlap
function chunk(words, size, overlap) {
  const pieces = [];
  const step = size - overlap;
  for (let from = 0; from < words.length; from += step) {
    const end = Math.min(from + size, words.length);
    pieces.push(words.slice(from, end).join(" "));
    if (end === words.length) break;
  }
  return pieces;
}

Another function prints every piece with its similarity to the question and which one is nearest:

// every piece with its similarity; the nearest at the end
function show(space, v, pieces) {
  let best = 0;
  let top = -1;
  pieces.forEach((p, i) => {
    const c = cosine(v, space.ofText(p));
    console.log(`${i + 1} ${c.toFixed(3)} ${p}`);
    if (c > top) {
      top = c;
      best = i;
    }
  });
  console.log(`→ ${best + 1}`);
}
show(space, v, chunk(words, 10, 0));
Output
1 0.515 you can return the item within 14 days of delivery
2 0.246 the item must be in the original box a courier
3 0.927 collects it for free the money goes back to your
4 0.813 card within five days items with your own print can
5 0.682 not be returned for damage write to us within two
6 0.463 days
→ 3

Nearest is piece 3 (0.927). It ends with "back to your", and "card within five days" is in piece 4. The model will read "the money goes back to your" and will not know when. The cut fell right in the middle of the answer.

The idea

Overlap

The cut is blind: it does not know where the sentence ends. The fix is an overlap: every piece starts a little before the end of the one before. With pieces of 10 words and an overlap of 4, every piece starts 6 words after the one before, and its last 4 words are the first 4 of the next one.

So the edges repeat, and a sentence cut in one piece is whole in the one next to it. The price is more pieces and repeated words.

See it

With an overlap of 4

show(space, v, chunk(words, 10, 4));
Output
1 0.515 you can return the item within 14 days of delivery
2 0.478 14 days of delivery the item must be in the
3 0.189 must be in the original box a courier collects it
4 0.847 a courier collects it for free the money goes back
5 0.959 the money goes back to your card within five days
6 0.813 card within five days items with your own print can
7 0.692 your own print can not be returned for damage write
8 0.738 returned for damage write to us within two days
→ 5

Now the nearest is piece 5 (0.959), and it holds the whole sentence: "the money goes back to your card within five days". There are 8 pieces instead of 6.

// pieces of size words; the step is size - overlap
function chunk(words, size, overlap) {
  const pieces = [];
  const step = size - overlap;
  for (let from = 0; from < words.length; from += step) {
    const end = Math.min(from + size, words.length);
    pieces.push(words.slice(from, end).join(" "));
    if (end === words.length) break;
  }
  return pieces;
}


const v = space.ofText(QUESTION);
show(space, v, chunk(words, 10, 0));
Step by step, in words
  1. The returns policy: 51 words. In orange is the sentence with the answer: "the money goes back to your card within five days".
  2. Pieces of 10 words with no overlap: 6 pieces, one after the other.
  3. The similarity of every piece to the question. Nearest is piece 3 (0.927).
  4. But piece 3 ends with "back to your": "card within five days" is in the next piece. The answer is cut in half.
  5. Overlap 4: every piece starts 6 words after the one before, so the edges repeat. 8 pieces.
  6. Nearest is piece 5 (0.959), and it holds the whole sentence.
  7. With no overlap the answer is split between two pieces; with overlap one piece holds it whole. The price: 8 pieces instead of 6, and repeated words.

Try it

Predict the output

A sentence of 8 words in pieces of 4 with an overlap of 1. How many pieces come out?

const ws = "delivery to a courier office costs 3 euro".split(" ");
console.log(chunk(ws, 4, 1).length);
Answer

Try it

Fill in

How many words later does every next piece start? What goes in the place of ___?

// pieces of size words; the step is size - overlap
function chunk(words, size, overlap) {
  const pieces = [];
  const step = size - ;
  for (let from = 0; from < words.length; from += step) {
    const end = Math.min(from + size, words.length);
    pieces.push(words.slice(from, end).join(" "));
    if (end === words.length) break;
  }
  return pieces;
}

Try it

Find the bug

A colleague cuts with an overlap of 4, but part of the policy is in no piece at all. On which line is the bug?

Pick the line with the bug.

The idea

By sentences

A number of words is rough. Documents have natural borders: sentences, paragraphs, headings. If a piece is a whole sentence, the answer is not cut:

show(space, v, POLICY);
Output
1 0.515 you can return the item within 14 days of delivery
2 0.261 the item must be in the original box
3 0.132 a courier collects it for free
4 0.959 the money goes back to your card within five days
5 0.588 items with your own print can not be returned
6 0.693 for damage write to us within two days
→ 4

The answer sentence wins (0.959). But look at sentence 3: "a courier collects it for free", 0.132. "it" is the item from the sentence before. On its own the piece says neither what the courier collects nor that it is about returns. If the customer asks "does a courier come for the return", this piece has lost part of its meaning.

That is why real systems keep the link: they put the title of the document and of the section in front of every piece, add a sentence from each neighbour, or cut by paragraphs. And they keep for every piece where it is from: document, section, place. That is how the answer cites its source.

The idea

When it hurts

  • Pieces that are too small lose the meaning: "within 14 days" without the word "return" does not say what the 14 days are for.
  • Pieces that are too big blur like the whole page and fill the prompt.
  • The overlap brings repeats. Pieces 5 and 6 have the same words. If both are among the first k, repeated text goes into the prompt. Neighbouring pieces of one document are merged before they go to the model.
  • Tables, lists and code are not cut by number of words. A cut row of a price table makes no sense: such documents are cut by their structure.
  • Change the cut, and you change the index. All pieces are cut and computed again.

There is no size that is right for all documents. The size and the overlap are chosen with a test set of questions, like k in the previous lesson: see evaluation.

More precisely, if you are curious

The size in real systems. Pieces are measured in tokens, not in words: often 200 to 800 tokens with an overlap of 10 to 20% (example numbers). How many tokens a text is, is in the lesson on tokens. Here the pieces are 10 words and the sentences are almost as long, which is why the overlap is big.

When the overlap keeps the answer whole. A sentence of L words is whole in some piece if L ≤ overlap + 1. With an overlap of 4 that is sentences of up to 5 words. A longer sentence is whole only if some cut falls right before it: the exercise shows that this depends on where the cuts fall.

Cutting by meaning. Some systems cut where the vectors of neighbouring sentences drift apart: a change of topic is a border. That costs one call to the model for vectors per sentence.

Exercise

A check for the cut

Solve it in your editor.

Write a check: for pieces of 10 words and an overlap from 0 to 4, find the piece nearest to the question and print how many pieces there are, the similarity and whether the piece holds the whole answer sentence.

Output
overlap 0: 6 pieces, 0.927, answer whole: no
overlap 1: 6 pieces, 0.891, answer whole: no
overlap 2: 7 pieces, 0.959, answer whole: yes
overlap 3: 7 pieces, 0.967, answer whole: no
overlap 4: 8 pieces, 0.959, answer whole: yes

Overlaps 2 and 4 keep the answer whole, and 1 and 3 do not. With overlap 3 the similarity is the highest (0.967), but the piece ends with "your card", and "within five days" is cut off. A higher similarity does not mean a whole answer. Whether the answer is whole depends on where the cuts fall, so such a check runs every time the cut changes.

More tasks

  • Sentences up to 16 words. Glue whole sentences into a piece until it goes over 16 words. Which piece is the nearest?
  • No repeats. If two of the first k pieces overlap, merge them into one.

Test

Check yourself

1. Why is a long document not put into the index as one vector?
2. Pieces of 10 words with an overlap of 4. How many words apart do they start?
3. What is lost when every sentence is a separate piece?
4. The overlap is 4, and the nearest k = 3 pieces are 4, 5 and 6. What is the problem?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·