Skip to content
wannacode

Text in pieces: tokens

In short: A token is a piece of text (a word, part of a word or a character) that a language model works with; BPE learns them by gluing the most frequent pairs, and price and limits are counted in tokens.

Problem

A dictionary of words

The support bot has to read customer messages. A model works with numbers, so a colleague builds a dictionary: every word in the shop texts gets a number. These are the texts: support answers, delivery notes and reviews.

// the shop text: support answers, delivery notes, reviews
const TEXT = [
  "your order was sent with a courier today",
  "the order will arrive at your address tomorrow",
  "the order will arrive within three days",
  "the order arrived at the courier office",
  "your parcel was handed to a courier",
  "the parcel will arrive within two days",
  "the parcel will be with you tomorrow",
  "the parcel is late by one day",
  "a courier will come to your address tomorrow",
  "a courier will call you before the delivery",
  "delivery to an address costs 5 euro",
  "delivery to a courier office costs 3 euro",
  "delivery is free over 50 euro",
  "delivery takes up to two days",
  "the item will arrive in the original box",
  "you can return the item within 14 days",
  "return the item in the original box",
  "returning the item is free",
  "the money will be paid back within five days",
  "after the item is returned you will get the money",
  "the money goes back to your card",
  "you can pay by card or by bank transfer",
  "paying by card is safe",
  "the bank declined the payment with your card",
  "try the payment with another card",
  "the payment will be confirmed by the bank",
  "the item arrived fast and it is very good",
  "i am very happy with the order",
  "the courier was kind and fast",
  "the item is good but the delivery was late",
];

Then every message becomes a row of numbers:

// the dictionary: every word in the text gets a number
const vocab = [];
for (const line of TEXT)
  for (const word of line.split(" "))
    if (!vocab.includes(word)) vocab.push(word);
console.log(`dictionary: ${vocab.length} words`);
for (const message of ["the parcel will arrive tomorrow", "when will my orders arrive"]) {
  const ids = message.split(" ").map((word) => (vocab.includes(word) ? String(vocab.indexOf(word)) : "?"));
  console.log(`${message} → ${ids.join(" ")}`);
}
Output
dictionary: 82 words
the parcel will arrive tomorrow → 8 19 9 10 13
when will my orders arrive → ? 9 ? ? 10

The first message goes through. In the second, "when" and "my" are not in the text, and "orders" is a different word from "order". The model gets "?" in three places out of five.

A dictionary of all words does not work. Every word has several forms (order, orders, ordered, ordering), and new ones come every day: product names, numbers, typos. And if you read letter by letter, every text goes through, but it becomes very long: 22 characters for five words.

The idea

Pieces between a letter and a word

Between letters and words are the frequent pieces: "the", "er", "arrive". A model reads text as a row of such pieces. Each piece is called a token, and cutting text into tokens is tokenization.

Which pieces become tokens is not written by hand. It is learned from text with the BPE algorithm (byte pair encoding):

  1. Every word is a list of letters.
  2. Count all pairs of neighbours in all words. A word that appears 7 times counts its pairs 7 times.
  3. The most frequent pair becomes one piece: that is one rule. Glue it everywhere.
  4. Repeat from step 2, as many rules as you want.

The rules are kept in order. New text is cut with the same rules, in the same order. A word that is not in the text still goes through: at worst it stays as letters. There is no "?".

See it

60 rules from the shop text

A rule is a pair of tokens and how many times it was seen. A word starts as letters:

// the word as a list of letters
const letters = (word) => [...word];

Gluing goes over the tokens of one word from left to right. Where left and right stand side by side, it puts one token instead of them and skips both:

// glue every left, right that stand side by side
function merge(tokens, left, right) {
  const glued = [];
  let i = 0;
  while (i < tokens.length) {
    if (tokens[i] === left && tokens[i + 1] === right) {
      glued.push(left + right);
      i += 2;
    } else {
      glued.push(tokens[i]);
      i += 1;
    }
  }
  return glued;
}

Learning is the four steps from the previous card. Map keeps the order of insertion, so on a tie the pair seen first wins, and the output is the same in all four languages.

// learn count rules: each time glue the most frequent pair
function learn(text, count) {
  const freq = new Map(); // how many times each word is in the text
  for (const line of text)
    for (const word of line.split(" ")) freq.set(word, (freq.get(word) ?? 0) + 1);
  const split = new Map([...freq.keys()].map((word) => [word, letters(word)])); // for now every word is just letters
  const rules = [];
  for (let r = 0; r < count; r++) {
    const pairs = new Map(); // the pairs of neighbours, counted with the word frequency
    for (const [word, t] of split)
      for (let i = 0; i + 1 < t.length; i++) {
        const key = `${t[i]} ${t[i + 1]}`;
        pairs.set(key, (pairs.get(key) ?? 0) + freq.get(word));
      }
    let best = "", most = 0;
    for (const [key, n] of pairs)
      if (n > most) [best, most] = [key, n]; // the most frequent pair (on a tie, the first one seen)
    const [left, right] = best.split(" ");
    rules.push({ left, right, count: most });
    for (const [word, t] of split) split.set(word, merge(t, left, right)); // the new rule holds for every word
  }
  return rules;
}
const rules = learn(TEXT, 60);
rules.forEach((r, k) => {
  const cell = `${k + 1}. ${r.left}+${r.right} (${r.count})`;
  process.stdout.write(k % 4 === 3 ? cell + "\n" : cell.padEnd(22));
});
Output
1. t+h (39)           2. th+e (29)          3. e+r (23)           4. u+r (20)
5. w+i (19)           6. a+r (15)           7. o+ur (13)          8. i+v (13)
9. o+r (12)           10. a+y (12)          11. i+n (12)          12. l+l (11)
13. a+n (11)          14. t+o (10)          15. wi+ll (10)        16. r+e (10)
17. e+l (10)          18. wi+th (9)         19. t+e (9)           20. er+y (8)
21. c+our (7)         22. cour+i (7)        23. couri+er (7)      24. d+ay (7)
25. iv+e (7)          26. i+s (7)           27. i+te (7)          28. ite+m (7)
29. y+our (6)         30. a+s (6)           31. ar+r (6)          32. arr+ive (6)
33. d+el (6)          34. del+iv (6)        35. deliv+ery (6)     36. or+d (5)
37. ord+er (5)        38. day+s (5)         39. y+o (5)           40. yo+u (5)
41. b+y (5)           42. o+n (5)           43. m+e (5)           44. c+ar (5)
45. car+d (5)         46. p+ay (5)          47. w+as (4)          48. n+t (4)
49. with+in (4)       50. p+ar (4)          51. par+c (4)         52. parc+el (4)
53. b+e (4)           54. on+e (4)          55. re+t (4)          56. ret+ur (4)
57. retur+n (4)       58. a+d (3)           59. ad+d (3)          60. add+re (3)

The first rule is "t" + "h": the pair was seen 39 times. The second glues "th" with "e" into "the". Rules build on earlier ones: "c" + "our" (where "our" is rule 7), then "couri", then "courier" are rules 21 to 23, and after them "courier" is one token. After rule 35 "delivery" is one token too.

Try it

Predict the output

Here are the first three rules from the output above. What does the program print?

const rules = [
  { left: "t", right: "h", count: 39 },
  { left: "th", right: "e", count: 29 },
  { left: "e", right: "r", count: 23 },
];
console.log(encode("there", rules).join("|"));
Answer

See it

New text as tokens

Cutting a sentence: every word becomes letters, then every rule in order. show only prints the tokens of each word with | between them.

// a sentence → tokens: letters, then the rules in order
function encode(sentence, rules) {
  const tokens = [];
  for (const word of sentence.split(" ")) {
    let t = letters(word);
    for (const r of rules) t = merge(t, r.left, r.right);
    tokens.push(...t);
  }
  return tokens;
}
// the tokens of each word, with | between them
const show = (sentence, rules) =>
  sentence.split(" ").map((word) => encode(word, rules).join("|")).join("  ");

Here is how one sentence goes through the rules. The counter at the bottom says how many letters and how many tokens there are:

// glue every left, right that stand side by side
function merge(tokens, left, right) {
  const glued = [];
  let i = 0;
  while (i < tokens.length) {
    if (tokens[i] === left && tokens[i + 1] === right) {
      glued.push(left + right);
      i += 2;
    } else {
      glued.push(tokens[i]);
      i += 1;
    }
  }
  return glued;
}


// a sentence → tokens: letters, then the rules in order
function encode(sentence, rules) {
  const tokens = [];
  for (const word of sentence.split(" ")) {
    let t = letters(word);
    for (const r of rules) t = merge(t, r.left, r.right);
    tokens.push(...t);
  }
  return tokens;
}
Step by step, in words
  1. Every word starts as letters: 27 tiles, 27 tokens.
  2. Rule 1: t + h. Wherever the two stand side by side, they become one token, in every word.
  3. Rule 2: th + e → the. A rule can glue a token that an earlier rule made.
  4. Rule 13 finds nothing to glue here, so the number jumps to 14.
  5. Rule 52: parc + el. "parcel" is now one token.
  6. All 60 rules: 27 letters, 10 tokens. "tomorrow" stays in 6 pieces: it is rare in the text.
const rules = learn(TEXT, 60);
const sentences = ["the parcel will arrive tomorrow", "when will my orders arrive", "complaint about a broken television"];
for (const s of sentences) {
  console.log(show(s, rules));
  console.log(`  letters ${s.replaceAll(" ", "").length}, tokens ${encode(s, rules).length}`);
}
Output
the  parcel  will  arrive  to|m|or|r|o|w
  letters 27, tokens 10
w|h|e|n  will  m|y  order|s  arrive
  letters 22, tokens 10
c|o|m|p|l|a|in|t  a|b|o|u|t  a  b|r|o|k|e|n  t|el|e|v|is|i|on
  letters 31, tokens 27

The familiar sentence is 10 tokens instead of 27 letters. "tomorrow" stays in six pieces: it is in the text only three times, and the rules go to more frequent pairs. In the second sentence "orders" becomes "order|s", and "when" and "my" are letters, but not one of them is "?".

The third sentence is about a complaint, and the shop text has no such words. 31 letters become 27 tokens: almost no gluing. New text can be read, but it costs more tokens.

The model sees the numbers of the tokens, just like the dictionary of words: every token has a number in the list of all tokens, and that list is the model's vocabulary.

Try it

Fill in

How far does i move when left and right are glued into one token?

// glue every left, right that stand side by side
function merge(tokens, left, right) {
  const glued = [];
  let i = 0;
  while (i < tokens.length) {
    if (tokens[i] === left && tokens[i + 1] === right) {
      glued.push(left + right);
      i += ;
    } else {
      glued.push(tokens[i]);
      i += 1;
    }
  }
  return glued;
}

The idea

The price and the limit are in tokens

A model reads and writes tokens. So everything measured about it is in tokens:

  • the price: providers charge per million tokens, separately for the input and the answer;
  • the input limit: how much text the model takes at once (this is the context window);
  • the answer limit: how many tokens it can write before it stops.

A colleague estimates the price by words. Here is how far off that is for the same three messages at an example price of €3 per million tokens:

const rules = learn(TEXT, 60);
const messages = ["the parcel will arrive tomorrow", "when will my orders arrive", "complaint about a broken television"];
let words = 0, tokens = 0;
for (const m of messages) {
  const w = m.split(" ").length, t = encode(m, rules).length;
  words += w;
  tokens += t;
  console.log(`${m}: ${w} words, ${t} tokens`);
}
console.log(`tokens per word: ${(tokens / words).toFixed(2)}`);
const price = 3; // € per million tokens, an example price
const month = 200000;
console.log(`${month} messages a month, €3.00 per million tokens (an example price):`);
console.log("  by words: " + "€" + (month * (words / messages.length) * price / 1e6).toFixed(2));
console.log("  by tokens: " + "€" + (month * (tokens / messages.length) * price / 1e6).toFixed(2));
Output
the parcel will arrive tomorrow: 5 words, 10 tokens
when will my orders arrive: 5 words, 10 tokens
complaint about a broken television: 5 words, 27 tokens
tokens per word: 3.13
200000 messages a month, €3.00 per million tokens (an example price):
  by words: €3.00
  by tokens: €9.40

The estimate by words is €3.00, by tokens €9.40: more than three times as much. The rare message weighs most: 5 words, 27 tokens. The price is an example, and the exact numbers come from the provider: the API response says how many tokens went in and came out. How to keep to a budget is in the lesson on cost.

Try it

Find the bug

A colleague writes their own BPE and gets different rules: "delivery" never becomes one token. Which line is the bug on?

Pick the line with the bug.

The idea

When it hurts

  • The rare costs more. Product names, order numbers, typos and words from another language are cut into small pieces. Real tokenizers are trained mostly on English text, so the same text in Bulgarian or Greek usually takes more tokens than in English.
  • The model does not see letters. "tomorrow" is a few pieces for it, not eight letters. That is why counting letters, reversing a word and arithmetic with long numbers go badly: "1042" is four tokens here, and with another model it may be two.
  • Every model has its own tokenizer. The same message is a different number of tokens with different models. Do not estimate the price by words and do not carry it over from one model to another: count with the model's tokenizer or read how many tokens the API reported.
  • The answer can stop in the middle. When the model reaches the answer limit, it stops at the token it got to, even in the middle of a word. See the exercise.
More precisely, if you are curious

Bytes instead of letters. The tokenizers of real models are usually BPE over the bytes of UTF-8, not over letters. That way any text goes through, even emoji and alphabets they have never seen. A Cyrillic letter is 2 bytes, so a rare Bulgarian word can come out as more tokens than it has letters.

The space is part of the token. Here every word is cut separately and spaces are not counted. Real tokenizers glue the space to the word after it: " will" with the space in front is one token. So "will" at the start of a text and "will" in the middle are different tokens.

How many tokens there are. The vocabulary of a real model is, as an example, from 50,000 to 200,000 tokens, learned from a huge amount of text. For English the rough rule is one token per about 4 characters. There are also special tokens that are not text, such as "end of text".

The real tools. Ready tokenizers are libraries: tiktoken and Hugging Face tokenizers in Python, and their versions for JavaScript, Java and C#. Work with models is mostly in Python, but a tokenizer is only a table of rules, and the rules are the same in every language. Provider SDKs also have a call that counts the tokens of a message before you send it.

Exercise

The reply cut at the limit

Solve it in your editor.

A model writes the answer token by token and stops when it reaches the answer limit. Do the same: take the reply, cut it into tokens word by word, and stop after 6 tokens. From the word that is cut, keep only the tokens that fit. Print how many tokens the whole reply is and what is left up to the limit.

Output
the reply: 15 tokens
limit 6, up to the limit: your parcel will arrive within t

"two" is three tokens, "t|w|o", and the limit has room only for the first. The customer sees "within t" and does not learn how many days. So the program checks whether the answer stopped at the limit, and either gives more room or asks the model for a shorter answer.

More tasks

  • More rules. Learn 100 and 200 rules. How many tokens do the three sentences of the lesson become? When does it stop helping?
  • The most expensive word. Find the word in the shop text that is the most tokens after 60 rules.

Test

Check yourself

1. The dictionary of whole words has seen "order" but not "orders". What does BPE do with "orders"?
2. Why are the rules applied to new text in the order they were learned?
3. A colleague estimates the price of the bot by words. Why is the estimate lower than the invoice?
4. The model says "tomorrow" has two letters "o", but it has three. What is the most likely reason?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·