Skip to content
wannacode

Embeddings: meaning as vectors

In short: An embedding is a vector of numbers for a word or a text, so that texts close in meaning have close vectors, and closeness is measured by the cosine of the angle.

Problem

Search by words

The support bot has a base of ready answers: one text each for delivery, returns and payment. The customer writes a question, and the bot has to find the text that answers it.

// the answer base of the support team
const DOCS = [
  { topic: "delivery", text: "delivery takes up to two days" },
  { topic: "returns", text: "return the item within 14 days and get the money" },
  { topic: "payment", text: "pay by card or by bank transfer" },
];


// the questions of the customers
const QUESTIONS = [
  "when will my parcel come",
  "why was my payment refused",
];

A colleague searches by common words: the more words of the question are in the text, the better.

// how many words of the question are also in the text
function common(question, text) {
  const ws = text.split(" ");
  return question.split(" ").filter((word) => ws.includes(word)).length;
}
for (const q of QUESTIONS) {
  console.log(q);
  let best = null;
  let top = 0;
  for (const d of DOCS) {
    const n = common(q, d.text);
    console.log(`  ${d.topic.padEnd(10)}${n}`);
    if (n > top) {
      top = n;
      best = d;
    }
  }
  console.log("  → " + (best ? best.topic : "none"));
}
Output
when will my parcel come
  delivery  0
  returns   0
  payment   0
  → none
why was my payment refused
  delivery  0
  returns   0
  payment   0
  → none

Neither question finds anything. "parcel" is not in any text, and for the program "payment" and "pay" are different words. Customers ask in their own words, and the texts in the base are written in others.

What is needed is a comparison of meaning, not of letters.

The idea

Meaning as a direction

A word is known by the company it keeps. "parcel" and "order" appear with the same words ("courier", "arrive", "days"), so they mean something close, although their letters are different.

This is written down with numbers. Every word becomes a vector: a list of numbers, one for each axis. Words with a close meaning get close vectors, that is, arrows that point almost the same way. Such a vector, which stands for meaning, is called an embedding.

In the lesson the axes are the eight most frequent words in the shop text, and the number for each axis is in how many sentences the word is together with the axis word. Real models learn the vectors with a neural network and have hundreds or thousands of axes, but the idea is the same: close meaning, close direction.

See it

Words as vectors

Short words such as "the", "to" and "will" are everywhere and say nothing about the topic. They are not axes and do not go into the vectors:

// the short words that are everywhere and say nothing about the topic
const STOP = new Set([
  "the", "a", "an", "your", "you", "to", "with", "at",
  "by", "of", "in", "is", "was", "will", "be", "and",
  "or", "but", "it", "i", "am", "very", "on", "over",
  "within", "up", "after", "before", "one", "two", "can", "my",
  "when", "why",
]);

The space: the axes and the vector of a word. The vector of a text is the sum of the vectors of its words.

// words as vectors: one number for each axis
class Space {
  // the axes: the most frequent words, without the short ones
  constructor(text, stop, n) {
    this.text = text;
    this.stop = stop;
    const freq = new Map();
    for (const line of text)
      for (const w of line.split(" "))
        if (!stop.has(w)) freq.set(w, (freq.get(w) ?? 0) + 1);
    this.vocab = [...freq.keys()];
    this.axes = [...this.vocab]
      .sort((a, b) => freq.get(b) - freq.get(a))
      .slice(0, n);
  }


  // in how many sentences the word is together with each axis word
  of(word) {
    const v = new Array(this.axes.length).fill(0);
    for (const line of this.text) {
      const ws = line.split(" ");
      if (!ws.includes(word)) continue;
      for (const w of ws) {
        const j = this.axes.indexOf(w);
        if (j >= 0 && w !== word) v[j]++;
      }
    }
    return v;
  }


  // the vector of a text: the sum of the vectors of its words
  ofText(s) {
    const v = new Array(this.axes.length).fill(0);
    for (const w of s.split(" ")) {
      if (this.stop.has(w)) continue;
      this.of(w).forEach((x, j) => (v[j] += x));
    }
    return v;
  }
}
const space = new Space(TEXT, STOP, 8);
console.log("axes: " + space.axes.join(" "));
for (const word of ["parcel", "order", "payment"])
  console.log(word.padEnd(12) + `[${space.of(word).join(", ")}]`);
Output
axes: courier item delivery order days card arrive parcel
parcel      [1, 0, 0, 0, 1, 0, 1, 0]
order       [2, 0, 0, 0, 1, 0, 2, 0]
payment     [0, 0, 0, 0, 0, 2, 0, 0]

"parcel" is in one sentence with "courier", one with "days" and one with "arrive". "order" is there two, one and two times: the numbers are different, but the direction is almost the same. "payment" is only with "card": a completely different direction.

See it

Similarity: the cosine of the angle

How close two vectors are is told by the angle between them. The cosine of the angle is 1 when they point the same way, and 0 when they have no axis in common. Dividing by the lengths makes only the direction matter.

// cosine of the angle: 1 same way, 0 nothing shared
function cosine(a, b) {
  let dot = 0;
  let na = 0;
  let nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  if (na === 0 || nb === 0) return 0;
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}
const space = new Space(TEXT, STOP, 8);
const v = space.of("parcel");
console.log(`parcel and order: ${cosine(v, space.of("order")).toFixed(3)}`);
console.log(`parcel and payment: ${cosine(v, space.of("payment")).toFixed(3)}`);
const others = space.vocab
  .filter((o) => o !== "parcel")
  .sort((x, y) => cosine(v, space.of(y)) - cosine(v, space.of(x)));
console.log("nearest to parcel:");
for (const o of others.slice(0, 5))
  console.log(`  ${o.padEnd(12)}${cosine(v, space.of(o)).toFixed(3)}`);
Output
parcel and order: 0.962
parcel and payment: 0.000
nearest to parcel:
  order       0.962
  delivery    0.707
  three       0.667
  item        0.667
  address     0.577

"parcel" and "order" are 0.962: almost the same direction. With "payment" they are 0.000. The words nearest to "parcel" are all about delivery. There is also "three": in a small text chance neighbours pull too.

Try it

Predict the output

What is the cosine of [1, 1, 0] and [1, 0, 0]?

console.log(cosine([1, 1, 0], [1, 0, 0]).toFixed(2));
Answer

See it

Search by meaning

Now the question and every text in the base become vectors, and the answer is the text with the largest cosine:

const space = new Space(TEXT, STOP, 8);
for (const q of QUESTIONS) {
  const v = space.ofText(q);
  console.log(q);
  let best = DOCS[0];
  let top = -1;
  for (const d of DOCS) {
    const c = cosine(v, space.ofText(d.text));
    console.log(`  ${d.topic.padEnd(10)}${c.toFixed(3)}`);
    if (c > top) {
      top = c;
      best = d;
    }
  }
  console.log(`  → ${best.topic}: ${best.text}`);
}
Output
when will my parcel come
  delivery  0.696
  returns   0.347
  payment   0.000
  → delivery: delivery takes up to two days
why was my payment refused
  delivery  0.000
  returns   0.121
  payment   1.000
  → payment: pay by card or by bank transfer

The first question finds "delivery" with 0.696, while "returns" is 0.347. The second finds "payment" with 1.000: "payment" is in sentences with "card", just like the payment text. Neither question has a word in common with its answer.

Here is the first question as directions. The vectors are in 8 dimensions and the picture is flat, so it keeps only the angle of each text to the question:

// cosine of the angle: 1 same way, 0 nothing shared
function cosine(a, b) {
  let dot = 0;
  let na = 0;
  let nb = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    na += a[i] * a[i];
    nb += b[i] * b[i];
  }
  if (na === 0 || nb === 0) return 0;
  return dot / (Math.sqrt(na) * Math.sqrt(nb));
}


const space = new Space(TEXT, STOP, 8);
for (const q of QUESTIONS) {
  const v = space.ofText(q);
  console.log(q);
  let best = DOCS[0];
  let top = -1;
  for (const d of DOCS) {
    const c = cosine(v, space.ofText(d.text));
    console.log(`  ${d.topic.padEnd(10)}${c.toFixed(3)}`);
    if (c > top) {
      top = c;
      best = d;
    }
  }
  console.log(`  → ${best.topic}: ${best.text}`);
}
Step by step, in words
  1. The question "when will my parcel come" becomes a vector: [2, 0, 0, 0, 1, 0, 1, 0]. The picture keeps only the angle to it.
  2. "payment": cosine 0.000, angle 90°. Not a single axis in common: the arrows are at a right angle.
  3. "returns": cosine 0.347, angle 70°. Some axes in common, but far.
  4. "delivery": cosine 0.696, angle 46°. The smallest angle.
  5. Nearest is "delivery": "delivery takes up to two days". Not one word in common with the question.

Try it

Fill in

The cosine divides the dot product by the two lengths. What goes in place of ___?

return dot / (Math.sqrt(na) * Math.sqrt());

Try it

Find the bug

A colleague computes the similarity faster, but gets 8 for the first question and the delivery text, and the similarity should be between 0 and 1. Which line is the bug on?

Pick the line with the bug.

The idea

Attention as an idea

In the lesson "card" has one vector wherever it is. But in "pay by card" and "a birthday card" the word means different things: a bank card and a greeting card.

Real models solve this with attention. Every token looks at the other tokens in the text and mixes a little of theirs into its own vector, more from the ones that matter to it. "card" next to "pay" takes from "pay", next to "birthday" from "birthday", and the two vectors become different. This repeats through many layers, and so the vector of every token carries its meaning in exactly this sentence. The vector of the whole text is computed from them.

Here attention is only an idea: how exactly it is computed is not part of the course.

The idea

When it hurts

  • The small model knows words, not meaning. A customer wants their money back, and the search sends them to payment:

    const space = new Space(TEXT, STOP, 8);
    for (const q of ["i want my money back"]) {
      const v = space.ofText(q);
      console.log(q);
      let best = DOCS[0];
      let top = -1;
      for (const d of DOCS) {
        const c = cosine(v, space.ofText(d.text));
        console.log(`  ${d.topic.padEnd(10)}${c.toFixed(3)}`);
        if (c > top) {
          top = c;
          best = d;
        }
      }
      console.log(`  → ${best.topic}: ${best.text}`);
    }
    Output
    i want my money back
      delivery  0.426
      returns   0.647
      payment   0.667
      → payment: pay by card or by bank transfer

    "want" and "my" say nothing to the model. "money" is in sentences with "item", "days" and "card", and "back" is in "the money goes back to your card". So the vector of the question is [0, 1, 0, 0, 2, 2, 0, 0] and leans to "card": payment wins with 0.667, and returns, the right answer, is second with 0.647. "i want to return the item" itself finds returns in English, but in Bulgarian the same question goes to delivery: the misses differ between the locales because the texts differ.

    A real embedding model is trained on a huge amount of text. It knows that "money back" means a refund, and it sees the whole question, not only separate words. It can still miss, but much more rarely. How often it finds the right text is measured: in the module on search by meaning this is recall@k, the share of questions whose right text is among the first k found.

  • Close does not mean the same. Words with the opposite meaning often stand in the same sentences:

    const space = new Space(TEXT, STOP, 8);
    for (const [x, y] of [["free", "costs"], ["fast", "late"]])
      console.log(`${x} and ${y}: ${cosine(space.of(x), space.of(y)).toFixed(3)}`);
    console.log(`colour: [${space.of("colour").join(", ")}]`);
    Output
    free and costs: 0.632
    fast and late: 0.408
    colour: [0, 0, 0, 0, 0, 0, 0, 0]

    "free" and "costs" are 0.632, because both are next to "delivery". Real models do better, but "can" and "cannot" still come out close.

  • An unknown word is zero. "colour" is not in the text and its vector is all zeros. A question with only unknown words is 0.000 to everything, but the search still returns some text. That is why a threshold is needed: see the exercise.

  • The vectors come from the text. A small text gives chance neighbours ("three" next to "parcel"). Different models give different vectors that cannot be compared with each other: the base and the question must use the same model.

  • Many texts, many vectors. For thousands of documents the vectors are kept in a database that finds the nearest ones fast: vector databases. Search by meaning over the shop documents is a whole module later in the course. How to run an embedding model on a server and how to keep the index up is a job for the ZeroToOps course.

More precisely, if you are curious

Real embeddings. Embedding models take text and return hundreds or thousands of numbers. They are called through an API like other models, or run locally, most often from Python. The vectors usually have length 1, and then the cosine is just the dot product.

The zeros on the axes. In the lesson a word does not count its own axis. So an axis word that appears only with words outside the axes is all zeros: "card" is an axis, but its vector is [0, 0, 0, 0, 0, 0, 0, 0], and the word "card" itself does not help the search. The payment text is found through "pay", "bank" and "transfer", which are in sentences with "card". If a word counted its own axis too, that would become the biggest number in its vector and pull everything towards itself: "parcel" and "order" would fall from 0.962 to 0.197. Both choices are trade-offs of the small model; real models have no axes that are words.

Why cosine and not distance. The distance between the tips depends on the length too, and here the length is how many words the text has. The cosine looks only at the direction. For vectors of length 1 both give the same order.

The nearest k. The search in the lesson is k-NN from the lesson on the nearest neighbours with k = 1: only the vectors and the measure of closeness are different.

Exercise

Not sure is an answer

Solve it in your editor.

The bot must not answer at random. Write a function that returns the nearest text only if the cosine is at least 0.5, and otherwise "not sure, passing it to a colleague". Try it with the two questions from the lesson and with "can i change the colour".

Output
when will my parcel come → delivery takes up to two days
why was my payment refused → pay by card or by bank transfer
can i change the colour → not sure, passing it to a colleague

The third question is 0.000 to everything, and instead of a wrong answer the bot passes it to a person. The threshold is a trade-off: a high threshold passes on questions the bot knows the answer to, and a low one lets wrong answers through.

More tasks

  • More axes. Build the space with 16 axes. How do the similarities from the lesson change?
  • The first two. Print the two nearest texts for each question and how far the second is from the first.

Test

Check yourself

1. What is an embedding?
2. Two vectors point the same way, but one is three times as long. What is their cosine?
3. Why does the search by meaning find "delivery takes up to two days" for "when will my parcel come"?
4. "free" and "costs" have a similarity of 0.632. What does that say?

Made by Ivo Gergov as part of Evo Tech's campaign for accessible programming education ·