Embeddings

A computer can’t compare “dosa” and “idli” as words. To a program they’re just different strings. An embedding fixes that: it turns each item into a vector (a list of numbers) so that items with similar meanings get similar numbers. Each vector is a point in an embedding space, and the distance between two points tells you how related they are.

You’ve seen embeddings already: inside an LLM, every token becomes an embedding before the transformer runs (see What is an LLM?). The same idea powers semantic search, recommendations and the “retrieval” step that grounds an agent in your documents.

The usual way to compare two embeddings is cosine similarity: how closely the two vectors point in the same direction. Close to 1 means very similar. Close to 0 means unrelated.

Press Play to walk through the map. Afterwards, tap any word to see its nearest neighbours.

What to notice

1. Meaning becomes geometry. Foods cluster together, cities cluster together, weather words cluster together. Nobody drew those groups: they fall out of the numbers.
2. There’s structure inside a cluster too. Chennai sits nearer Madurai than Mumbai, and idli nearer dosa than vada pav, because the vectors also carry a sense of region.
3. Directions can mean something. Subtracting Chennai and adding Mumbai moves “idli” towards Mumbai’s food. Real embedding models show patterns like this, though they’re never this neat.
4. Real dimensions have no names. The toy example labels its 8 dimensions so you can read them. Real word embeddings typically have 256, 512 or 1024 dimensions, learned during training, and single dimensions rarely mean anything a person could name.
5. Similarity is a choice of maths. Google’s guide describes three common measures: Euclidean distance, cosine and dot product. Cosine ignores the length of the vectors; dot product doesn’t. When vectors are normalised to length 1, all three rank neighbours the same way.