Tokens vs. Embeddings: Two Completely Different Things

Many people talk about tokens and embeddings – often meaning “something to do with AI.” But they are two completely different things.

🔢 Tokens: The Text Building Blocks

Tokens are the text building blocks that a model works with. A sentence is broken down into small units (subwords, words, characters). The more text, the more tokens. Tokens are essentially a counting unit for input/output.

🧭 Embeddings: The Meaning Representation

Embeddings, on the other hand, are a meaning representation of text as a numerical vector. Imagine: “dog” doesn’t become “4 tokens,” but rather an arrow in a meaning space that places “dog” close to “puppy” and “animal” – and far away from “tax notice.”

💡 The Key Point

The length of an embedding does not depend on the number of tokens.

Why? Because an embedding typically has a fixed dimension (e.g., 768 or 1536 numbers) – regardless of whether you embed a single word or an entire paragraph. The model “compresses” the content into the same vector length, just like a photo can always be 1024x1024 pixels, no matter how much or how little is happening in the image.

What Depends on What

• What depends on tokens: Computational effort and context processing when generating the embedding (more tokens = more to process).
• What does not depend on tokens: The size/dimension of the embedding itself.

🔹 In Short

• Tokens = “How much text?” (counting unit)
• Embeddings = “What does the text mean?” (vector of fixed length)