Tokens and tokenizers
Models read and write tokens, not words. See how a tokenizer cuts up English, place names, prices, code and Tamil, and why the token count decides your limits and your bill.
A model can’t read letters. Before your text reaches it, a tokenizer cuts the text into tokens and swaps each one for a number, its ID in the tokenizer’s vocabulary. A token can be a whole word, part of a word, a single character or a piece of punctuation.
Tokenizers learn their vocabulary from a large amount of text. Common words get their own token. Rarer words are built from smaller pieces the tokenizer has seen often. That’s why “Chennai” can be two tokens (“Chen” + “nai”) while “weather” is one.
This matters because everything is counted in tokens: how much fits in the context window, how fast you hit rate limits, and how much you pay. For Gemini models, the docs give a rule of thumb: a token is about 4 characters, and 100 tokens is about 60 to 80 English words.
Press Play, then switch Try it between the examples, or type your own text.
- Text
- Split
- Token IDs
- Count + cost
Press Play to tokenize.
Models never see letters or words. Before anything else, a tokenizer cuts the text into tokens. Press Play, pick another example, or type your own text.
This is a toy tokenizer with a few hundred pieces, so its splits and counts are illustrative. Real tokenizers learn their vocabulary from data and differ between models. To get the real count for Gemini, use the API's count_tokens method. Token IDs and the price are made up.
What to notice
- Spaces belong to tokens. The space usually sticks to the front of the next word (shown as ·). “·weather” and “weather” can be different tokens.
- Rare words and names cost more. Common English words are one token each. Place names, product codes and invented words split into several.
- Numbers, symbols and code are expensive. Digits, brackets, colons and underscores often become separate tokens, so a price list or a block of code uses more tokens than its length suggests.
- Language matters. Text in many languages takes more tokens per word than English, because those languages were less common in the text the tokenizer learned from. The same question can cost more in Tamil than in English.
- Images and audio are tokens too. In the Gemini API, a small image (384 pixels or less on both sides) counts as 258 tokens, and audio counts as 32 tokens per second. See Multimodal models.
Counting tokens for real
Don’t guess for anything that matters. The Gemini API has a count_tokens method (client.models.count_tokens(...) in Python) that returns the number of tokens in your input before you send it. After a call, the response also reports the input, output and total tokens actually used. The toy tokenizer above is only for building intuition: real tokenizers differ between models.