Multimodal models
Gemini can read photos, listen to audio and watch video because all of them become tokens in the same sequence as your text. Follow a photo, a voice note and a video into the model, and count their tokens.
A multimodal model can take in more than one kind of input: text, images, audio, video and documents. Gemini is multimodal. You can send a photo with a question, a voice note with an instruction, or a video with a request, all in one prompt.
The trick is the same one you’ve seen for text (see Tokens and tokenizers). Each kind of media is turned into tokens, and those tokens go into the same sequence as your text. Then the transformer’s attention (see Attention and transformers) works across all of them, so a word in your question can relate to a corner of a photo or a moment in a recording.
Because media becomes tokens, it also uses your context window and is billed like tokens. The Gemini docs give the counts.
| Input | How it’s counted (Gemini API) |
|---|---|
| Image | 258 tokens if both sides are 384 px or less. Larger images are tiled into 768×768 tiles, 258 tokens each. |
| Audio | 32 tokens per second (1,920 per minute). |
| Video | Frames sampled at 1 per second by default, plus the audio. About 100 tokens per second at the default (low) media resolution. |
Press Play to follow a photo of a bill into the model. Then switch Try it to the voice note and the video, and compare the token counts in the log.
- Text
- Media
- Tokens
- Answer
Press Play to watch it run, or step through with ← →.
Token count
No tokens yet. Press Play.
What to notice
- One sequence, one model. You don’t call a separate image or speech service first. Media tokens and text tokens sit side by side, and the same model attends across them.
- Media is expensive in tokens. A one-line question is a few tokens. One photo is hundreds or thousands; a minute of video is thousands. Plan your context window and costs around that.
- Resolution is a trade-off. Gemini’s media resolution setting controls the maximum tokens per image or video frame. Higher resolution helps read small text and fine detail, but costs more tokens and time.
- Video is sampled, not watched. At the default 1 frame per second, very fast action between frames can be missed. Timestamps (MM:SS) let you ask about, and check, specific moments.
- Big files go through the Files API. Inline data is limited to 20 MB for the whole request. For larger files, or files you’ll reuse, upload them first and refer to them by URI.