This series used language models as its teaching vehicle because text is where the methods were developed and are easiest to see. But the same ideas now train image, speech and video models, and systems that handle several at once. This post maps what carries over and what does not, without trying to cover each modality in depth; the site’s speech posts and Foundations series go further.
What carries over almost unchanged
- The Transformer works on any sequence of vectors. Once an image or a sound is a sequence, the decoder architecture applies with few changes.
- Training mechanics: the training update, learning-rate schedules, mixed precision, checkpoints and the diagnosis of a training run are the same.
- Scale: data, tensor and pipeline parallelism and sharding are indifferent to what the tokens mean.
- Data discipline: filtering, deduplication, mixture weights and decontamination, from the corpus and mixtures posts, apply to images and audio, though the filters differ.
- Scaling laws in the same form, fitted separately for each modality and objective.
- Post-training and evaluation habits: SFT on demonstrations, preference learning, held-out evaluations and suspicion of single benchmarks.
What changes: turning the signal into tokens
Text arrives as discrete symbols. Images and sound are continuous and much larger, so the first question for any modality is how to turn it into a sequence the model can handle, and how long that sequence is.
Images. The Vision Transformer cuts an image into fixed-size patches, 16 × 16 pixels, flattens each into a vector and treats the patches as a sequence (Dosovitskiy et al., 2021). A 224 × 224 image becomes 196 patches; a 1,024 × 1,024 image becomes 4,096, and because attention cost grows with the square of sequence length, resolution is expensive. Earlier vision models used convolutions, which build in the assumption that nearby pixels matter together; ViT drops that assumption and needs more data to compensate, but then scales well.
Speech. Audio is sampled thousands of times per second (24,000 for the codecs on this site), far too long a sequence to model directly. A neural audio codec compresses it: Mimi turns speech into 12.5 frames per second, and with residual vector quantization each frame becomes 8 discrete codes in Moshi’s setting, 100 tokens per second (Mimi post; Moshi review). Other systems skip discrete tokens and model continuous latents, as Pocket TTS does.
The token rate decides cost, exactly as tokenizer premiums did for languages in the tokenization post.
Try it: what fits in a context? An 8,192-token context holds how many 224 × 224 images at one token per patch? How many seconds of audio at 100 tokens per second? How much English text, at about 1.3 tokens per word?
Answer. 41 images; 82 seconds of audio; about 6,300 words of text. A minute of speech, about 150 words, is roughly 195 text tokens but 6,000 codec tokens: 31 times more. Long audio or high-resolution images strain context, KV cache and compute in a way text rarely does, which is why multimodal systems compress aggressively and why the inference arithmetic bites harder there.
What changes: the objective
Next-token prediction is not the only objective, and for some data it is not the best:
- Contrastive learning trains two encoders so that matching pairs end up close. CLIP learned image representations from 400 million image–caption pairs by predicting which caption goes with which image, and could then classify images zero-shot by comparing them with text descriptions of classes (Radford et al., 2021). The supervision is the pairing, not the next token.
- Denoising (diffusion and flow matching) learns to turn noise into data step by step, and dominates image, video and much audio generation; the Foundations post on diffusion and flow explains it.
- Encoder–decoder transcription maps one modality to text: Whisper is trained on 680,000 hours of audio paired with web transcripts, with next-token prediction on the text side only.
- Next-token prediction over codec tokens generates audio the way a language model generates text, as in speech language models.
The choice follows the data: is there a natural order to predict, paired data to align, or a continuous signal to reconstruct?
Connecting modalities to a language model
A common way to build a vision–language model reuses components: a pre-trained vision encoder, a pre-trained language model, and a small learned projection that maps image features into the language model’s embedding space, so image patches enter as if they were tokens. LLaVA built this way and instruction-tuned it on image–text conversations generated with GPT-4 from image captions (Liu et al., 2023). The training recipe then looks familiar: align the projection first, then fine-tune on instructions, as in the SFT post, with the same questions about data mixtures, masking and evaluation.
What changes: judging quality
Evaluation is where modalities differ most. Text has exact-match and judged quality; images are judged by human preference and by distributional measures; speech by intelligibility (word error rate of a transcription), similarity to a target speaker, and listening tests. The speech evaluation post walks through those metrics. The discipline from the application-evaluation post holds unchanged: real test sets, paired comparisons, error bars and suspicion of any single number.
Everyday picture: a factory that once made only books retooling for photographs and records. The presses, the logistics, the accounting and the quality control are reused; what changes is how each product is cut into pieces, how big each order is, and how a customer decides whether it is good.
What to remember
- Transformers, training mechanics, distributed training, data discipline, scaling laws and post-training transfer across modalities.
- The first design question for a new modality is tokenization: patches for images (196 for a 224 × 224 image), codec frames for speech (100 tokens per second for Mimi in Moshi).
- Token rates set cost: a minute of speech is about 31 times as many tokens as its transcript.
- Objectives follow the data: next-token prediction, contrastive alignment of pairs (CLIP), denoising (diffusion, flow) or transcription (Whisper).
- Vision–language models often join a vision encoder to a language model through a learned projection, then instruction-tune.
- Quality is judged differently per modality; the evaluation discipline is the same.
Sources
- A. Dosovitskiy, L. Beyer, A. Kolesnikov et al. (2021). “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” ICLR 2021. arXiv:2010.11929.
- A. Radford, J. W. Kim, C. Hallacy et al. (2021). “Learning Transferable Visual Models From Natural Language Supervision.” ICML 2021. arXiv:2103.00020.
- H. Liu, C. Li, Q. Wu and Y. J. Lee (2023). “Visual Instruction Tuning.” NeurIPS 2023. arXiv:2304.08485.
Further reading
- Stanford CS336, lecture 17, “Alignment, multimodality”.
- Berkeley CS294-280 (Spring 2025), lectures on multimodal agents (10 and 17 Mar).