The series took decisions one stage at a time: data, tokenization, architecture, scale, distributed training, post-training, serving and application evaluation. In a real project they are not independent. A choice made early fixes costs that appear much later, and a requirement discovered late can only be met cheaply if it was anticipated early. This short post traces the connections.
The main couplings
| Decision | What it constrains later |
|---|---|
| Tokenizer | Compute and context per language; vocabulary parameters; how much of the corpus a token budget covers; serving cost per request |
| Data mixture | Which capabilities the base model has; what post-training can draw out; what evaluation will find |
| Model size and token budget | Training cost, but also serving latency and memory for the model’s whole life |
| Attention heads (GQA) | KV-cache size, and therefore batch size, throughput and cost when serving |
| Context length | Training memory and cost; position-encoding choices; the usefulness of retrieval |
| Post-training recipe | Format, refusal and reasoning behaviour that applications rely on; regressions applications must test |
| Evaluation set | What every earlier decision is optimized towards |
Tracing a requirement: serving Khmer users cheaply
Suppose the requirement is a Khmer-language assistant that answers in under two seconds at low cost.
- Tokenizer. Under an English-centred tokenizer, Khmer cost 15 times as many tokens as English for the same content; under a multilingual one, 6.5 times (tokenization post). Every later cost is multiplied by that factor, so a tokenizer with Khmer pieces is the single largest saving available, and it can only be chosen before pre-training.
- Data. Khmer is a low-resource language. Its share of the mixture, and how many times it is repeated, decide how well the model learns it (mixtures post). Language identification must not discard mixed Khmer–English pages (corpus post).
- Size. A model that serves many requests should be smaller and trained longer than the compute-optimal size (scaling laws), but a small model has less capacity for a language it saw little of. This tension has to be measured, not assumed.
- Architecture and serving. Long Khmer token sequences fill the KV cache sooner, so grouped-query attention and quantization matter more than for English (inference post).
- Evaluation. The test set must be in Khmer, with Khmer-speaking judges or validated automatic metrics; English benchmarks say little about the requirement.
A team that fixes the tokenizer and mixture first and discovers the Khmer requirement at step 5 can only patch it with fine-tuning and retrieval, at higher serving cost for the life of the product.
Tracing a requirement: reliable mathematical answers
Now the requirement is that answers to arithmetic and algebra problems are right, not just plausible.
- Data and base model. Mathematical skill comes mostly from pre-training data; DeepSeekMath’s 120-billion-token math corpus did much of the work before any RL (DeepSeekMath review). The same SFT mix gave 31.5 on MATH from one base model and 56.4 from a math-specialized one (SFT post).
- Post-training. Answers are checkable, so verifiable rewards and RL fit (RL post), with prompts chosen at the edge of the model’s ability so groups carry signal.
- Inference. Sampling several solutions and voting improves accuracy (prompting post), but multiplies serving cost; a calculator tool makes arithmetic exact for the price of a tool call (retrieval and tools).
- Evaluation. Pass@1 for users, Pass@K and Maj@K to tell sharpening from new capability, and fresh problems the training data could not contain.
Here the cheapest reliable system may combine a moderately sized model, RL with a verifier, and a tool, rather than a much larger model.
A real example: Llama 3’s interlocking choices
Llama 3’s report is useful precisely because its decisions visibly anticipate later stages (Llama 3, §3):
- a 128,000-token vocabulary, adding 28,000 tokens for non-English languages, improved English compression from 3.17 to 3.94 characters per token, letting the model read more text for the same training compute (§3.2);
- grouped-query attention with 8 key-value heads to reduce the KV cache and speed up decoding (§3.2);
- a flagship size chosen from scaling laws fitted on its own data mix, with the flat optimum used to pick a convenient 405B (§3.2.1), while smaller models were trained far longer for cheaper serving;
- annealing on high-quality data, which helped the 8B model more than the 405B, and a document mask that mattered for long-context training (§3.1.3, §3.2).
None of these is remarkable alone. Together they show the habit this series has tried to teach: every choice is judged by what it costs or enables at the other stages.
Try it: trace a requirement. Your application must cite the document behind every answer, from a collection that changes daily. Which stages does that requirement reach?
Answer. It is mostly a retrieval requirement: knowledge that changes daily belongs in an index, not in weights, and citations need retrieved passages (adaptation post). It reaches context length and serving (passages consume tokens and KV cache), prompting or fine-tuning for a consistent citation format, evaluation (does the cited passage actually support the claim?), and security (retrieved documents can carry injected instructions). It barely touches pre-training: a model that follows instructions and handles long contexts is enough.
What to remember
- Stages are coupled: the tokenizer, mixture, size and attention design fix costs that appear at serving time.
- Start from the requirement hardest to meet in production, and trace it back through evaluation, serving, post-training, architecture, data and tokenizer before fixing any stage.
- Some requirements can only be met cheaply early (tokenizer, language coverage); others belong late (fresh knowledge in retrieval).
- Read technical reports for how decisions interlock, not only for their final numbers.