Post-training data: curated prompts, synthetic examples, rejection sampling and distillation

The previous post treated fine-tuning data as given. In practice, assembling it is most of the work, and most of it is no longer written by hand. Prompts are collected and synthesized; responses are generated by strong models, filtered by other models or by programs, and selected from many candidates. Each step is a filter, and like the filters in the corpus post, each favours something. This post follows the pipeline and asks, at every stage, what it selects for.

Prompts: what the model is asked

The prompt pool decides which skills get practice. Tülu 3’s recipe starts from the skills to be improved, then collects prompts for each (Tülu 3, §3.1):

• Public datasets, chosen for diversity and clear provenance, including real user conversations such as WildChat.
• Synthetic prompts for target skills. To get diverse prompts from a model, which tends to repeat itself, Tülu 3 conditioned generation on about 250,000 different personas (“a machine learning researcher focused on neural networks”) combined with a task (“create a coding problem”). This produced about 30,000 instruction-following examples with verifiable constraints, about 220,000 math problems and 35,000 coding problems (§3.1.2).
• Safety and noncompliance prompts: requests the model should decline, and requests that look risky but should be answered.

The prompt pool is also where contamination enters. Real chat logs contain benchmark questions that users pasted in; Tülu 3 found several popular public datasets overlapping with test sets (App. B.2).

Responses: who writes the answer

For each prompt, someone or something writes the response:

• Humans, for the hardest or most sensitive cases. Expensive and slow.
• A stronger model, which is distillation at the level of data: the student is fine-tuned on the teacher’s outputs. Tülu 3 generated many of its responses with GPT-4o and Claude (§3.1.2).
• The model itself, sampled several times and filtered, which is rejection sampling.

Distillation is remarkably effective at transferring capabilities. DeepSeek-R1’s report compares two ways of making a 32B model reason: large-scale reinforcement learning on the base model for over 10,000 steps, against fine-tuning the same base on about 800,000 examples generated by the much larger DeepSeek-R1. The distilled model scored 72.6 on AIME 2024 against 47.0 for the RL-trained one (DeepSeek-R1, App. F.1, Table 16). The authors conclude that distilling a stronger model into a smaller one is economical and effective, while advancing beyond what existing models can do may still require stronger base models and large-scale RL. Distillation copies the teacher’s mistakes and habits along with its skills, and the student rarely surpasses it.

Rejection sampling: choosing among the model’s own answers

Rejection sampling (also called best-of-K sampling) generates $K$ responses per prompt and keeps the best one, judged by a reward model or a verifier, then fine-tunes on the kept responses. Llama 3 sampled $K$ between 10 and 30 outputs per prompt from its latest model and kept the one its reward model scored highest, repeating the process over six rounds of post-training (Llama 3, §4.1.6, §4.2.2). Sharing the prompt’s key-value cache across the $K$ samples doubled the throughput of this step, a direct use of the KV-cache arithmetic.

With a verifier, for example a program that checks a math answer, a prompt yields a usable example only if at least one of $K$ samples passes. If each sample passes independently with probability $p$,

Larger $K$ recovers more prompts, but prompts the model rarely solves stay rare.

Try it: what survives? Your prompt pool has 1,000 easy prompts (the model solves each sample half the time) and 1,000 hard ones (2% of the time). With $K = 16$ and a perfect verifier, what fraction of the kept examples are hard?

Answer. Easy: $1{,}000 \times (1 - 0.5^{16}) \approx 1{,}000$ kept. Hard: $1{,}000 \times (1 - 0.98^{16}) \approx 276$. Hard prompts were half the pool but are 22% of the kept data, after 32,000 generations. Rejection sampling teaches the model most about what it can already do. Remedies include larger $K$ for hard prompts, a stronger teacher for them (distillation), or keeping hard prompts for reinforcement learning, where partial progress can still produce a learning signal.

Keeping synthetic data honest

Generated data has failure modes of its own:

• Collapse of diversity. Models repeat patterns; without deliberate variation (personas, varied seeds and prompts), a synthetic set covers a narrow slice of the task.
• Inherited errors. A teacher’s wrong answers become training targets. Verifiers catch some (math, code, format constraints); open-ended answers need a judge or a human.
• Contamination. A generator can reproduce benchmark items it has seen, and prompt pools collected from users contain them.

Tülu 3’s decontamination is a concrete, reproducible procedure (§3.2). It compares training prompts with evaluation prompts (responses are often regenerated, so they are not compared) using 8-word overlaps: a test item counts as matched by a training example if more than half of its tokens fall inside 8-grams shared with that example. Training sets matching more than 2% of an evaluation’s items were removed or filtered. The team tried embedding-based matching too, but found it hard to tell real paraphrases from mere similarity of topic, the same limitation the corpus post described. The rule catches near-copies: a word problem with only its numbers changed shares 53% of its tokens in 8-grams with the original and is flagged, while a reworded version shares none and is not.

Everyday picture: a kitchen that tastes every dish before it leaves. It keeps quality up, but only for the dishes the cook can already make; a dish that comes out right one time in fifty rarely reaches a customer, so the menu drifts towards what is easy.

What to remember

• The prompt pool decides which skills are practised; synthetic prompts need deliberate diversity, such as persona conditioning.
• Distillation fine-tunes a student on a teacher’s outputs; it transfers skills cheaply (a distilled 32B model beat RL on the same base) and copies the teacher’s errors.
• Rejection sampling keeps the best of $K$ samples; a prompt survives with probability $1 - (1-p)^K$, so hard prompts are under-represented in the result.
• Decontaminate prompts against evaluations; n-gram overlap catches near-copies, not rewordings.

Sources

• N. Lambert et al. (2024). “Tülu 3: Pushing Frontiers in Open Language Model Post-Training.” arXiv:2411.15124. §§3.1–3.2, App. B.2.
• Llama Team, AI @ Meta (2024). “The Llama 3 Herd of Models.” arXiv:2407.21783. §§4.1.6, 4.2.2.
• DeepSeek-AI (2025). “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv:2501.12948. App. B.3.3, F.1.

Further reading

• Tülu 3 review.
• Stanford CS336, lecture 14, post-training data: synthetic data and what it resembles.