Choosing a model: start from the task and build a baseline worth beating

The posts so far followed a model from raw text to a post-trained assistant. This track starts at the other end: you have a task, and existing models to build it with. You do not need the training posts to follow it; the Foundations post on language models is enough.

The first decision is which model to use, and the most common mistake is to make it before deciding how success will be measured. A model that tops a public leaderboard may be the wrong choice for summarizing your contracts in Khmer at a cost of a cent per request.

Start from the task, not the leaderboard

Write down the constraints before looking at models:

• Quality on your task, defined concretely: what makes an answer correct or acceptable, and how you will check it.
• Latency: how long a user can wait for the first token and for the whole answer.
• Cost per request at your expected volume.
• Context: how much input each request needs, measured in the model’s tokens.
• Languages and domains: a tokenizer that charges six times more tokens for your language, as the tokenization post measured for Khmer, raises cost and latency six times and shrinks the usable context.
• Deployment: whether data may leave your systems, whether you need open weights you can host and fine-tune, and what the licence allows.

These constraints often eliminate most candidates before any quality comparison.

Build a small evaluation set first

The core difficulty of evaluation, as CS336’s evaluation lecture puts it, is turning an abstract construct (“good summaries”) into a concrete metric. For an application the most valid metric comes from your own data: 50 to 200 real or realistic inputs, each with a reference answer or a clear rubric. Real inputs matter because public benchmarks are often far from real use; exam-style questions say little about customer-service replies (CS336 lecture 12, on ecological validity).

Then decide how to score: exact match or a checker where answers are verifiable, a rubric graded by a person or a model where they are not. The application-evaluation post covers scoring in depth; for choosing a model, a simple, consistent score on real examples is enough.

What leaderboards are good for

Public leaderboards narrow the field. Chatbot Arena collects pairwise votes: a user asks a question, two anonymous models answer, and the user picks the better one. By early 2024 it had over 240,000 votes, ranked with the Bradley–Terry model (Chiang et al., 2024), the same model used for reward models in the preference post. It reflects real prompts, but their distribution is whatever users happened to ask, and votes reward what users like, including length and style.

Static benchmarks (MMLU, GSM8K) measure specific skills reproducibly, but may be contaminated and may not resemble your task. Use both to choose a shortlist of three or four models, never to make the final choice.

Baselines

A baseline tells you whether anything you build later is an improvement. Three are worth running on the evaluation set:

1. The simplest non-LLM solution, if one exists: a keyword rule, a search engine, a template. If it scores close to the models, you may not need a model at all.
2. The strongest model you could use, with a plain prompt. This is roughly the ceiling for prompting alone, and tells you whether the task is feasible.
3. The cheapest model that might work, with the same prompt. The gap between 2 and 3 is what better prompting, retrieval or fine-tuning would have to close.

Keep the prompt identical across models at this stage, so the comparison measures models, not prompt-writing effort.

Try it: which model is cheaper? A request has 1,500 input and 300 output tokens. Model L costs $3 per million input tokens and$15 per million output tokens, and answers 92% of your evaluation set correctly. Model S costs $0.15 and$0.60, and scores 85%. Every wrong answer must be caught and fixed by a person, at about $0.50 each. Which is cheaper per 1,000 requests? (Prices are illustrative.)

Answer. Tokens: L costs $1{,}500 \times 3 + 300 \times 15$ = 9,000 millionths of a dollar per request, $9.00** per 1,000; S costs about **$0.40, 22 times less. Errors: L makes 80 per 1,000, costing $40 to fix; S makes 150, costing$75. In total L costs $49** and S **$75. The model that is 22 times more expensive per token is the cheaper system. If errors cost nothing, as in a low-stakes autocomplete, the answer flips. Cost decisions depend on what errors cost, which only you know.

Measure with error bars

On a small evaluation set, scores are uncertain. With $n$ examples and accuracy $p$, the standard error is $\sqrt{p(1-p)/n}$, so a 95% interval is about $\pm 2\sqrt{p(1-p)/n}$. At $n = 100$, 92% is uncertain by about ±5 points and 85% by about ±7: the two intervals overlap, and the measured gap could be noise. At $n = 1{,}000$ the intervals shrink to about ±2. Comparing both models on the same examples helps a lot, because what matters is how often they disagree, a point the application-evaluation post develops. Before switching models over a few points, check that the difference is larger than the noise.

Everyday picture: hiring. A CV and a reputation (the leaderboard) decide who gets an interview; a work sample on your actual job (your evaluation set) decides who gets hired; and you budget for the cost of mistakes, not only the salary.

What to remember

• List the constraints first: quality, latency, cost, context, languages, deployment and licence. They eliminate most candidates.
• Build 50–200 real examples with a scoring method before comparing models.
• Leaderboards (Chatbot Arena, static benchmarks) make a shortlist, not a decision.
• Run three baselines: a non-LLM solution, the strongest model with a plain prompt, and the cheapest plausible model.
• Count the cost of errors as well as tokens, and measure with error bars: about ±5 points at 100 examples.

Sources

• W.-L. Chiang, L. Zheng, Y. Sheng et al. (2024). “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference.” arXiv:2403.04132.
• Stanford CS336, lecture 12, “Evaluation” (lecture file lecture_12.py): constructs and metrics, ecological validity, train–test overlap.

Further reading

• High throughput, minutes of waiting: why a fast system can still keep users waiting.