Blog
Explainers and field notes on speech AI, LLM inference and representation learning: how the systems work, and what the evidence does and does not show.
All posts · 57
Showing 57 of 57 posts
Post 3 of 26
A training corpus is not one dataset but several, and the sampling weights decide how many times the model sees each one. Turn weights into token exposure, see how temperature sampling and epoch caps protect small sources, and learn what is and is not known about repeating data.
Post 4 of 26
The tokenizer is trained before the model and fixed after it, and it quietly decides how much every sentence costs. Follow text through a real tokenizer, compare BPE with Unigram, measure what the same 1,012 sentences cost in English and Khmer under three tokenizers, and see what goes wrong when tokenizers and text disagree.
Post 5 of 26
A modern language model is still a stack of Transformer blocks, but a handful of choices decide how stably it trains, how much it costs to run and how long a context it handles. Count where Llama 3 8B's parameters go, see why grouped-query attention shrinks the KV cache fourfold, and learn what mixture-of-experts changes.
Post 6 of 26
Train a one-million-parameter language model on a laptop CPU, end to end: tokenizer, data splits, model, AdamW with warmup and cosine decay, gradient clipping, validation, checkpoints that resume exactly, and a precision check. Then break it on purpose and read the curves.
Post 7 of 26
Before a large run starts, someone has to decide how big the model should be and how many tokens it should see. Learn where the 6ND estimate of training compute comes from, what Chinchilla's experiments say about splitting a budget, why the answer changes once the model has to be served, and what small experiments can and cannot predict.
Post 8 of 26
Before splitting training across many GPUs, it helps to know exactly what one GPU holds and why it is slow. Count the bytes of weights, gradients, optimizer state and activations, estimate training time from FLOPs and utilization, and use arithmetic intensity to tell compute-bound work from memory-bound work.
Post 9 of 26
The simplest way to use many GPUs is to give each a copy of the model and a different slice of the batch, then average the gradients. Learn what has to stay identical for that to compute the same update as one big GPU, why averaging per-GPU losses quietly changes the objective, and what the gradient all-reduce costs.
Post 10 of 26
Data parallelism gives every GPU a full copy of the training state, which stops working once that state outgrows a GPU. Learn how ZeRO and FSDP shard weights, gradients and optimizer state across data-parallel GPUs, what each stage saves and costs in communication, and how activation recomputation trades compute for memory.
Post 11 of 26
When one layer's matrices, one sequence's activations or one model's experts no longer fit or run fast enough on a GPU, the model itself is split. Learn how tensor parallelism splits a matrix multiplication exactly, why pipelines have bubbles and how microbatches shrink them, and what sequence, context and expert parallelism add.
Post 12 of 26
Two ideas make modern training fast without changing what it computes: move less data, and use fewer bits. See how FlashAttention computes exact attention without ever storing the full score matrix, why BF16 replaced FP16, what loss scaling was for, and what FP8 training needs to work.
Post 13 of 26
A large run is a machine that fails. Learn to measure whether it is running well (MFU), find where time goes with a profiler, plan checkpoints around how often thousands of GPUs break, and choose a parallel layout by working through memory first and communication second, with Llama 3's real configuration as the example.
Post 14 of 26
A pre-trained model continues text; it does not answer. Supervised fine-tuning teaches it to, with the same next-token loss on example conversations. See what a chat template really is, which tokens the loss is computed on and why, how averaging choices reweight examples, and what the open Tülu 3 recipe found about data.
Post 15 of 26
Most post-training data is now generated, filtered or chosen by models. Learn where prompts come from, how synthetic data is kept diverse, why best-of-K rejection sampling quietly favours easy prompts, how distillation transfers a strong model's behaviour, and how decontamination works in an open recipe.
Post 16 of 26
People find it easier to say which of two answers is better than to write the perfect one. Learn how pairwise preferences become a training signal: the Bradley–Terry model behind reward models, the KL-regularized objective that keeps a policy close to where it started, and how DPO optimizes that objective without a separate reward model.
Post 17 of 26
Supervised fine-tuning and DPO learn from fixed examples. Reinforcement learning lets a model learn from its own attempts, scored by a reward. Derive the policy gradient on a four-answer toy, see why baselines matter, learn what PPO's clipping and value model do and what GRPO replaces them with, and why verifiable rewards changed reasoning training.
Post 18 of 26
Every post-training method optimizes a proxy: a reward model, a verifier, a benchmark. Learn how optimizing a proxy too hard makes the real thing worse, why longer answers win judgements they should not, how a recipe can overfit its own development benchmarks, and how to evaluate post-training so these failures show up before users find them.
Post 19 of 26
Before prompting, retrieval or fine-tuning, an application needs two things: a way to measure success on its own task, and a baseline. Learn which constraints narrow the choice of model, why leaderboards are a starting point rather than an answer, and how error costs can make the expensive model the cheap one.
Post 20 of 26
A prompt is the cheapest way to change a model's behaviour, and the easiest to get subtly wrong. Learn what instructions, examples and reasoning steps actually do, how to budget and order a context window, why sampling several answers and voting helps, and how constrained decoding guarantees valid structured output.
Post 21 of 26
A model only knows what it was trained on and can only produce text. Retrieval brings in information it never saw; tool calls let its text trigger actions. Build a tiny retriever and watch it fail on synonyms, see how a tool call works, and work out why multi-step agents fail more often than their individual steps.
Post 22 of 26
Prompting, retrieval and fine-tuning solve different problems: missing instructions, missing knowledge and missing behaviour. Learn to tell which one a task needs, then see how LoRA fine-tunes a large model by training a tiny low-rank update, why it starts from no change at all, and why it costs nothing extra at inference.
Post 23 of 26
Serving a model is a different problem from training it: generation is sequential and limited by memory bandwidth, not arithmetic. Work out why one sequence cannot exceed about 200 tokens per second on an H100, how batching trades latency for throughput, why the KV cache sets how many users fit, and what quantization, paged memory and speculative decoding change.
Post 24 of 26
An application built on a model is never finished: prompts change, models are swapped, users ask new things. Learn to build an evaluation that tracks what matters, use a model as a judge without inheriting its biases, tell a real improvement from noise, learn from production, and recognize the privacy and security risks that arrive with retrieval and tools.
Post 25 of 26
Each post in this series took one decision at a time. Real systems are constrained end to end: a requirement at the application layer reaches back into the tokenizer, the data mix, the architecture and the training budget. Trace two requirements through the whole stack, and read Llama 3's design as a set of interlocking choices.
Post 26 of 26
Almost everything in this series was taught with language models. Much of it transfers directly to images, speech and multimodal systems; some of it does not. Learn which principles carry over, how images and sound become tokens, where objectives and architectures differ, and why a minute of audio can cost thirty times more tokens than its transcript.
Post 2 of 26
Most of the web is not text you want a model to learn from. Follow a crawl through the stages that turn it into a training corpus (extraction, language identification, quality filters, deduplication, contamination checks and personal-data removal) and learn to judge each one the way that matters: under a fixed training budget, did it improve the model, and what did it cost in coverage and data?
Post 1 of 26
Everything in pre-training, and everything a distributed system must preserve, starts from one update. Follow it from a batch of token IDs to a changed weight: shifted targets, attention masks, packing, how the loss is averaged, the backward pass, gradient clipping and one AdamW step, each on numbers small enough to check in your head.
Post 1 of 15
A neural network is a pile of tiny calculators that multiply, add and bend. This post builds one with two inputs, two hidden neurons and one output, runs it forward, blames each weight for the error, and nudges every weight once, with every number small enough to check by hand.
Post 2 of 15
Training needs two things: one number that says how wrong the network is, and a rule for nudging every weight to make that number smaller. This post builds both, with every step small enough to check by hand.
Post 3 of 15
Plain gradient descent zigzags in narrow valleys, crawls on flat ground and uses one step size for every weight. This post fixes each problem in turn, momentum, per-weight step sizes, Adam and schedules, on the same small bowl from the last post, with every step worked by hand.
Post 4 of 15
A network can learn its training data perfectly and still be useless on anything new. This post shows that failure on eight noisy points, then works through the standard cures, weight decay, early stopping and dropout, with every number small enough to check by hand.
Post 5 of 15
Numbers on wildly different scales make training slow and fragile. Normalisation rescales them to a common size, inside the network as well as at the input. This post works batch norm, layer norm and RMSNorm by hand on the same four-by-three table.
Post 6 of 15
A network only understands numbers, but language and codec audio arrive as symbols. An embedding gives every symbol a learned list of numbers, so that similar symbols end up close together. This post builds one by hand, trains one, and follows it into speech models.
Post 7 of 15
A fully connected layer gives every pixel and every audio sample its own private weights, which is hopelessly wasteful. A convolution slides one small filter along the input instead. This post computes every output by hand: in one dimension, in two, across channels, with padding, stride and dilation, and backwards through the gradient.
Post 8 of 15
Sentences, waveforms and speech frames arrive in order, and their length changes. A recurrent network reads them one step at a time and carries a small memory forward. This post runs one by hand, then follows the gradient backwards through time and watches it shrink to nothing or blow up.
Post 9 of 15
A plain recurrent network forgets its first input because the gradient is multiplied by a small number at every step. The LSTM adds a second memory lane that is updated by adding, with small dimmer switches called gates deciding what to keep, what to write and what to show. This post runs one by hand, then a GRU, and trains both kinds on the remember-the-first-input task.
Post 10 of 15
A recurrent network has to squeeze a whole sentence into one small state. Attention lets every position look back at every other position directly, choosing how much to take from each. This post builds it from a soft library lookup up to multi-head, masked, scaled dot-product attention, with every number worked by hand.
Post 11 of 15
Attention on its own is one clever layer. The Transformer stacks it with a small feed-forward network, wraps each in a residual connection and a normalisation, and adds a sense of position. This post builds every part from scratch and runs a whole block on three tokens by hand.
Post 12 of 15
An autoencoder squeezes data through a narrow code and rebuilds it. A variational autoencoder makes that code a little cloud instead of a point, so the space of codes has no holes and you can pick a random point and decode something real. Every step worked on tiny numbers.
Post 13 of 15
A generative adversarial network trains two networks against each other: a forger that turns random noise into samples and a detective that scores how real they look. Every loss worked by hand, a one-number GAN trained live, and the ways the game goes wrong.
Post 14 of 15
Blur data into noise on purpose, then train a network to undo one small blur at a time. Diffusion models and flow models both generate by walking from noise back to data. Every step worked by hand on a toy whose data is just two points, +1 and −1.
Post 15 of 15
A language model does one small thing: it gives a probability to every possible next token. Chain those guesses together and you can score a sentence, train on raw text, and write new text one piece at a time. Counting, tokenizing, training and sampling, each worked on tiny numbers, and how the same machine speaks.
Post 1 of 8
A microphone turns air into a long list of numbers. This post follows those numbers from a single sample to the mel spectrogram that speech models read, with every number small enough to check by hand.
Post 2 of 8
Neural audio codecs turn every slice of sound into a handful of whole numbers. This post builds the trick behind that, residual vector quantization, from a box of paint colours up to EnCodec, DAC and Mimi, with every example small enough to check by hand.
Post 7 of 8
Khmer letters stack, vowels sit on every side of a consonant, and words have no spaces. This post follows one syllable from the screen to code points, bytes and tokens, and shows what each step means for a text-to-speech model.
Post 8 of 8
How an English text-to-speech model learned to read Khmer: the data, the cleaning, the cut into prompt and target, the new tokenizer, what was trained and what stayed frozen, and why the best checkpoint was not the one with the lowest loss.
Post 5 of 8
A cascaded voice assistant passes the turn between three models and waits at every handoff. A full-duplex model listens and speaks at once, 80 ms at a time. Where the latency goes, what the transcript throws away, and how interruptions actually work.
Post 6 of 8
WER, CER, speaker similarity, MCD, F0 RMSE, Fréchet distances, TTSDS, MOS, CMOS, MUSHRA and learned MOS predictors: what each one measures, the math, and where it stops agreeing with people.
Post 3 of 8
Kyutai's Pocket TTS generates speech as continuous vectors, not codec tokens, and turns noise into each audio frame in a single network call. The full path from text to waveform, with the shapes and the math.
Post 4 of 8
The codec under Pocket TTS turns each second of speech into 12.5 frames of 32 numbers and back. Every layer, stride and tensor shape on the way down and up, and why the latent is continuous.
Post 1 of 1
Joint-embedding training has a shortcut that always works: produce the same vector for everything. The geometry of that failure, and the linear algebra needed to read it.
Post 1 of 7
I traced a failed GH200 model startup to checkpoint file pages occupying HBM and measured the KV capacity recovered after targeted cache advice.
Post 2 of 7
I reconciled token counts, prompt lengths, cache reuse, and unfinished requests before comparing GH200 serving runs.
Post 3 of 7
I mapped the two Grace Hopper locality domains, checked worker placement, and tested NUMA restrictions while leaving KV-buffer locality for direct measurement.
Post 4 of 7
I adapted a CPU-KV path, checked reload output and actual movement, and investigated allocator rollback and transfer drain under pressure.
Post 5 of 7
I compared weight-backend footprints and fixed a rank-incomplete autotune cache so a historical serving profile could start with validated artifacts.
Post 6 of 7
A long-context soak kept producing output while first-token latency made interactive use difficult.
Post 7 of 7
I investigated why workers stopped while their container stayed alive, then separated process ownership, model readiness, and workload admission.
No posts match this filter.