Diffusion and flow models: generating from noise

In the VAE post we generated by picking a random code and decoding it in one go. In the GAN post a generator also jumped from noise to a sample in one go, trained against a critic. This post is about a third family that takes its time. It starts from pure noise and makes it a little less noisy, again and again, until a sample appears. These are diffusion models and their close cousins, flow models.

They are behind most image generators of the last few years, and they are behind the speech models on this site: F5-TTS writes a spectrogram by flow matching, and the head of Pocket TTS turns noise into each audio frame. This post is the gentle on-ramp. The deeper version, with every equation of the original paper, is my Flow Matching review, and I link into it wherever this post stops.

The whole model at a glance

Here is a whole diffusion model at work, on the toy this post uses throughout: data that are only ever −1 or +1. Training and sampling use the same network in two different loops. Training: take a clean point, add a random amount of noise, ask the network to guess the noise, and score the guess. Sampling: start from pure noise and call the network a few times, each time stepping to a slightly cleaner point. The noise schedule, the network’s job and the sampling step each get a section below; step through both paths once now.

The generation problem

We have a pile of examples: photos, or short clips of speech. We want a machine that makes new ones that look like they came from the same pile. In the language of the loss post, the examples come from some distribution, a rule that says how likely each possible example is. We never get to see that rule. We only see samples from it.

Everyday picture: you have a jar of a thousand pebbles from one beach. Nobody gives you a map of the beach. Can you build a machine that hands you a new pebble the beach could plausibly have produced?

Throughout this post the “beach” is tiny so we can do everything by hand:

• The toy data: every example is a single number, and it is either +1 or −1, each half the time.

A good generator must output values near +1 or −1, about half each, and almost never 0. That is surprisingly hard to do with one simple formula, which is why it is a good test.

Noise into data, a little at a time

Turning random noise into a +1 or −1 in one jump is a hard problem. Undoing a tiny bit of blur is an easy one. So the trick is:

1. Take real data and blur it with noise, a little at a time, until nothing is left but noise. This is easy: it needs no learning at all.
2. Train a network to undo one small blur.
3. To generate, start from pure noise and apply the undo step many times.

Everyday picture: a drop of ink spreading in a glass of water. Watching it spread is easy. Running the film backwards, ink gathering into a drop, looks like magic, but each frame of the backward film only differs a little from the one before. If you can learn to predict one frame back, you can play the whole film backwards.

The idea of learning to reverse a slow noising process comes from Sohl-Dickstein, Weiss, Maheswaranathan and Ganguli (2015), who borrowed it from physics (the “diffusion” of the name is the ink spreading). Ho, Jain and Abbeel (2020) made it work very well on images with a much simpler training loss, under the name DDPM (denoising diffusion probabilistic models).

The forward process: blurring on purpose

The formula

In words: at noise level $t$, keep a shrunken copy of the clean example and add a scaled dose of fresh random noise.

• $x_0$ is the clean example (here +1 or −1). In diffusion papers the subscript 0 means “clean data”.
• $t$ is the noise level or time, from $0$ (clean) to $1$ (all noise). DDPM counts it in whole steps from 1 to 1,000; I use a number between 0 and 1 to keep things simple.
• $\varepsilon$ (epsilon) is a fresh random number drawn from the standard normal (bell curve) with average 0 and spread 1.
• $\bar\alpha_t$ (alpha-bar) is the share of the signal’s power that survives at time $t$. It starts at 1 and falls to 0. How it falls is the noise schedule.
• $\sqrt{\bar\alpha_t}$ is how much of the clean example we keep; $\sqrt{1-\bar\alpha_t}$ is how much noise we add. Their squares add to 1, so if the data has spread 1 the result keeps spread 1 (this is why it is called variance preserving).

To keep the arithmetic friendly I use the schedule

so the “keep” and “noise” weights are the two sides of a quarter-turn. It is a simplified version of the cosine schedule of Nichol and Dhariwal (2021); DDPM itself used a schedule that is linear in a different quantity.

Everyday picture: a radio slowly turned from one station to static. At every moment you hear a mix; the dial sets how much is music and how much is hiss.

Worked by hand

Take the clean point $x_0 = 1$ and one fixed noise draw $\varepsilon = -1.5$.

By $t = 0.5$ the point has crossed zero: you can no longer tell for sure which side it started on. By $t = 1$ it is just the noise. If $\varepsilon$ had been $+1.5$ instead, the point would have moved away from zero and ended at $+1.5$.

Do this for many points and many noise draws and the two sharp spikes at ±1 spread into two bumps, then merge into one bell curve. Figure 2 shows that, with the formula filled in for one point.

A useful fact: we can jump straight to any $t$ in one line. We never have to simulate the small steps one by one during training. That is what makes training cheap.

The network’s job: guess the noise

The loss

In words: show the network a blurred example and the noise level, and ask “what noise was added?”; score it by the squared error.

• $\varepsilon$ is the noise we actually added (we know it, because we added it).
• $\varepsilon_\theta(x_t, t)$ is the network’s guess of that noise, given only the blurred point and the time. $\theta$ (theta) is all its weights, as in the first post.
• $\|\cdot\|^2$ is the squared distance, the squared error of the loss post, summed over every number in the example.

One training step: pick a real example, pick a random $t$, pick a random $\varepsilon$, build $x_t$ with the formula, ask the network, compute the loss, take a gradient step. Ho and colleagues found this plain loss (they call it the “simple” loss) worked better in practice than the weighted version their theory first suggested.

Worked: from the table, at $t = 0.5$ the network sees $x_t = -0.354$. Suppose it guesses $\varepsilon_\theta = -1.0$. The loss is $(-1.5 - (-1.0))^2 = (-0.5)^2 =$ 0.25. Had it guessed $-1.5$ exactly, the loss would be 0.

Three things it could predict

Since $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon$ ties the three together, knowing $x_t$ and any one of them gives the other:

• The noise $\varepsilon$ (DDPM’s choice).
• The clean example $x_0$: then $\hat\varepsilon = (x_t - \sqrt{\bar\alpha_t}\,\hat x_0)/\sqrt{1-\bar\alpha_t}$.
• A velocity, a direction to move in, which is what flow models predict and which we meet below.

They are three ways of writing down the same knowledge; they differ in which errors the loss punishes most.

Why guessing the noise is learnable

Here is the worry: the network cannot possibly know the exact $\varepsilon$ we drew. At $t = 0.5$, the point $-0.354$ could have come from $x_0 = 1$ with a big negative noise, or from $x_0 = -1$ with a small positive noise. So what does the loss teach it?

It teaches it the average answer. The prediction with the smallest average squared error is the average of all the noises that could have produced this $x_t$ (that is a property of squared error: the best single guess is the mean). And for our toy, that average has an exact formula. The best guess of the clean point is

• $\tanh$ is the S-shaped squashing function from the first post, running from −1 to +1.
• The fraction is how strongly the evidence points to the +1 side: large when $x_t$ is far from 0 and the noise is small.
• The result is “chance it came from +1, minus chance it came from −1”, which is the average clean value. (This comes from Bayes’ rule for two equal bell curves centred at $\pm\sqrt{\bar\alpha_t}$; I checked it numerically in the script.)

Then the best noise guess follows from the line above: $\hat\varepsilon = (x_t - \sqrt{\bar\alpha_t}\,\hat x_0)/\sqrt{1-\bar\alpha_t}$.

Worked for our point, $x_t = -0.354$ at $t = 0.5$:

1. The fraction: $\sqrt{\bar\alpha_t}/(1-\bar\alpha_t) = 0.707 / 0.5 = 1.414$, and $1.414 \times (-0.354) = -0.500$.
2. $\hat x_0 = \tanh(-0.5) =$ −0.462. The network’s honest answer is “probably −1, but I am not sure”.
3. $\hat\varepsilon = (-0.354 - 0.707 \times (-0.462))/0.707 = (-0.354 + 0.327)/0.707 =$ −0.038.

So the best possible network guesses a noise of −0.038, not the −1.5 we actually used. Its loss on this example is about $(-1.5 + 0.038)^2 = 2.14$, and no network can do better on average, because the information is simply not there. Training does not need perfect answers; it needs the right average.

Why it is learnable: at each noise level the network only has to solve a small, local problem: “given this blurry thing, which way is the clean version, on average?”. Near $t = 0$ that is a nudge; near $t = 1$ it is a rough guess at the overall shape. No single question asks for a whole sample at once. If you move $x_t$ to $2$ (same $t$) the answer becomes $\hat x_0 = 0.993$: far from zero the network is nearly sure.

Sampling: stepping backwards

The DDPM step in words

To generate, start at $t = 1$ with pure noise and walk down to $t = 0$ in $N$ steps. At each step, from time $t$ to a slightly earlier time $s$:

1. Ask the network for its noise guess, and turn it into a guess of the clean point $\hat x_0$.
2. Work out the average point at time $s$ that lies “between” where you are and where the clean guess is. Statistically, it is the average of $x_s$ given $x_t$ and $\hat x_0$.
3. Add a little fresh noise, so that you land on a random point near that average rather than exactly on it.

Step 3 is what makes DDPM sampling stochastic (random): run it twice from the same start and you get different samples.

The average and the size of the fresh noise have a fixed formula from the schedule. Writing $a_t = \sqrt{\bar\alpha_t}$ and $b_t = \sqrt{1-\bar\alpha_t}$, and $r = a_t / a_s$ for how much the signal shrinks between $s$ and $t$:

You do not need to memorise this; it is a weighted average of “where I am” and “where the clean guess is”, with weights set by the schedule. Ho and colleagues write the same step in terms of the predicted noise.

One step by hand

From $x_t = -0.354$ at $t = 0.5$ to $s = 0.25$, with the fresh noise draw $z = 0.3$:

1. $\hat x_0 = -0.462$ (from the denoiser).
2. $a_t = 0.707$, $b_t^2 = 0.5$; $a_s = 0.924$, $b_s^2 = 0.146$; $r = 0.707/0.924 = 0.765$.
3. $b_t^2 - r^2 b_s^2 = 0.5 - 0.586 \times 0.146 = 0.414$.
4. Weight on $x_t$: $0.765 \times 0.146 / 0.5 = 0.224$. Weight on $\hat x_0$: $0.924 \times 0.414 / 0.5 = 0.765$.
5. Mean $= 0.224 \times (-0.354) + 0.765 \times (-0.462) = -0.079 - 0.354 =$ −0.433.
6. Spread $= \sqrt{0.414 \times 0.146 / 0.5} = \sqrt{0.121} = 0.348$.
7. New point $= -0.4329 + 0.3483 \times 0.3 = -0.32845$, about −0.328.

The point moved a little toward −1, then the fresh noise pushed it back a bit. Many such steps later it settles on −1 or +1.

Deterministic sampling: DDIM

Song, Meng and Ermon (2021) showed you can drop the fresh noise. Their DDIM (denoising diffusion implicit models) step simply rebuilds $x_s$ from the two guesses using the forward formula at the earlier time:

In words: “pretend my clean guess and noise guess are right, and blur the clean guess to level $s$ with that same noise”.

Same point: $0.924 \times (-0.462) + 0.383 \times (-0.038) = -0.427 - 0.015 =$ −0.441. No randomness: the same start always gives the same sample. DDIM also works well with far fewer steps than DDPM, and it uses the same trained network.

Song and colleagues (2021, score SDE paper) gave the bigger picture: the noising process can be written in continuous time, and there is an ordinary differential equation (ODE), the probability-flow ODE, whose paths carry the noise distribution to the data distribution with no randomness at all. DDIM is a way of stepping along it. That ODE is our bridge to flow models.

Many particles, many steps

Run the walk for 200 noise draws at once and watch the cloud split into two. Figure 5 compares 5, 20 and 100 steps. On this toy the last step always lands on the denoiser’s clean guess, so with the exact denoiser even 5 steps work fairly well: the 200 samples end on average 0.031 (DDPM) and 0.034 (DDIM) away from the nearest of ±1, and with 20 or 100 steps that falls to below 0.001. Roughly half end on each side. Real networks are not exact, and real data is not two points, which is why real samplers needed hundreds of DDPM steps or tens of DDIM steps.

The flow view: a velocity field

Watch the time direction

Flow papers count time the other way round, and this trips everyone up once. From here on, $t = 0$ is noise and $t = 1$ is data, $x_0$ is the noise and $x_1$ is the data point. This matches the Flow Matching review, F5-TTS and Pocket TTS (which writes τ).

Following arrows

A flow model learns a velocity field $v_t(x)$: for every position $x$ and time $t$, which way to move and how fast. To generate, drop a noise point and follow the arrows from $t = 0$ to $t = 1$. Picture a leaf on a river: the current at each spot is the velocity, and the leaf’s path is the flow.

Computers follow arrows in steps. Euler’s method says: look at the arrow here, move along it for one step, repeat. With $N$ steps of size $h = 1/N$:

• $x_k$ is where you are after $k$ steps; $t_k$ is the time then.
• $v_{t_k}(x_k)$ is the arrow there; each look costs one network call.

The review works Euler on three roads in its Euler figure; here we do it on our toy.

Straight paths: flow matching and rectified flow

The path for one pair

Pick one noise point $x_0$ and one data point $x_1$ and join them with a straight line:

• At $t = 0$ you are at the noise, at $t = 1$ at the data, at $t = 0.5$ exactly halfway.
• The velocity does not depend on $t$: constant speed in a constant direction, “destination minus start”.

Training is regression again: pick a pair and a random $t$, build $x_t$, and score the network with $\| (x_1 - x_0) - v_\theta(x_t, t) \|^2$. This is the conditional flow matching loss of Lipman, Chen, Ben-Hamu, Nickel and Le (2023); Liu, Gong and Liu (2023) arrived at the same straight-line recipe under the name rectified flow, and added “reflow”, retraining on the model’s own noise and sample pairs to make paths even straighter.

Worked: noise $x_0 = 0.5$, data $x_1 = 1$, $t = 0.5$. Then $x_t = 0.5 \times 0.5 + 0.5 \times 1 =$ 0.75 and the target velocity is $1 - 0.5 =$ 0.5. If the network says 0.3, the loss is $(0.5 - 0.3)^2 = 0.04$.

What the network actually learns

The same worry as before: many lines, from many noise points to both +1 and −1, pass through $x = 0.75$ at $t = 0.5$. The network learns their average velocity. On our toy that average again has a closed form:

In words: “guess the destination, then head there at the speed that arrives on time”. Why the average is the right target, and why matching single pairs still trains it, is the heart of the review: see Build the big path out of tiny ones and Conditional flow matching, with the MarginalAverage and CfmGradient figures there.

Euler steps by hand

Start from the noise point $x_0 = 0.5$ with $N = 2$ steps, $h = 0.5$:

1. At $t = 0$: $\hat x_1 = \tanh(0) = 0$ (no idea yet), $v = (0 - 0.5)/1 = -0.5$. Move: $0.5 + 0.5 \times (-0.5) =$ 0.25.
2. At $t = 0.5$: the fraction is $0.5 \times 0.25 / 0.25 = 0.5$, so $\hat x_1 = \tanh(0.5) = 0.462$; $v = (0.462 - 0.25)/0.5 = 0.424$. Move: $0.25 + 0.5 \times 0.424 =$ 0.462.

Two steps end at 0.462, far from +1. With $N = 4$ the stops are 0.5, 0.375, 0.305, 0.425 and then 1.000. Notice the first steps move toward zero: early on, the average of “go to +1” and “go to −1” points at the middle. Each pair’s path is straight, but the averaged field bends. The review calls this “straight per example, curved in aggregate” (Optimal transport).

Why straighter paths need fewer steps

Euler assumes the arrow stays the same for a whole step. On a straight road that is exactly right; on a bend a big step overshoots. So the less your paths curve, the fewer steps you need.

To see it on the toy, compare two paths from noise to the same ±1 data:

• Straight (flow matching): $x_t = (1-t)x_0 + t x_1$.
• Curved (diffusion-style): $x_t = \cos(\tfrac{\pi t}{2})\,x_0 + \sin(\tfrac{\pi t}{2})\,x_1$, which is our cosine diffusion schedule run in the flow direction, followed along its probability-flow ODE.

For the same 200 seeded start points I measured the mean distance between Euler’s end and the exact end:

To get the mean miss under 0.01, the straight path needs 6 steps and the curved one 12. Same data, same exact field, half the network calls. On real images the review reports the same direction of effect at much larger scale (sampling with few steps), and its DiffusionVsOt and StepBudget figures show it in 2-D.

Arrows in two dimensions

Everything works the same in more numbers. Put four data points at the corners $(\pm1, \pm1)$. Then each coordinate behaves like our 1-D toy, and the field is a picture of arrows that start out pointing at the middle and later swing toward the nearest corner. A particle starting at $(0.5, -0.3)$ with $N = 4$ Euler steps visits $(0.375, -0.225)$, $(0.305, -0.183)$, $(0.425, -0.267)$ and ends at (1.000, −0.997), the corner $(1, -1)$.

Conditioning and classifier-free guidance

So far the model makes “any pebble from the beach”. To make this sentence in this voice, we give the network a condition $c$ (text, a speaker clip) as an extra input. Classifier-free guidance (CFG), from Ho and Salimans (2022), trains the same network sometimes with the condition and sometimes with it hidden. At sampling time it asks both and pushes further in the direction the condition points: guided velocity = told + strength × (told − untold). With told 2, untold 1 and strength 2, that is $2 + 2 \times 1 = 4$. The F5-TTS review works this through with its GuidanceArrows figure (classifier-free guidance), and the DSM review applies the same trick to token scores with GuidanceMixer (voices, turns and guidance). Each guided step costs two network calls.

Diffusion and flow in speech

• F5-TTS generates a whole mel spectrogram by straight-path flow matching with Euler steps (32 by default in its main table), conditioned on padded text and a reference clip. It also chooses which times to step at, spending more steps early: see Sway Sampling.
• Pocket TTS uses a small flow head to turn noise into each 32-number audio latent. Its post derives flow matching from scratch (Part 4) and then shows how it learns the whole trip as one jump, so it needs a single network call per frame (Part 6).

The 32 numbers Pocket TTS generates are the code of a VAE, which ties this post back to the previous one but one: a VAE makes a smooth space, and a flow model generates points in it.

What to remember

• Generating from examples is hard in one jump and easy in many small denoising steps.
• Forward process: $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\varepsilon$. No learning; any $t$ in one line.
• The network predicts the noise (or the clean data, or a velocity) with a plain squared error. The best it can do is the average answer, and that is exactly what sampling needs.
• DDPM steps back with fresh noise each time; DDIM drops the noise and follows a deterministic path, the probability-flow ODE.
• Flow models learn a velocity field and follow it with Euler steps. Flow matching and rectified flow train on straight lines, target velocity $x_1 - x_0$.
• Straighter paths need fewer steps: 6 against 12 on our toy for a miss under 0.01.
• In flow papers $t = 0$ is noise; in diffusion papers $t = 0$ is data.

Try it yourself

• Figure 2: set $t = 0.5$. The red point must read $0.707 \times 1 + 0.707 \times (-1.5) =$ −0.354. Set $t = 0.75$: −1.003.
• Figure 3: set $x_t$ to 2 at $t = 0.5$: $\hat x_0$ must read 0.993 and $\hat\varepsilon$ 1.835. At $x_t = 0$ both must read 0.
• Figure 4: with $z = 0.3$, DDPM must show −0.328 and DDIM −0.441. Set $z = 0$: DDPM shows its mean, −0.433.
• Figure 5: DDPM with 5 steps must report a mean distance of 0.031; DDIM with 5 steps 0.034.
• Figure 6: $N = 2$ from 0.5 must end at 0.462; $N = 4$ at 1.000.
• Figure 7: at $N = 6$ the straight path reads 0.008 and the curved 0.045; at $N = 12$ the curved reads 0.009.
• Figure 8: from $(0.5, -0.3)$ with $N = 4$ the end must read (1.000, −0.997).

Sources

• J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan and S. Ganguli (2015). “Deep Unsupervised Learning using Nonequilibrium Thermodynamics.” ICML 2015. arXiv:1503.03585.
• J. Ho, A. Jain and P. Abbeel (2020). “Denoising Diffusion Probabilistic Models.” NeurIPS 2020. arXiv:2006.11239.
• J. Song, C. Meng and S. Ermon (2021). “Denoising Diffusion Implicit Models.” ICLR 2021. arXiv:2010.02502.
• Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon and B. Poole (2021). “Score-Based Generative Modeling through Stochastic Differential Equations.” ICLR 2021. arXiv:2011.13456.
• A. Nichol and P. Dhariwal (2021). “Improved Denoising Diffusion Probabilistic Models.” ICML 2021. arXiv:2102.09672.
• Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel and M. Le (2023). “Flow Matching for Generative Modeling.” ICLR 2023. arXiv:2210.02747.
• X. Liu, C. Gong and Q. Liu (2023). “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.” ICLR 2023. arXiv:2209.03003.
• J. Ho and T. Salimans (2022). “Classifier-Free Diffusion Guidance.” arXiv:2207.12598.
• Y. Chen et al. (2024). “F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching.” arXiv:2410.06885.