Zero loss, empty representation

Train a classifier and the labels keep it honest: there is an external answer it has to match. Joint-embedding self-supervised methods have no such anchor. They hide part of an input, predict the hidden part’s representation from the visible part, and score the prediction against the encoder’s own output. Both sides of the comparison are produced by the thing being trained.

That creates a shortcut. If the encoder maps every input to the same vector, the prediction is exact and the loss is zero. The constant encoder is not a local minimum the optimizer stumbles into; it is a global optimum of the objective, sitting in the open, available from the first step.

The stop-gradient, the moving-average target and the explicit distribution regularizer all exist to keep training away from it. What follows is what “degenerate” actually means, the linear algebra you need to read it off a run, and why the most convenient diagnostics are the ones most likely to tell you everything is fine.

The energy picture

The cleanest framing is the energy-based one. Think of a scalar energy assigned to each pair of inputs: low when they are compatible, high when they are not. Training should carve a landscape with valleys on the data manifold and high ground elsewhere.

A purely predictive loss can reach its minimum in two ways. It can learn structure, so that energy is low exactly where it should be. Or it can flatten the landscape, so energy is low everywhere. The second is degeneracy, and the loss cannot tell you which happened.

This also explains a failure that has nothing to do with the encoder. If the predictor takes a latent variable with too much capacity, it can map any context to any target. Energy becomes low off the manifold as well as on it, and nothing has collapsed in the encoder at all. The discrimination has collapsed. This is why formulations of joint-embedding architectures insist on limiting the information content of that latent.

The one object every diagnostic is computed on

Before the failure modes, the algebra. Suppose you push N inputs through an encoder f and stack the outputs as rows. For the sake of something concrete, say 10,000 driving clips encoded to 256 dimensions each:

• $x_i$ is input number $i$ (one driving clip), and $f$ is the encoder.
• $z_i = f(x_i)$ is that clip’s embedding, a list of $d$ numbers ($d = 256$ here).
• $Z$ stacks all $N$ embeddings as rows, so it has $N$ rows and $d$ columns: 10,000 × 256.

Read a row and you are asking what the encoder thinks about one clip. Read a column and you are asking what one feature responds to. Sort it and you get the clips that excite that coordinate most. Rank questions are easiest in the row view, correlation questions in the column view.

Two derived objects. The mean μ is the column-wise average, a single vector in the same space as the embeddings: the “average input” according to the encoder. It is rarely near zero, and that will matter later. The covariance is

• $\mu$ is the mean embedding, and $\mathbf{1}\mu^{\top}$ is that mean copied onto every row.
• $\bar{Z}$ (Z-bar) is $Z$ with the mean subtracted from every row, so each column now averages to zero. This is centring.
• $\bar{Z}^{\top}\bar{Z}$ multiplies every column by every other column and adds over the $N$ rows; dividing by $N$ makes it an average.

Tiny example with two features over three clips, centred columns $(1, 0, -1)$ and $(2, 0, -2)$: $C_{12} = (1 \cdot 2 + 0 + (-1)(-2))/3 = 4/3$. Positive, because the two features rise and fall together.

Note the shape change: Z is 10,000 × 256, but C is 256 × 256. The dataset dimension has been summed away. C is not about clips any more; it is a table of relationships between features. Entry C_jk takes column j and column k, centres both, multiplies elementwise and averages. It answers this question: across the dataset, when feature j runs above its average, does feature k do the same?

Eigenvalues are variances

C is symmetric and positive semi-definite, and that second property has a concrete reading worth internalising. For any unit vector v, project every embedding onto v and take the variance of those N numbers:

• $v$ is a direction (a list of $d$ numbers with length 1).
• $\bar{Z}v$ projects every embedding onto that direction, giving one number per clip, like casting each point’s shadow onto a line.
• $\operatorname{Var}$ is the variance of those shadows: how spread out they are along $v$.

So the quadratic form vᵀCv is “how much do my embeddings spread along direction v”. It is non-negative because variance cannot be negative. That is all positive semi-definite means here.

Now ask which direction has the most spread. Maximising vᵀCv over unit vectors gives the top eigenvector u₁, and the maximum value is the top eigenvalue λ₁. Repeat in the orthogonal complement for u₂, λ₂, and so on. The spectral theorem says this works cleanly for symmetric matrices:

• $U$ holds the eigenvectors $u_1, u_2, \dots$ as columns: the cloud’s main directions, longest first.
• $U^{\top}U = I$ says these directions are at right angles to each other and have length 1 ($I$ is the identity matrix).
• $\Lambda$ (capital lambda) is a diagonal matrix holding the eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots$: how spread out the cloud is along each direction.

Picture a rugby ball: $u_1$ points along its long axis with a large $\lambda_1$, and $u_2$, $u_3$ point across it with smaller values.

λᵢ is the variance along direction uᵢ. That one sentence is why eigenvalue spectra became the standard collapse diagnostic: it turns an abstract algebra question into “how fat is the cloud in each direction”.

Papers say “singular value” where you expected “eigenvalue” because the two are the same information in different clothes. Writing Z̄ = UΣVᵀ gives C = (1/N)·VΣ²Vᵀ, so λᵢ = σᵢ²/N. In practice take the SVD of the embedding matrix directly rather than forming C: squaring the matrix squares the condition number and destroys precision exactly in the small-eigenvalue tail you care most about.

Coordinates are arbitrary, and that has consequences

Nothing in this kind of training makes “dimension 37” mean anything. Which coordinate ends up carrying rain or traffic density is an accident of initialisation, and the information is smeared across coordinates.

A rotation is a change of measuring rulers. The cloud sits fixed; you choose a different frame to describe it in. Distances, angles and shape are unchanged. Only the numbers you write down change.

Concretely, u₁ is a 256-long vector of weights. Think of it as a recipe. Applying it, u₁ᵀzᵢ, gives one number per input: a synthetic feature built as a weighted blend of all 256 coordinates. The eigendecomposition picks the recipe whose outputs have the largest spread, then the next largest orthogonal to it.

Worth saying plainly: there is no guarantee eigenvectors align with human concepts. They are directions of maximum variance, and variance is not semantics. If your dataset has wildly varying camera exposure, u₁ will be exposure. This is a real failure mode when people over-read PCA plots of embeddings.

The split that matters for diagnostics is what survives a rotation:

• Rotation-invariant: rank, trace, the eigenvalue spectrum, all pairwise distances and angles, and therefore all retrieval results and rank scores. These describe the cloud.
• Not rotation-invariant: the variance of any individual coordinate, the off-diagonal entries of C. These describe the cloud as seen from one frame.

That is why a method like VICReg needs two separate terms. Its variance term pushes the diagonal of C up so every coordinate actually varies; its covariance term pushes off-diagonal entries down so coordinates stop duplicating each other. Both are frame-dependent by construction: together they push the encoder’s own output axes towards being the eigenframe with equal variance along each. Diagonalising without equalising still permits a spectrum cliff; equalising without diagonalising still permits redundancy.

Degeneracy is a family, not a state

With the algebra in place the failure modes separate cleanly.

Complete collapse. The constant encoder. Covariance is zero in every direction; rank is zero.

Dimensional collapse. Variation survives but occupies a few directions out of many. A nominally 256-dimensional embedding might effectively live in twelve. Jing et al. attribute this to implicit regularisation in gradient descent rather than to the loss having a bad optimum. Broad distinctions survive, fine ones do not, which is the worst case for search: the rare item you most wanted to find lands on top of a common one.

Informational collapse. Full rank, but heavily correlated coordinates create redundant dimensions. Distinct from dimensional collapse in that per-coordinate variance is preserved. This is the failure VICReg’s covariance term targets.

Latent-variable degeneracy. The predictor-side failure from earlier: energy low off the manifold because the latent carries too much capacity. The encoder need not collapse at all.

Cluster collapse. In prototype-based methods, every sample routed to one prototype.

One term worth separating: neural collapse (Papyan, Han and Donoho) is a different phenomenon. In the terminal phase of supervised training, class means converge to a simplex equiangular tight frame. Reviewers conflate it with the above often enough that it is worth a footnote in anything you write.

A methodological point that matters more than it sounds: measure at the right layer. In methods with a projector, the loss lives in projector space and the backbone can retain rank while the projector collapses, or the reverse. In predictive architectures you have a context encoder, a target encoder and a predictor, and degeneracy can be asymmetric across them.

Reading a spectrum

The standard figure plots log λᵢ against index, sorted descending. The log axis is not cosmetic: healthy spectra span many orders of magnitude and a linear axis flattens every interesting eigenvalue against zero.

Rank itself is discrete and brittle. The flattened cloud almost never has an exactly-zero eigenvalue; it has λ₂ ≈ 10⁻⁹. Technically full rank, effectively one-dimensional. Every “effective rank” measure exists to give a continuous, threshold-free answer instead:

• $\sigma_i$ (sigma) are the singular values of the centred embedding matrix, largest first. Each $\sigma_i^2$ is proportional to the eigenvalue $\lambda_i$.
• Stable rank divides the total spread by the spread along the biggest direction.
• $p_i$ turns the singular values into shares that add up to 1, and $-\sum_i p_i \log p_i$ is the entropy of those shares: low when one direction dominates, high when they are even.

Example with three directions: $\sigma = (1, 1, 1)$ gives stable rank 3 and RankMe 3. With $\sigma = (1, 0.01, 0.01)$, stable rank is $1.0002 / 1 \approx 1$, and RankMe is about 1.1.

RankMe is the exponentiated Shannon entropy of the normalised singular values: mass on one direction gives about 1, mass spread evenly over k gives about k. It is the continuous relaxation of counting non-zeros. A related diagnostic fits λᵢ ∝ i^−α and reports the exponent. For predictive architectures specifically, LiDAR was built because entropy-of-spectrum measures underperform there, using a discriminant-style rank in predictor space.

Here is the structural catch. Every one of these is a ratio. Scale cancels. If a representation shrinks toward numerical noise, that noise is isotropic, and isotropic noise has excellent rank. A ratio-valued diagnostic can rise while a representation dies, not because it malfunctioned but because it answered the question it was asked, which was about shape, at a moment when the informative question was about magnitude. Read any rank number beside total variance, never instead of it.

Rank is not the whole story: angular structure

Retrieval using cosine similarity discards norms entirely. Only angles survive to influence the ranking, and angle is a different axis from rank.

The canonical decomposition is Wang and Isola’s alignment and uniformity: alignment is the expected distance between positive pairs, uniformity measures how close the embedding distribution is to uniform on the sphere. Complete collapse is uniformity at its worst; the simplex frame of neural collapse is maximal angular separation. Most degenerate solutions sit somewhere on that axis.

Rank metrics are global and linear. Retrieval only cares about the ordering of angles in each query’s neighbourhood. This is a local, ordinal property. These come apart. The clearest symptom is hubness: in high-dimensional angular spaces a few points appear in a disproportionate share of nearest-neighbour lists, which you can check directly by counting how often each item appears in others’ top-k.

A caution about the third panel: it is a 2D cartoon of something that behaves differently at real dimensionalities. In high d, random unit vectors are near-orthogonal by default, so cosines concentrate near zero with variance about 1/d even under a random encoder. Angular spread is the null hypothesis, not the achievement. What distinguishes a good representation is positive-pair mass sitting well above that concentration.

Centering decides which object you measured

This is the piece that most often explains two diagnostics disagreeing about the same encoder.

Covariance-based measures subtract the mean first, so they describe spread and nothing else. Measures computed on the raw embedding matrix keep the mean, which contributes a rank-one term to the spectrum. Algebraically the two second-moment matrices differ by exactly that:

• The left side is the same average of products as $C$, but on the raw $Z$, without subtracting the mean.
• $\mu\mu^{\top}$ is the mean multiplied by itself, a $d \times d$ matrix pointing entirely along the mean’s direction.

In words: raw spread = spread around the centre + where the centre is.

So a representation can have a beautiful centered spectrum while the uncentered one is dominated by a single rank-one term. That is the cone: every embedding pointing roughly the same way from the origin, documented as anisotropy in language models by Ethayarajh among others. Since retrieval never centers, cosine ranking is then decided by a thin residual. Computing the spectrum both ways and reading the gap tells you how much of your apparent structure is just the mean.

Whitening closes the loop: transform z̃ = C^−1/2(z − μ) so the new covariance is the identity. All eigenvalues are equal, and the condition number is one. This is the operation behind Barlow Twins driving a cross-correlation matrix to identity, and behind post-hoc “all-but-the-top” fixes. It is also why near-zero eigenvalues are dangerous beyond rank counting: inverting one amplifies noise, which is why these methods regularise the inverse.

Two families of fix

Methods divide by where they put the solution, and it is worth knowing which camp a method is in before claiming anything about it.

Dynamics. BYOL, SimSiam and predictive architectures leave the degenerate optimum in the landscape and rely on the optimisation trajectory not reaching it. Stop-gradient, the moving-average target and the predictor create a system whose trajectories avoid collapse; Tian et al. analyse this in terms of predictor eigenspace alignment. Collapse-avoidance here is an optimization property.

Landscape. VICReg and Barlow Twins instead remove the degenerate optimum by construction, adding terms that are large exactly when the representation degenerates.

Which camp a method belongs to determines whether a claim about it is a stability claim or a loss-design claim, and reviewers will ask.

What to measure

The useful summary is not a ranking of metrics but a note of which axis each one can see.

A practical logging set, with the reason each earns its place:

• Total variance, beside every rank number rather than instead of it. It is the quantity that falls when scale is lost.
• Mean pairwise cosine, the cheapest direct read on the angular structure retrieval depends on.
• The spectrum both ways, centered and raw, treating the gap between them as information about the mean.
• A retrieval metric, if retrieval is what the representation is for. A linear probe answers whether a decision boundary exists somewhere in the representation; retrieval answers whether near neighbours share an attribute. These are different questions and one does not stand in for the other.
• Per-layer, not just at the loss. Backbone, projector and predictor can disagree.

The underlying rule is simple enough to state in one line: a metric can only detect a failure along an axis it is sensitive to. Scale-invariant measures cannot see a loss of scale; rotation-invariant measures cannot see a bad choice of frame; centered measures cannot see a drifting mean. None of them are wrong. They are answering a question that is not the one collapse happens to be posing.