Home

Embedding Diagnostics

Back to Projects

Self-Supervised Learning · August 2026

A controlled study of which label-free diagnostics can tell that a self-supervised embedding has degenerated, first on CIFAR-10 and then on BDD100K driving scenarios, where the downstream task is scenario retrieval.

Role: ML Research Engineer Impact: Geometric collapse shown to be separable from information loss Stack: PyTorch · masked JEPA · SIGReg · BDD100K

TL;DR

  • Problem: Scenario mining embeds huge numbers of unlabelled driving clips, and there are no labels to check whether the embedding has gone wrong. Which label-free diagnostics can you trust?
  • Approach: Manufacture known degeneration in masked JEPA training, using stop-gradient, EMA, SIGReg and a projector as controlled interventions: 7 conditions × 5 paired seeds, on CIFAR-10 and then on BDD100K, with endpoints and the claim rule fixed in advance.
  • Outcome: An encoder that every geometric measure calls destroyed stays nearly as linearly decodable as one that genuinely trained, while retrieval, the task scenario mining depends on, clearly degrades.

What I Built

  • Controlled degradation bench: seven training conditions, including a control with no collapse prevention that is built to fail, run over five paired seeds on CIFAR-10 (Phase A) and BDD100K driving scenarios (Phase B).
  • Diagnostics under test: total variance, mean pairwise cosine, RankMe, participation ratio, linear probes and retrieval precision, each documented with its formula, a worked example and its blind spot.
  • Validated components: the SIGReg loss matches the reference implementation bit for bit, and that validation found a real bug in the first version. The rest of the system (masked latent prediction, a project-specific projector) is not a LeJEPA reproduction.
  • Minimal sufficient panel: an exhaustive search over subsets found that no single label-free diagnostic covers every collapse mode, but total variance plus RankMe does, with zero misclassifications across all 14 condition-dataset pairs.
  • Reproducible runs: environment checks before training, crash recovery that reruns only unfinished jobs, one locked CUDA 12.8 environment across x86 and Grace Hopper machines, and CI that runs lint, 300+ tests and the panel validation.

Results

Decodability survives

0.914 vs 0.927

Time-of-day probe accuracy, contracted control vs healthy encoder (majority floor 0.483), although the control's total variance is 0.0001 against 107.71.

Retrieval degrades

0.566 vs 0.675

Retrieval P@10 for the same two encoders (chance 0.429); unlike the probe gap, this one clears the across-seed noise.

Effective rank inverts

25.45 vs 10.99

Participation ratio rates the contracted encoder as higher-dimensional than the healthy one; RankMe gets it right (1.06 vs 78.36).

What I withdrew: an earlier headline claimed the standard probe was blind to collapse. It was wrong: the unstandardized probe's optimiser stopped before fitting on very small features. Re-measuring Phase B from saved encoders moved the control's score from 0.4832 to 0.9137 and refuted the claim, and the write-up says so.

Scope: all figures above are Phase B (BDD100K), 7 conditions × 5 paired seeds. Phase A probe numbers are not quoted because they have not been re-measured. SIGReg is paired here with masked latent prediction, so none of this is evidence about LeJEPA's own claims.