Why could a quantized checkpoint look small enough for the GPU but fail during model startup? I investigated how the serving backend represented the weights in memory. A second question followed: could a saved tuning cache support every rank on restart? These were startup checks, not timed request benchmarks. Request sizes and client concurrency were not the variables under test.
The historical MiMo v2.6 B0 profile used a pinned vLLM development build, reported as 0.29.1rc1.dev449+geb8798058, on dual GH200. Tensor parallelism (TP) split model work across two GPU ranks; expert parallelism (EP) placed experts across two ranks. I compared backend preparation outcomes within that historical runtime, then checked rank-specific tuning records. The archive does not provide matched total startup durations for the backend comparison.
One rank's cache cannot certify the other rank's tactics.
The diagram shows two things the runtime needs: enough space to prepare its weights, and tuning records for each rank. It is a conceptual map, not a timing plot. KV means the key-value cache, which stores attention state for requests.
The failing phase determines the memory question
Checkpoint size is not peak startup memory. I distinguish three parts of startup:
- Checkpoint loading: read model state and create the initial runtime allocations.
- Backend preparation: construct the backend-specific representation, packing buffers, and conversion workspaces.
- Later runtime preparation: profile capacity, capture graphs, and reserve KV as required by the particular engine path.
The relative order of graph and KV work depends on that path; this list does not establish their order in every runtime. A lower KV-memory target cannot rescue a failure that occurs before the engine has begun allocating KV.
The backend changed between these startup attempts. The recorded outcomes were:
| Backend path | Reported outcome | What it established |
|---|---|---|
| Marlin preparation | About 145.9 GiB per rank at the failing preparation stage | Representation and temporary work exceeded the available budget before KV creation |
| Triton path | Activation compatibility rejection | A memory-only fix would not make this path valid |
| FlashInfer CUTLASS | Loaded at about 83 GiB per rank | This historical representation fit and enabled the validated B0 serving profile |
The selected FlashInfer CUTLASS path fit at about 83 GiB per rank, while Marlin preparation reached about 145.9 GiB per rank and failed before KV creation. The Triton attempt failed for compatibility rather than an established memory shortage. That distinction mattered: reducing the future request-cache budget could not repair a failure in weight preparation or activation support. These historical observations do not rank current backends or isolate a single flag’s effect.
The practical diagnostic is to record the failure phase, such as load, packing, profiling, graph capture, or KV reservation, before adjusting a budget for a later phase.
A separate GLM quantized-and-speculative track reinforced the boundary between weights and request state. Its planning record reported about 338,624 GPU KV tokens under one tensor-parallel configuration: fitting quantized weights did not remove the KV budget for long requests. The exact model revision, engine build, and output/request counts were not recovered well enough to make its recalled throughput figure a benchmark result, so I use this track only for the capacity lesson.
A rank-specific cache must be complete
The next problem involved expert-parallel autotuning, which tries execution choices and saves the selected tactics for reuse. A persisted cache with 42 genuine records covered one rank, but a second rank needed its own keys. Rank zero could find a tuned entry while rank one missed or fell back. The local workaround generated and persisted 84 genuine records, 42 for each rank, before a real startup using only the cache. The report recorded cache hits on both ranks in that historical profile.
Copying a rank-zero record and changing its key would not show that its tactic had been tuned or validated for rank one. The cache must agree with the actual backend, shapes, version, and rank identity. FlashInfer’s autotuning documentation describes public persistence concepts; it does not certify that this older local workaround is required or suitable for today’s stack.
The change was from one rank’s records to genuine records for both ranks. The runtime still needed the correct backend, shapes, version, and rank keys. The figure shows those independent sets beside the startup phases; it does not show measured phase durations.
Faster startup has fixed constraints
Graph capture, compilation, and tuning can make initialization expensive because they prepare a serving configuration that may be faster later. An eager launch or fewer captures might reduce startup time while changing throughput, coverage, or KV capacity. That would answer a different objective. vLLM’s CUDA Graphs design documentation describes why graph modes cover different execution shapes.
For this work, the constraint was to keep runtime performance, quality, features, graph coverage, and KV capacity while removing avoidable repeated startup work. One separate report recorded about 69 seconds of graph capture in a historical profile. It was a phase observation, not an A/B-proven startup improvement. Another raw startup log warned that referenced compiled artifacts were missing; the archive does not establish their performance impact or a verified repair.
A populated cache directory and an HTTP-ready process are insufficient. After restart, I would validate:
- Rank-specific cache hits: every expected rank uses its genuine tuned records.
- Effective graph path: the intended graph mode and coverage remain active.
- KV capacity: the capacity budget remains consistent with the validated profile.
- Serving behavior: the same workload still completes with the required performance, quality, and features.
The local integration produced a fitting backend choice and a rank-complete cache for this historical profile. It did not establish a measured startup speedup or complete reuse of compilation artifacts. A current runtime would need its own restart validation.