I wanted to know whether a server could keep completing long requests while becoming too slow for an interactive conversation. A high output rate would answer only part of that question. I also needed to measure how long each caller waited before the first output appeared.
I tested a MiMo serving profile on dual GH200 hardware, using vLLM and its configured speculative decoder. My work was deployment, integration, and measurement; the model, engine, and decoder came from their respective projects. This was a historical stress test, not a measurement of a current production service.
What the soak actually asked the server to do
The fixed stress recipe was:
- Input: 131,072 tokens per request.
- Output target: 16,384 tokens per request.
- Client load: up to 62 outstanding requests.
- Submission schedule: eight hours.
The request recipe stayed fixed during the eight-hour submission schedule. The client kept supplying work rather than letting the server empty its queue. Outstanding requests could be running or waiting; 62 was the client limit, not a count of requests all executing at once.
The key-value (KV) cache holds attention state that the model reuses while generating a response. The matching recovery startup reported 2,947,463 logical KV cache tokens of engine capacity, counted once across the distributed profile, and 40.15/41.45 GiB of KV allocation on its two ranks. These startup allocations describe the reserved pool, not how much request state occupied it at every moment.
One full request budget is 147,456 tokens. Dividing the reported logical KV capacity by that maximum yields about 20 full-budget request equivalents. That is a planning ratio, not a measurement that exactly 20 requests must always run. State grows as requests generate output, and output can stop early. Scheduling and preemption, where a request is paused to make room for other work, also change which state is resident. The client-side 62 outstanding requests were not 62 simultaneous full-budget residents.
2,947,463 logical KV tokens ÷ 147,456 maximum tokens per request ≈ 20 full-budget equivalents
The archive showed persistent waiting and high KV pressure: maximum sampled occupancy of the reserved KV pool was about 99.97%. That is the fraction of the engine’s pool occupied by request state, not the fraction of all GPU high-bandwidth memory (HBM) in use. As requests finished, their KV space became reusable, but the client continued supplying requests. The archive continued to show waiting requests. This pattern is consistent with sustained capacity pressure. The records do not separate queue time from prefill, the work of processing an input before generation, so they do not establish how much of each wait came from either source.
A separate million-token probe also had long first-token waits with substantially lower KV occupancy. Its full setup is not presented here, so I use it only as a reminder that spare KV space alone does not guarantee a quick response.
Throughput and latency must appear together
For that fixed stress workload, the server completed 2,252 requests. Its aggregate output rate counts tokens produced across requests together, using the recorded 29,100.679-second denominator. It does not measure the generation speed experienced by one caller.
Time to first token (TTFT) is the wait from submitting a request until its first output appears. Among completed requests, TTFT was 550.876 seconds at p50, the median, and 567.503 seconds at p95, the 95th percentile. Median end-to-end time was 818.576 seconds, about 13.6 minutes. The typical completed caller waited about 9.2 minutes just to see the first token. The server remained operational, but that was too slow for an interactive conversation.
Historical soak · completed requests only
9 min 11 s
| Historical soak measure | Archived observation |
|---|---|
| Completed successful rows | 2,252 |
| Observed failed rows | 0 |
| In-flight outcomes unknown at last sample | 38; drain incomplete |
| Output rate | 1,267.90 output tokens/s |
| Recorded rate denominator | 29,100.679 s |
| TTFT among completed requests | p50 550.876 s · p95 567.503 s |
The submission timer and later finalization timestamps do not align cleanly in the surviving archive, so I retain the recorded denominator rather than calculate a replacement. Source: canonical pack, EXP-GH200-1118 / SRC-05.
A separate B0 benchmark asked what happened when more clients shared the MiMo v2.6 reference profile on dual GH200. The request sizes stayed at 128 input / 512 output tokens, while outstanding clients increased from 16 to 32. The C32 point used a sequence limit of 16. The surviving comparison does not give the elapsed duration of each point, and it is not a matched comparison with the long-context soak.
With 16 clients, the report recorded 1,018.93 output tokens/s and 0.177-second median TTFT. With 32, output rate was 1,021.06 tokens/s and median TTFT was 7.788 seconds. The graph separates the two metrics. C16 and C32 mean outstanding client counts; they are not request completion positions. Each panel uses its own scale.
Throughput barely changed; first-output wait rose sharply at C32.
Output rate barely changed, but the median first-token wait grew from less than a fifth of a second to nearly eight seconds. Request sizes stayed the same. The pattern suggests that adding clients at this point mainly added waiting rather than useful output capacity; these two points alone do not identify the exact scheduling mechanism.
This is why I report client-outstanding, running, and waiting counts separately. Dividing aggregate output tokens/s by a configured concurrency value does not measure an individual’s decode speed when some clients spend most of their time waiting.
A completed subset is not every submitted request
The soak archive contains zero observed failed rows and 2,252 completed successes. At the last monitor sample, 38 requests were still running or waiting, and the drain did not complete. Their eventual outcomes are unknown. I do not count them as successes or failures. The latency percentiles cover completed requests only and may omit the slowest requests.
A separate very-long-input probe also lost service after a small completed subset; its root cause was not established and cannot be generalized to every long request.
Three capacity limits need separate names:
- Model context limit: the supported token budget for one request.
- KV pool size: the available storage for resident request state across the serving profile.
- Accepted client connections: the requests the service accepts, including requests that may wait rather than run.
Usable service also needs a first-token latency target and an explicit policy for admitting heavy work.
The practical next step would be to measure queue time and prefill separately, then admit interactive and long background work against a defined latency objective. That is a proposed operating policy, not an optimization shown to be deployed by these results. The historical soak proves that the engine could continue completing a large number of requests under pressure; it also shows why completion and peak output rate were insufficient measures of user experience.