The 27 GB Opportunity: Aggressive Throughput Tuning on B200 GPUs

In the middle of a 41-hour generation run producing 913,786 completions for DFlash speculative decoding training data, the assistant made a bold decision: stop everything and reconfigure. The subject message at <msg id=7640> captures the moment of diagnosis and action — a turning point where the assistant identified 27 GB of wasted GPU memory per card and pivoted from a "good enough" configuration to an aggressively tuned one that would ultimately deliver a 78% throughput improvement.

The Context: A High-Stakes Generation Pipeline

By the time of this message, the team had already navigated a complex journey. The original 914K-sample tokenized dataset had been discovered to contain essentially empty responses — 87% of samples had only 6 tokens of meaningful content. This forced a complete pivot: instead of using the existing dataset, the team needed to regenerate all completions using Qwen3.6-27B with thinking mode enabled. After benchmarking SGLang on a 4× RTX PRO 6000 Blackwell node and calculating a prohibitive 16.5-day timeline, the user provisioned a 7× B200 NVL node (183 GB each, NVLink mesh) that could deliver dramatically higher throughput.

The generation had been running for about 11 minutes when the user asked at <msg id=7638>: "Do one round of aggresive tuning to try and get more per gpu throughput." This request came after the assistant had already reported solid performance — ~2,000 tok/s per GPU, ~13,500 tok/s aggregate, 41-hour ETA — and the user wanted to see if there was headroom to push harder.

The Diagnosis: Reading the Memory Map

The subject message opens with a crisp memory breakdown that reveals the opportunity:

~27 GB free per GPU — wasted. Currently: - Model: 54 GB - Mamba cache (80 slots + extra_buffer): ~21 GB - KV cache: ~35 GB (but only 2-3% used!) - CUDA graphs + overhead: ~18 GB - Free: ~27 GB

This analysis is the result of combining two data points gathered in the preceding message <msg id=7639>: nvidia-smi output showing 155 GB used out of 183 GB per GPU (leaving ~27 GB free), and the SGLang logs showing KV cache allocation of 1.14 million tokens consuming 35 GB while only 2-3% was actually utilized. The assistant's reasoning in <msg id=7639> reveals the thought process behind this diagnosis:

"The KV cache is massively over-allocated for our workload (short prompts, 16 concurrent)."

The key insight is that the generation workload — short prompts being fed to the model for completion — doesn't need the massive KV cache that SGLang allocated by default. The model weights consume 54 GB, the Mamba cache (SSM state + intermediate buffers) takes about 21 GB for 80 slots, and CUDA graph overhead accounts for roughly 18 GB. That leaves 27 GB completely idle, representing memory that could be repurposed for higher concurrency.

The Assumption Under the Diagnosis

The assistant makes a critical assumption here: that the bottleneck is the Mamba cache slot count, not raw memory. With 80 Mamba slots and the extra_buffer scheduler strategy, SGLang had auto-capped max_running_requests at 16 — meaning only 16 requests could be actively processed at once, even though the queue depth was 48 per server. The assistant's reasoning in <msg id=7639> explores this:

"The constraint might not actually be memory but rather the Mamba cache slots themselves, since each running request consumes extra slots through the overlap scheduler's buffering mechanism."

This assumption is grounded in the architecture of SGLang's Mamba scheduler. The extra_buffer strategy allocates additional Mamba cache slots beyond the running requests to enable overlap between prefill and decode phases. With 80 slots and 16 running requests, the effective utilization was about 59 slots (as seen in the Mamba usage of 74%), leaving headroom. By increasing max-mamba-cache-size to 200 and raising max-running-requests to 64, the assistant bets that SGLang will auto-calculate a much higher concurrency limit.

The Decision: Stop the Run

The most striking aspect of this message is the decision to kill the running generation and servers mid-flight:

[bash] ssh root@213.173.111.134 -p 36472 ' # Kill generation first, then servers pkill -f generate_completions 2>/dev/null sleep 2 pkill -9 -f sglang 2>/dev/null sleep 3 echo "Killed. Progress saved at ..."

This is a high-risk operation. The generation had already completed 3,981 samples (0.4% of the total) and uploaded 7 batches to S3. Killing the servers mid-flight could have corrupted the progress tracking or left the GPUs in an inconsistent state. The assistant mitigates this by using a two-stage kill (first the generation script, then the SGLang servers with a 2-second delay) and by checking the progress file to confirm it was saved.

The (no output) response from the bash command is concerning — it suggests the SSH session may have timed out or the commands didn't produce visible output. The assistant follows up in <msg id=7641> and <msg id=7642> with additional kill commands and process checks, eventually confirming all processes are dead.

The Reasoning Process Visible in the Message

While the subject message itself is terse and action-oriented, the preceding message <msg id=7639> contains the full reasoning trace. The assistant systematically evaluates multiple tuning levers:

  1. Increase max_running_requests: Currently auto-set to 16, limited by Mamba state with extra_buffer strategy
  2. Increase max-mamba-cache-size: Currently 80 slots consuming ~21 GB; pushing to 120-160 would consume ~42 GB but still leave 87 GB for KV cache
  3. Increase mamba-full-memory-ratio: From 0.4 to 0.6 to allocate more memory to Mamba state
  4. Disable MTP: Would remove extra_buffer constraint but lose 3x decode speedup — judged not worth it
  5. FP8 KV cache compression: Would halve memory footprint but adds complexity
  6. Reduce context length: From 8192 to 4096 would free per-request memory
  7. Piecewise CUDA graphs: Could improve throughput with varied batch sizes
  8. Increase num_continuous_decode_steps: Reduces scheduling overhead The assistant ultimately settles on the most impactful combination: double the Mamba cache to 200 slots, raise the memory fraction to 0.93, set max-running-requests to 64, and keep MTP enabled. This represents a calculated bet that the Mamba slot count was the primary bottleneck, not raw memory.

The Outcome: Validated by Subsequent Messages

The results of this tuning are visible in <msg id=7649> where the assistant reports:

| Metric | Before | After | Change | |---|---|---|---| | Running req/GPU | 16 | 40 | 2.5x | | Per-GPU tok/s | ~2,000 | ~3,500 | +75% | | Total tok/s | ~14,000 | ~24,900 | +78% | | ETA | 41h | 29.6h | -28% |

The actual max_running_requests settled at 40 (not the requested 64), suggesting SGLang's auto-calculation still imposed a cap based on available memory. But the 2.5x increase in concurrent requests translated directly into a 75% throughput improvement — validating the assistant's diagnosis that the Mamba cache was the bottleneck.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

Output Knowledge Created

This message produces several critical pieces of knowledge:

Conclusion

The subject message at <msg id=7640> is a masterclass in production ML inference tuning. It demonstrates the importance of reading the actual memory map rather than accepting default configurations, the courage to stop a running job when optimization opportunity is clear, and the value of systematic reasoning about bottlenecks. The 27 GB of wasted memory — representing 15% of each GPU's capacity — was transformed into a 78% throughput improvement that saved over 11 hours of generation time. This single decision, executed in under a minute of wall time, had a direct impact on the project timeline and cost.