The 30-Second Tuning That Unlocked 75% More Throughput

In a single message spanning just a few lines of markdown, the assistant delivered the results of an aggressive tuning intervention that transformed a large-scale language model generation pipeline. Message [msg 7649] is a compact victory lap—a before-and-after comparison table showing that a 30-second server restart, guided by careful reasoning about GPU memory allocation, increased per-GPU throughput from ~2,000 to ~3,500 tokens per second and cut the estimated completion time for a 900K-sample generation run from 41 hours to under 30. But behind this brief status update lies a dense chain of technical reasoning, resource trade-offs, and real-time debugging that reveals how modern inference serving is as much about memory management as it is about model architecture.

The Context: A Generation Pipeline at Scale

To understand why this message was written, we need to step back. The broader session (Segment 44) involved a pivot: the team had discovered that a 914K-sample tokenized dataset had essentially empty responses—87% of samples contained only a trivial six-token output. This rendered the entire dataset useless for training a DFlash speculative decoding drafter. The response was drastic: regenerate all 900K completions using Qwen3.6-27B with thinking mode enabled, producing full reasoning traces. This required deploying a fast inference engine on a newly provisioned 7× B200 NVL node (each GPU with 183 GB of HBM3e memory connected via NVLink mesh).

The initial deployment used SGLang 0.5.11 with MTP (Multi-Token Prediction) speculative decoding, running 7 independent data-parallel (DP) instances. The initial configuration achieved a respectable ~2,000 tok/s per GPU and ~14,000 tok/s aggregate, with an estimated 41-hour completion time. The user then asked a pointed question at [msg 7638]: "Do one round of aggresive tuning to try and get more per gpu throughput." This single request triggered the chain of reasoning and action that culminated in message [msg 7649].

The Reasoning: Diagnosing the Bottleneck

The assistant's reasoning process, visible in [msg 7639], is a textbook example of systematic performance analysis. Rather than blindly tweaking parameters, the assistant first identified the actual bottleneck. The key insight was that the KV cache—typically the dominant memory consumer in transformer inference—was barely utilized. Each GPU had 183 GB of memory, the model occupied ~54 GB, and the KV cache was allocated at 35 GB but only 2-3% full. Meanwhile, 27 GB sat completely free. Yet SGLang had auto-capped max_running_requests at just 16.

The assistant correctly identified the real constraint: the Mamba SSM (State Space Model) cache. Qwen3.6-27B is a hybrid architecture combining transformer attention layers with Mamba-2 state space layers. The Mamba layers maintain a "conv state" and "SSM state" for each sequence being processed, and with the extra_buffer scheduler strategy (required for MTP speculative decoding), each running request consumes additional buffer slots. At the default configuration of 80 Mamba cache slots, the SSM state consumed ~11.4 GB and the intermediate caches another ~9.6 GB. This 21 GB footprint, combined with the overhead of the extra_buffer strategy, limited concurrency to 16 requests even though the KV cache was practically empty.

The reasoning explored several levers: disabling MTP to remove the extra_buffer constraint (but that would sacrifice the ~3x decode speedup from speculative decoding), reducing context length from 8192 to 4096, enabling FP8 KV cache compression, or increasing num_continuous_decode_steps. The assistant ultimately settled on the most impactful change: dramatically increasing the Mamba cache size from 80 to 200 slots, raising the mamba_full_memory_ratio from 0.4 to 0.6, and setting mem_fraction_static to 0.93 to allow SGLang to allocate more memory upfront. The target was to force SGLang's auto-scheduler to compute a higher max_running_requests value.

The Execution: Kill, Relaunch, Benchmark

The execution was swift and aggressive. At [msg 7640], the assistant confirmed the memory situation with nvidia-smi—155 GB used, 27 GB free per GPU—and declared the free memory "wasted." It then killed the running generation process and all 7 SGLang servers. This was a risky move: killing an active generation run mid-stream could lose progress. But the generation script had been designed with resume support via a .done_indices file, so the 3,981 completed samples were safe.

After confirming all processes were dead ([msg 7644]), the assistant relaunched with the tuned configuration at [msg 7645]. The new launch script set --max-mamba-cache-size 200, --mamba-full-memory-ratio 0.6, --mem-fraction-static 0.93, and --max-running-requests 64. The servers loaded in just 30 seconds ([msg 7646])—remarkably fast for a 54 GB model, thanks to loading from /dev/shm (a 923 GB RAM disk). The new allocation showed: Mamba cache at 200 slots consuming 28.3 GB for SSM state plus 23.1 GB for intermediate caches, KV cache reduced to 798K tokens (24.4 GB each for K and V), and max_total_num_tokens at 798,080. Critically, SGLang auto-calculated max_running_requests at 40—up from 16.

The generation was relaunched with concurrency increased from 48 to 96 ([msg 7647]), and after 90 seconds of ramp-up, the results came in.

The Results: A 75% Throughput Gain

Message [msg 7649] presents the comparison table:

| Metric | Before | After (tuned) | Change | |---|---|---|---| | Running req/GPU | 16 | 40 | 2.5x | | Per-GPU tok/s | ~2,000 | ~3,500 | +75% | | Total tok/s | ~14,000 | ~24,900 | +78% | | Request rate | 6.1/s | 8.5/s | +40% | | ETA | 41h | 29.6h | -28% | | GPU util | 83-99% | 99-100% | maxed | | Power draw | 815-934W | 920-993W | pushing harder |

The numbers tell a clear story. The 2.5x increase in concurrent requests per GPU translated into a 75% improvement in per-GPU generation throughput. The aggregate throughput jumped from ~14,000 to ~24,900 tok/s—a 78% gain. The request rate increased more modestly at 40%, because each request now produced slightly fewer output tokens on average (2,374 vs 2,490), likely because the higher concurrency caused more requests to be in prefill or early decode stages during the measurement window.

The GPU utilization hitting 99-100% across all 7 GPUs confirms that the bottleneck has shifted from the scheduler's concurrency cap to the raw compute capacity of the hardware. The power draw increase to 920-993W per GPU indicates the B200s are being pushed to their thermal and electrical limits—these GPUs have a TDP around 1000W, so the system is now operating at maximum capacity.

Assumptions and Potential Issues

The message also notes 18 failures (up from 2 in the previous run). The assistant attributes these to "higher concurrency causing some timeout races during ramp-up" and notes they "will be retried if we rerun." This is a reasonable assumption—when concurrency jumps from 48 to 96 requests feeding into servers that just started, some requests may arrive before the server is fully warmed up, causing connection timeouts. However, there's an alternative possibility: the higher Mamba cache utilization (147 Mamba slots in use out of 200, vs ~59 out of 80 before) might be causing memory pressure that manifests as sporadic failures for long-context requests. The 18 failures represent 0.002% of the 909,786 total samples, so they're statistically negligible, but they warrant monitoring.

Another assumption embedded in this tuning is that the extra_buffer scheduler strategy remains optimal. The assistant explicitly considered disabling MTP to remove the extra_buffer constraint and enable even higher concurrency, but judged that the 3x decode speedup from speculative decoding was worth more than the additional concurrency. This trade-off is correct for this workload (short prompts, long generations), but might not hold for other use cases.

Input and Output Knowledge

To fully understand this message, the reader needs knowledge of: the Mamba-2 architecture and its SSM state memory requirements; SGLang's scheduler strategies (extra_buffer vs alternatives); the relationship between KV cache size, Mamba cache size, and max_running_requests in SGLang's auto-configuration; the B200 GPU's memory capacity (183 GB) and TDP (~1000W); and the concept of MTP speculative decoding with acceptance length metrics.

The message creates new knowledge: it demonstrates that for hybrid Mamba-Transformer models on high-memory GPUs, the Mamba cache—not the KV cache—is the primary concurrency bottleneck, and that aggressively reallocating memory from KV to Mamba cache can yield 75% throughput improvements. This is a non-obvious finding that contradicts the typical inference optimization playbook, which focuses on KV cache compression and management.

The Significance

What makes this message remarkable is its density. In just a few lines, it encapsulates the entire arc of a performance optimization cycle: diagnosis, intervention, measurement, and validation. The assistant didn't just tweak parameters—it understood the memory architecture of a hybrid model, identified the real bottleneck (Mamba SSM state, not KV cache), reallocated resources accordingly, and validated the improvement with real metrics. The 75% throughput gain came not from any algorithmic breakthrough but from simply giving the scheduler more room to operate—a reminder that in modern ML serving, the hardware is often capable of far more than the default configuration allows.

The message also marks a critical inflection point in the broader session. With the ETA reduced to ~30 hours, the generation run became practically feasible. The 902,087 completions produced by this run would go on to enable the online training architecture for the DFlash drafter, ultimately allowing the team to train a speculative decoding model without the impractical 90 TB of storage that offline extraction would have required. This tuning round, executed in about 30 seconds of server downtime, saved roughly 11 hours of wall-clock time and hundreds of dollars in GPU compute costs.