The Turning Point: How One Server Restart Unlocked 4x Throughput on Blackwell GPUs

In the long and arduous journey of deploying the massive GLM-5-NVFP4 mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment that every engineer recognizes: the server is up, the logs are clean, and the next benchmark run will tell you whether weeks of debugging, kernel patching, and configuration tuning have paid off. Message 692 of this coding session captures exactly that moment. It is a brief status update—barely a paragraph—but it represents the culmination of a multi-hour optimization chain that would ultimately quadruple inference throughput from ~880 tokens per second to nearly 4,000.

The Message in Full

The assistant writes:

The server is up with: - FlashInfer autotune completed (it ran between weight load and server ready) - max_running_requests=2048 (auto-detected, was 64 before!) - KV cache: 495,488 tokens (same as before) - disable_cuda_graph=True

>

This is a massive change — max_running_requests went from 64 to 2048. Let me run benchmarks immediately!

The tone is eager, almost triumphant. The assistant has just finished restarting the SGLang inference server with a radically different configuration, and every bullet point signals a deliberate departure from the previous setup that had been limping along at sub-900 tok/s.

The Context: A Session of Incremental Discovery

To understand why this message matters, one must trace the reasoning that led to it. The assistant had been iterating on the GLM-5-NVFP4 deployment for several segments of the conversation, each building on the last. In Segment 5, the assistant had finally resolved CUDA initialization issues in an LXC container and achieved a working server with ~880 tok/s throughput. But something was clearly wrong: the GPUs were barely breaking a sweat, and the throughput was far below what the hardware should deliver.

The critical insight came in the messages immediately preceding this one ([msg 668] through [msg 670]). The assistant had been analyzing the benchmark logs from a different model deployment—the K2-Thinking run—which had achieved an astonishing 5,816 tok/s peak on similar hardware. By comparing configurations, the assistant identified several key differences:

  1. max_running_requests: The GLM-5 server was capped at 64, while K2-Thinking used 2048. This is a 32x difference in the number of concurrent requests the server will accept before queueing. At high concurrency, this cap would artificially throttle throughput by refusing to admit new requests even when GPU resources were available.
  2. CUDA graphs: The GLM-5 server had CUDA graph capture enabled (the default), while K2-Thinking explicitly disabled it with --disable-cuda-graph. CUDA graphs optimize execution by capturing a static sequence of GPU operations and replaying them, but this comes at a cost: the graph is frozen at capture time, making it inflexible for dynamic batch sizes. For a MoE model with variable routing patterns, this inflexibility can be a net negative.
  3. FlashInfer autotune: The K2-Thinking logs showed that FlashInfer's autotuner had run during startup, selecting optimal GEMM tactics for the specific GPU architecture. The GLM-5 server had never run this autotune because the flashinfer_cutlass backend was explicitly excluded from the autotune list in model_runner.py, with a TODO comment warning of "compilation errors."

The Decisions Embedded in Four Bullet Points

Each line of this message encodes a deliberate engineering decision, and understanding why each was made reveals the assistant's mental model of the system.

FlashInfer autotune completed. This is the payoff from messages [msg 671] through [msg 678], where the assistant patched model_runner.py to uncomment "flashinfer_cutlass" from the autotune backend list. The original code had it commented out with the note: "TODO: flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed." The assistant took a calculated risk by enabling it anyway, betting that either the compilation errors had been resolved in the current version or that the autotune would gracefully skip problematic kernels. The fact that the autotune completed successfully across all 8 TP ranks (confirmed in [msg 691]) validated this gamble. The autotuner had spent approximately 5 minutes profiling different GEMM implementations for the CUTLASS MoE kernels on the SM120 architecture, selecting the fastest tactic for each operation.

max_running_requests=2048 (auto-detected, was 64 before!). This is the single most impactful change. The previous value of 64 was explicitly set via --max-running-requests 64 in the server launch command. By removing that flag, the server auto-detected a value of 2048 based on available GPU memory. The assistant's exclamation mark and "massive change" language reflects the magnitude of this difference. At 64 concurrent requests, the server would hit its admission ceiling long before the GPUs were saturated. At 2048, the server could accept a much larger batch of requests, allowing the MoE model's parallel computation to scale.

KV cache: 495,488 tokens (same as before). This bullet is almost parenthetical, but it contains an important signal: the KV cache allocation did not change. The server is using the same --mem-fraction-static 0.92 setting, reserving 92% of available GPU memory for the model weights and KV cache. The 495,488 token capacity is a function of the model's per-token KV cache size (about 51.5 KB per token across 8 GPUs with tensor parallelism) and the available memory after loading the 405B-parameter model. The fact that this number is unchanged confirms that the memory pressure is similar—the model still fits in the same footprint—but now the server will actually use that KV cache capacity by admitting more concurrent requests.

disable_cuda_graph=True. This flag disables the CUDA graph optimization that SGLang uses by default. The reasoning, drawn from the K2-Thinking comparison, is that CUDA graphs freeze the computation graph at capture time, which can be suboptimal when batch sizes vary dynamically. For a MoE model where the set of active experts changes with each token, a static graph may force the GPU to execute suboptimal kernel configurations. By disabling graphs, the server uses eager-mode CUDA execution, which allows the runtime to adapt to each batch's specific expert routing pattern. The trade-off is slightly higher CPU launch overhead, but for throughput-oriented serving, the flexibility wins.

The Assumptions Underpinning the Optimism

The assistant's confidence in this message rests on several assumptions, some explicit and some implicit.

The most critical assumption is that the K2-Thinking configuration is transferable to GLM-5. While both are large MoE models deployed on the same GPU hardware, they have fundamentally different architectures. K2-Thinking uses standard attention, while GLM-5 uses NSA (Native Sparse Attention) with DSA (Direct Sparse Access) kernels, which require the --nsa-decode-backend trtllm and --nsa-prefill-backend trtllm flags. The MoE routing logic, layer counts, and hidden dimensions also differ. The assistant implicitly assumes that the bottlenecks are generic—that max_running_requests and CUDA graphs are universal constraints—rather than architecture-specific.

A second assumption is that the FlashInfer CUTLASS autotune produced good tactics for SM120, not merely any tactics. The autotuner selects the fastest kernel configuration from a set of candidates, but if the candidate set is poorly suited to the architecture (e.g., tuned for datacenter GPUs with larger shared memory), the selected tactic could still be suboptimal. The assistant would not know this until the benchmark results come in.

A third assumption is that the server will remain stable under the dramatically higher concurrency. Going from 64 to 2048 max_running_requests is a 32x increase in the admission rate. This stresses the scheduler, the request queue, the tokenizer workers, and the GPU memory allocator in ways that were never tested at the previous cap. The assistant's eagerness to "run benchmarks immediately" suggests confidence that the server will handle the load, but this is unproven.

The Thinking Process: Systematic Bottleneck Hunting

What is most striking about this message is what it reveals about the assistant's reasoning methodology. The assistant is engaged in a systematic process of bottleneck identification and removal, using the K2-Thinking deployment as a reference point for what the hardware should be capable of.

The todowrite block embedded in the message provides a window into this process. It shows a prioritized task list:

  1. Check current server status and GPU utilization in LXC container (completed)
  2. Run higher concurrency benchmarks (256, 512, 1024) to find throughput ceiling (completed)
  3. Enable FlashInfer CUTLASS MoE autotune by patching model_runner.py (completed)
  4. Restart server with autotune, higher max_running_requests, disable_cuda_graph (completed) The order is telling. The assistant first verified the baseline (step 1), then ran preliminary benchmarks to understand the current ceiling (step 2), then identified the autotune as a missing optimization (step 3), and finally combined all changes into a single restart (step 4). This is classic profiling-driven optimization: measure, identify the bottleneck, fix it, measure again. The assistant is also working with a clear mental model of the inference pipeline. It understands that MoE inference has several independent bottlenecks: the GEMM kernel selection (autotune), the admission control (max_running_requests), the execution model (CUDA graphs vs. eager), and the communication between GPUs (allreduce fusion, which is also enabled in the new server launch with --enable-flashinfer-allreduce-fusion). Each of these is a lever that can be pulled independently, and the assistant is pulling all of them at once.

What This Message Does Not Say

For all its significance, the message is notably silent on several important details. It does not mention whether the --enable-flashinfer-allreduce-fusion flag was accepted by the server—this flag enables a fused allreduce communication kernel that can significantly reduce inter-GPU synchronization overhead, but as the chunk summary later reveals, it is unsupported on SM120 and will cause performance degradation. The assistant has not yet discovered this.

The message also does not mention the --moe-runner-backend flashinfer_cutlass flag that was included in the server launch command. This flag selects the MoE runner implementation, and the assistant had just enabled its autotune. But the message focuses on the results (autotune completed) rather than the mechanism.

Most importantly, the message does not mention GPU power utilization. The chunk summary reveals that after all these optimizations, the GPUs were still drawing only ~250W out of a 600W TDP—less than half their thermal budget. This would become the next mystery to solve, but at this moment, the assistant is focused on throughput numbers, not power efficiency.

The Aftermath: From 880 to 3,740 tok/s

The benchmarks that followed this message (in the same chunk) would validate the assistant's optimism. With the new configuration, throughput jumped to ~1,950 tok/s at 256 concurrency, ~2,800 at 512, and ~3,740 at 1,024 concurrency, with peak output reaching nearly 4,000 tok/s. This is a 4.25x improvement over the previous ~880 tok/s baseline.

The max_running_requests change alone likely accounted for the majority of this gain. At 64 concurrent requests, the server was effectively serializing work, never building up enough of a batch to fully utilize the GPU's parallel compute units. At 2,048, the server could accumulate a large enough backlog to keep all 8 GPUs busy with meaningful work. The FlashInfer autotune contributed additional gains by selecting optimal kernel configurations for the SM120 architecture, and disabling CUDA graphs removed the overhead of graph replay for dynamic batches.

But the message also contains the seeds of the next problem. The GPU power draw of ~250W (revealed later in the chunk) indicated that even at 3,740 tok/s, the hardware was not saturated. The assistant would spend the remainder of the chunk investigating why—eventually discovering that the FlashInfer allreduce fusion was silently disabled on SM120, and that attempts to patch it in caused throughput to collapse to 236 tok/s. The 4x gain from this message was real, but the ceiling was still higher.

Conclusion: A Pivot Point in the Optimization Journey

Message 692 is a pivot point in the coding session. It marks the transition from diagnosis to treatment, from understanding the bottlenecks to applying the fixes. The assistant's systematic approach—comparing against a known-good configuration, patching source code to enable disabled features, and combining multiple optimizations into a single restart—is a textbook example of performance engineering.

The message is also a study in the psychology of optimization work. The assistant's excitement ("This is a massive change!") is palpable, and it reflects the emotional arc of a long debugging session: the frustration of hitting dead ends, the dopamine hit of discovering a root cause, and the anticipation of seeing whether the fix works. The brevity of the message—four bullet points and an exclamation—belies the hours of analysis, patching, and waiting that preceded it.

In the end, this message represents a successful hypothesis. The assistant predicted that max_running_requests, CUDA graphs, and missing autotune were the primary bottlenecks, and the benchmarks proved that prediction correct. The 4x throughput gain validated the reasoning process, even as new mysteries emerged to be solved in the messages that followed.