The Pivot: How a Single Server Restart Unlocked 4x Throughput on Blackwell GPUs

In the course of a lengthy optimization session for deploying the GLM-5-NVFP4 model on 8x NVIDIA RTX PRO 6000 Blackwell GPUs, one message stands out as a critical inflection point. Message [msg 682] is the moment where the assistant, after hours of debugging, patching, and comparative analysis, launches a server with fundamentally different parameters — a restart that would ultimately catapult throughput from roughly 880 tokens per second to nearly 4,000. This message is not merely a command execution; it is a synthesis of everything learned up to that point, a deliberate hypothesis about what was bottlenecking the system, and a bet on a set of configuration changes derived from a prior successful deployment of a similar model.

The Context: Stuck at 880 tok/s

To understand why [msg 682] matters, one must appreciate the frustration that preceded it. The assistant had successfully deployed GLM-5-NVFP4 on the 8-GPU LXC container, resolved CUDA initialization issues, and established a baseline throughput. But the baseline was disappointing: around 880 tokens per second at high concurrency, with GPU power draw hovering near 250 watts per card — less than half the 600-watt thermal design power (TDP) of the RTX PRO 6000 Blackwell. The hardware was fundamentally underutilized, and the assistant had been systematically investigating why.

The investigation had taken several paths. The assistant examined MoE kernel configuration files, checked whether the Triton MoE configs were being loaded correctly, and dug into the FlashInfer CUTLASS code path for NVFP4 quantization. It consulted a local research repository (/home/theuser/glm-kimi-sm120-rtx6000bw) from a prior deployment of Kimi K2-Thinking on the exact same hardware — a goldmine of empirical data. By reading the K2-Thinking benchmark logs, the assistant discovered a critical discrepancy: the K2 deployment had achieved 5,816 tok/s peak throughput using a configuration that differed from the current GLM-5 setup in several key ways.

The Diagnostic Leap: What the K2 Logs Revealed

Message [msg 670] captures the moment of insight. The assistant enumerated three critical differences between the K2-Thinking deployment and the current GLM-5 setup:

1. --max-running-requests: The GLM-5 server was capped at 64 concurrent requests. The K2 run used 2048 (effectively uncapped). This was identified as the "most likely bottleneck at high concurrency."
2. --disable-cuda-graph: The K2 run disabled CUDA graphs, while the GLM-5 server had them enabled. CUDA graphs can reduce launch overhead for fixed-size batches, but they also impose rigidity — if batch sizes vary, graph recompilation can be expensive. For a production server with variable request sizes, disabling graphs can actually improve throughput.
3. Attention backend: K2 used attention_backend='triton', while GLM-5 used flashinfer. This difference was understood to be model-specific: GLM-5 uses Decoupled Shared Attention (DSA) which requires the flashinfer backend for its NSA (Native Sparse Attention) kernels. The assistant correctly judged this was not a changeable parameter. Additionally, the assistant had just patched the FlashInfer CUTLASS MoE autotune. The code in model_runner.py had a telling TODO comment: "flashinfer_cutlass will cause some flashinfer compilation errors. To be fixed." The flashinfer_cutlass backend was explicitly excluded from the autotune list. The assistant overrode this exclusion in [msg 677] by uncommenting the relevant line, enabling autotune for SM120 — a gamble that the compilation errors either wouldn't manifest or would be non-fatal.

The Message Itself: A Deliberate Synthesis

Message [msg 682] opens with the assistant summarizing the key changes:

Good. Now let me start the server with optimized parameters. Key changes from before: - Remove --max-running-requests 64 (let it auto-detect, K2 used 2048) - Add --disable-cuda-graph (K2-Thinking used this; avoids graph capture overhead and allows flexible batch sizes) - Keep --moe-runner-backend flashinfer_cutlass (with autotune now enabled) - Keep --attention-backend flashinfer (GLM-5 uses DSA which needs flashinfer for NSA)

This is not a random tweak. Each decision is explicitly justified with a reference to empirical evidence from the K2 run. The assistant is practicing evidence-based optimization: rather than guessing, it is transferring validated configuration patterns from one model to another, while respecting architectural differences (the attention backend constraint).

The bash command that follows is a dense bundle of engineering choices. Let us unpack its components:

• source /root/ml-env/bin/activate: Activates the Python virtual environment with all dependencies.
• PYTHONUNBUFFERED=1: Ensures log output is not buffered, critical for real-time debugging.
• NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=5 NCCL_MIN_NCHANNELS=8: NCCL tuning for the specific interconnect topology. P2P_LEVEL=5 enables NVLink peer-to-peer, MIN_NCHANNELS=8 ensures sufficient communication channels for 8 GPUs.
• OMP_NUM_THREADS=8: Limits OpenMP threads to prevent oversubscription.
• SAFETENSORS_FAST_GPU=1: Accelerates model loading by using GPU for safetensor deserialization.
• CUDA_HOME=/usr/local/cuda-12.8: Points to the secondary CUDA toolkit (12.8) that was installed specifically for flash-attn compatibility, rather than the system CUDA 13.1. The server flags encode the optimization strategy:
• --tp 8: Tensor parallelism across all 8 GPUs.
• --mem-fraction-static 0.92: Aggressive memory allocation (up from defaults).
• --quantization modelopt_fp4: The NVFP4 quantization format specific to this model.
• --attention-backend flashinfer --nsa-decode-backend trtllm --nsa-prefill-backend trtllm: The attention stack that was painstakingly debugged earlier in the session to resolve NaN crashes.
• --moe-runner-backend flashinfer_cutlass: The MoE execution backend, now with autotune enabled.
• --enable-flashinfer-allreduce-fusion: An optimization that fuses allreduce communication with kernel execution — though this would later prove problematic on SM120.
• --disable-cuda-graph: The K2-derived change to allow flexible batch sizes.

Assumptions and Risks

The message rests on several assumptions, some of which would prove incorrect:

That the FlashInfer CUTLASS autotune would work on SM120. The sglang source code explicitly warned that this caused "compilation errors." The assistant assumed either that the errors were non-fatal (perhaps warnings that could be ignored) or that the SM120 architecture would be handled gracefully. In practice, the autotune did work — the server started successfully — but the subsequent allreduce fusion attempt would fail badly on SM120, dropping throughput to 236 tok/s and requiring a revert.

That K2-Thinking configuration patterns transfer to GLM-5. While both models use NVFP4 quantization and run on the same hardware, they have different architectures (K2 uses a different MoE layout, different attention mechanisms). The assistant assumed the bottlenecks were generic (concurrency limits, graph overhead) rather than model-specific. This assumption was largely validated by the subsequent throughput improvements.

That --enable-flashinfer-allreduce-fusion is beneficial. The K2 run did not use this flag. The assistant kept it from the previous configuration without re-evaluating whether it was appropriate for SM120. This would later prove to be a mistake — the allreduce fusion kernels from TRT-LLM only supported SM90 (Hopper) and SM100 (datacenter Blackwell), not SM120 (consumer Blackwell).

That removing --max-running-requests entirely is safe. Without an explicit cap, the server could theoretically accept more requests than it can handle, leading to memory exhaustion or degraded latency. The assistant trusted that the server's internal scheduling would handle this gracefully.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in the messages leading up to [msg 682] reveals a systematic, hypothesis-driven approach. In [msg 670], the assistant explicitly enumerates the three differences between the current and reference configurations, ranks them by likely impact ("--max-running-requests 64 is the most likely bottleneck"), and formulates a plan. In [msg 671], it locates the autotune gating code in model_runner.py and assesses the risk of enabling it. In [msg 674], it weighs the tradeoffs: "The comment says 'flashinfer_cutlass will cause some flashinfer compilation errors.' That's risky. Let me take a different approach — the quickest win is to..." — before deciding to proceed anyway.

This is not blind trial-and-error. The assistant is building a mental model of the system's bottlenecks, cross-referencing it with empirical data from a similar deployment, and forming a multi-part hypothesis: (1) concurrency limiting is the primary bottleneck, (2) CUDA graph overhead is a secondary bottleneck, (3) MoE kernel autotuning can improve per-request efficiency. The server restart in [msg 682] is the test of this hypothesis.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

• The GLM-5-NVFP4 model: Its architecture (MoE with DSA attention), its quantization format (modelopt_fp4/NVFP4), and its specific requirements (NSA backends, flashinfer attention).
• The hardware: 8x RTX PRO 6000 Blackwell GPUs (SM120 architecture), their interconnect topology (PCIe with NVLink P2P), and their power characteristics (600W TDP).
• The software stack: sglang server architecture, FlashInfer kernel library, NCCL communication library, CUDA graph capture semantics, and the MoE execution backends (flashinfer_cutlass, flashinfer_trtllm, flashinfer_mxfp4).
• The prior K2 deployment: That a similar model (Kimi K2-Thinking) was successfully deployed on the same hardware with specific configuration choices, and that its logs are available for comparison.
• The autotune mechanism: That FlashInfer has a warmup/autotune phase that profiles and selects optimal kernel configurations, and that this was gated behind a backend whitelist that excluded SM120.

Output Knowledge Created

This message creates several forms of knowledge:

• A validated configuration baseline: The combination of --disable-cuda-graph, removed --max-running-requests cap, and enabled FlashInfer CUTLASS autotune becomes the new baseline for further optimization.
• Evidence that K2 configuration patterns transfer to GLM-5: The subsequent throughput improvements (from ~880 to ~3,740 tok/s) validate the hypothesis that concurrency and graph overhead were the primary bottlenecks, not model-specific architecture issues.
• Documentation of the autotune patch: The modification to model_runner.py is a concrete, reproducible change that future deployments on SM120 can adopt.
• A negative result about allreduce fusion: The subsequent failure of --enable-flashinfer-allreduce-fusion on SM120 (not visible in this message but occurring in the same chunk) creates knowledge about the limitations of TRT-LLM communication kernels on consumer Blackwell hardware.

Conclusion

Message [msg 682] is a masterclass in evidence-driven optimization. It demonstrates how to synthesize findings from multiple investigative threads — code inspection, log analysis, comparative benchmarking, and source patching — into a single, decisive action. The assistant does not tweak parameters randomly; it forms a hypothesis, justifies each change with reference to empirical data, and executes the test cleanly. The subsequent results would show that this single restart improved throughput by over 4x, proving that the diagnostic work in the preceding messages was correct. It is a reminder that in systems optimization, the most valuable skill is not knowing the right answer, but knowing where to look for evidence.