The Pivot: Balancing Throughput and Stability in GLM-5-NVFP4 Inference

In the high-stakes world of large language model serving, a single decision can mean the difference between a breakthrough and a dead end. Message [msg 700] captures one such moment — a carefully reasoned pivot after a server crash threatened to derail an optimization campaign that had just doubled throughput. The assistant, having pushed GLM-5-NVFP4 inference from ~880 tok/s to an exhilarating 1,950 tok/s, now faced a crash during the next scaling step. The response was not a panicked retreat but a measured recalibration: keep the controversial --disable-cuda-graph flag, set --max-running-requests to a conservative 1024, and relaunch. This message is a masterclass in empirical decision-making under uncertainty, and it deserves close examination.

The Context: A Breakthrough and a Crash

The story leading to [msg 700] begins with a flash of insight. In [msg 670], the assistant compared the current GLM-5-NVFP4 setup against a reference configuration called "K2-Thinking" that had achieved an impressive 5,816 tok/s. The critical differences were stark: K2 used --disable-cuda-graph=True, had --max-running-requests uncapped at 2048, and ran FlashInfer CUTLASS MoE autotune. The current GLM-5 setup, by contrast, was capped at a mere 64 running requests and had CUDA graphs enabled. The assistant immediately set to work.

Over the next several messages, the assistant patched the SGLang codebase to enable flashinfer_cutlass in the autotune list (<msg id=671-678>), stopped the old server, and launched a new one with the K2-inspired configuration ([msg 682]). The server loaded successfully with the autotune running across all 8 tensor-parallel ranks (<msg id=690-692>). Then came the payoff: at 256 concurrency, throughput hit 1,950 tok/s — more than double the previous best of 879 tok/s ([msg 694]). The assistant, visibly excited, declared "Let me push higher!" ([msg 695]) and launched a 512-concurrency benchmark.

The server crashed.

But the crash was not the simple out-of-memory error the assistant initially suspected. Investigation revealed an AttributeError: &#39;PrefillMetadata&#39; object has no attribute &#39;page_table_1_flattened&#39; (<msg id=696-697>) — a code bug in the NSA (Native Sparse Attention) path, likely triggered by the combination of large batches and the absence of CUDA graphs. The 256-concurrency run had completed successfully, but 512 pushed the server over the edge.

The Decision: Reversing Course on CUDA Graphs

In [msg 698], immediately after diagnosing the crash, the assistant announced: "Let me restart with CUDA graphs enabled to get better low-batch performance, and set --max-running-requests to something reasonable like 512." This was a natural response — if disabling CUDA graphs contributed to the crash, re-enabling them would be the safe choice.

But by [msg 700], the assistant had reconsidered. The reasoning is laid out explicitly in the message's opening sentence:

"Now restart — keep --disable-cuda-graph since the crash was related to the prefill metadata, not cuda graphs per se, and the K2-Thinking run achieved 5,816 tok/s without cuda graphs."

This is a textbook example of causal reasoning in debugging. The assistant distinguished between correlation and causation: the crash coincided with --disable-cuda-graph being active, but the root cause was a missing attribute in the prefill metadata code path — a software bug, not a resource exhaustion or synchronization issue that CUDA graphs might have mitigated. The K2-Thinking run served as a powerful counterexample: if a similar configuration achieved 5,816 tok/s without CUDA graphs, then the flag itself was not inherently destabilizing.

The assistant also revised the --max-running-requests target upward from the previously planned 512 to 1024. This is another interesting decision. The crash occurred at 512 concurrency, so setting the cap to 512 would be the most conservative choice. Setting it to 1024 suggests the assistant believed the crash was a transient bug rather than a hard resource limit — and that the bug might not reproduce at every run. It's a calculated risk: 1024 is high enough to demonstrate the system's potential ceiling but not as aggressive as the auto-detected 2048 that was active during the crash.

The Launch Command: A Cumulative Configuration

The bash command in [msg 700] is more than just a server restart — it is the accumulated wisdom of an entire optimization campaign distilled into a single line. Every flag tells a story:

Assumptions and Risks

The assistant's decision in [msg 700] rests on several assumptions, each carrying its own risk:

Assumption 1: The crash was unrelated to CUDA graphs. The assistant concluded that the page_table_1_flattened error was a pre-existing code bug in the NSA path, not triggered by --disable-cuda-graph. This is a reasonable inference — the attribute name suggests a missing data structure in the prefill metadata, not a CUDA graph issue. However, it is possible that CUDA graphs, by enforcing a fixed execution pattern, paper over certain race conditions or memory ordering issues that surface when graphs are disabled. If so, the crash could recur.

Assumption 2: K2-Thinking is a valid reference. The assistant repeatedly invokes the K2-Thinking run (5,816 tok/s) as proof that --disable-cuda-graph is not inherently problematic. But K2-Thinking may have been running a different model, different batch sizes, or different software versions. The transferability of its configuration to GLM-5-NVFP4 is not guaranteed.

Assumption 3: 1024 is a safe upper bound. The crash occurred at 512 concurrency, yet the assistant set the cap to 1024. This assumes the crash was a fluke or a non-deterministic bug that won't reproduce. If the bug is deterministic at high concurrency, the server will crash again.

Assumption 4: The autotune results persist across restarts. The FlashInfer CUTLASS autotune had completed successfully in the previous run, selecting optimal tactics for SM120. The assistant assumes the same autotune will succeed in the new run — but autotuning is a profiling process that can fail due to timing, memory pressure, or kernel compilation issues.

Knowledge Flow

To fully understand [msg 700], one must be familiar with: SGLang's server architecture and its myriad flags; the distinction between CUDA graphs (which capture and replay GPU operations) and dynamic execution; the role of FlashInfer and TRT-LLM as attention and MoE backends; the NSA (Native Sparse Attention) mechanism used by GLM-5; the concept of tensor parallelism and its interaction with PCIe P2P; and the specific hardware constraints of Blackwell SM120 GPUs versus datacenter SM100 GPUs.

The message creates new knowledge in several forms. First, it establishes a hypothesis about the crash's root cause — that it was a prefill metadata bug, not a CUDA graph issue — which will be tested by the next benchmark run. Second, it sets a new operating point (1024 max-running-requests, no CUDA graphs) that will serve as the baseline for subsequent optimization. Third, it implicitly documents the assistant's decision-making process, preserving the reasoning behind each flag choice for future reference.

Conclusion

Message [msg 700] is a moment of equilibrium in a turbulent optimization process. The assistant had just witnessed a stunning 2x throughput improvement followed by a server crash. The natural instinct would be to retreat to safety — re-enable CUDA graphs, lower the concurrency cap. Instead, the assistant held its ground, trusting the evidence: the crash was a bug, not a fundamental limitation; the K2-Thinking reference validated the approach; and the potential reward (approaching 5,816 tok/s) justified the risk.

This message exemplifies the kind of reasoning that separates effective system optimization from trial-and-error flailing: the ability to distinguish correlation from causation, to use reference configurations as evidence rather than dogma, and to make calibrated risk assessments under uncertainty. Whether the next benchmark confirms or refutes these assumptions, the reasoning itself is a model worth studying.