The Validation Point: How a 689 tok/s Disappointment Became a 1,950 tok/s Breakthrough

In the long arc of optimizing GLM-5-NVFP4 inference on eight RTX PRO 6000 Blackwell GPUs, there comes a moment that every engineer recognizes: the validation point. This is the instant when a series of speculative changes — patches to source code, tweaks to server parameters, enabling of previously disabled features — either pay off or prove to be a dead end. Message [msg 694] captures exactly such a moment, and it is a study in how careful reasoning, domain knowledge, and a willingness to trust a hypothesis over an initial disappointment can unlock a dramatic performance gain.

The Scene: After the Autotune

To understand what is happening in this message, we must first understand what led to it. The assistant had been engaged in a multi-session effort to deploy the GLM-5-NVFP4 mixture-of-experts model on a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe in a Proxmox virtualized environment. Earlier segments had resolved NaN crashes during decode, established baseline throughput around 757–880 tok/s, and identified that PCIe P2P latency was a bottleneck. But the assistant suspected there was more performance to be wrung out of the system.

In the messages immediately preceding [msg 694], the assistant had made three critical changes:

  1. Enabled FlashInfer CUTLASS MoE autotune by patching model_runner.py in the sglang source code. The autotune list originally excluded flashinfer_cutlass with a TODO comment noting compilation errors. The assistant uncommented it, adding the backend to the autotune list (see [msg 677]–[msg 678]).
  2. Raised --max-running-requests from a conservative 64 to an auto-detected 2048. This was inspired by examining the K2-Thinking benchmark log, which showed that model running with uncapped request limits achieved 5,816 tok/s peak throughput ([msg 670]).
  3. Added --disable-cuda-graph, again following the K2-Thinking configuration. CUDA graphs can improve performance by capturing a fixed computation graph, but they also limit flexibility in batch sizes and can introduce overhead for dynamic workloads. The server had just finished loading, and the assistant confirmed that the FlashInfer autotune had run successfully across all eight tensor-parallel ranks ([msg 691]). The stage was set for a benchmark.

The Message: A Hypothesis Under Test

Here is the subject message in full:

689 tok/s at 64 concurrency — slightly worse than before (757 tok/s). This is likely because --disable-cuda-graph hurts at low batch sizes. Let me test at higher concurrency where the real benefit should show:

>

`` Total input text tokens: 33093 Total generated tokens: 16548 Total generated tokens (retokenized): 16428 Request throughput (req/s): 10.06 Input token throughput (tok/s): 1300.31 Output token throughput (tok/s): 650.21 Peak output token throughput (tok/s): 1379.00 Peak concurrent requests: 256 Total token throughput (tok/s): 1950.52 Concurrency: ... ``

On its surface, this is a simple benchmark report. But the reasoning embedded in those first two sentences reveals a sophisticated mental model of how the system behaves.

The Reasoning: Why 689 tok/s Was Not a Setback

The assistant's first benchmark after restarting the server used 64 concurrent requests — the same concurrency level that had previously yielded 757 tok/s. The result was 689 tok/s, a roughly 9% regression. For a less experienced operator, this might have triggered a rollback or a search for what went wrong. But the assistant immediately identified the likely cause: --disable-cuda-graph incurs a per-request overhead that is most noticeable when batch sizes are small. At low concurrency, the overhead of not having a captured CUDA graph outweighs the flexibility benefit. The assistant understood that the optimization strategy was not aimed at low concurrency — it was aimed at enabling much larger batch sizes that would saturate the GPUs.

This is a crucial insight about modern inference serving. CUDA graphs work by capturing a sequence of GPU kernel launches into a single graph that can be replayed with minimal CPU overhead. This is excellent for latency-sensitive, low-concurrency scenarios where the same computation pattern repeats. But for throughput-oriented serving with variable batch sizes, the graph must be recaptured frequently, and the overhead of graph capture can negate the benefits. The K2-Thinking model, which achieved 5,816 tok/s, ran with --disable-cuda-graph, and the assistant correctly inferred that the same trade-off would apply to GLM-5.

The assistant's hypothesis was clear: the benefit of the three changes (autotune, higher max_running_requests, disabled CUDA graphs) would manifest at higher concurrency, where the autotuned MoE kernels could process larger batches more efficiently and the uncapped request limit would allow the server to keep more work in flight simultaneously. The 64-concurrency test was not a failure — it was a baseline check to confirm the server was stable and the changes hadn't broken anything.

The Result: A Dramatic Validation

The 256-concurrency benchmark validated the hypothesis spectacularly. Total token throughput jumped to 1,950.52 tok/s — more than double the previous best of 879 tok/s. Peak output token throughput reached 1,379 tok/s, indicating that the GPUs were finally being fed enough work to approach their computational limits. The request throughput of 10.06 req/s meant the server was completing ten requests per second, each generating 128 tokens of output.

This result is significant for several reasons. First, it confirms that the FlashInfer CUTLASS MoE autotune is working effectively on SM120 (the Blackwell architecture of the RTX PRO 6000). The autotune process, which runs during server startup, profiles different GEMM (general matrix multiply) tactics and selects the fastest ones for the specific GPU architecture. By enabling flashinfer_cutlass in the autotune list, the assistant unlocked MoE kernel optimizations that were previously unavailable.

Second, the jump from 64 to 256 concurrent requests demonstrates that the previous --max-running-requests 64 setting was a severe bottleneck. With only 64 slots for concurrent requests, the server could never build up enough batch parallelism to saturate eight GPUs. Raising the limit to 2048 (and later capping at 1024 for stability) allowed the server to accumulate a large queue of requests and process them in larger, more efficient batches.

Third, the result shows that --disable-cuda-graph is not a penalty at higher concurrency — it is an enabler. Without CUDA graph capture overhead, the server can dynamically adjust batch sizes and accommodate the variable computation patterns that arise from different requests reaching different stages of generation at different times.

The Broader Context: What This Message Enabled

Message [msg 694] is the turning point in segment 6 of the session. The 1,950 tok/s result was not the final peak — subsequent benchmarks at 512 and 1024 concurrency would push throughput to ~2,800 and ~3,740 tok/s respectively ([chunk 6.0]). But this message is where the assistant proved that the optimization direction was correct. Without this validation, the assistant might have reverted the changes and pursued a different strategy.

The message also reveals the assistant's working style: it does not treat benchmarks as pass/fail tests but as diagnostic tools. The 64-concurrency result was not a failure — it was a data point that confirmed the assistant's understanding of the system's behavior. The assistant then used that understanding to design a more revealing test at 256 concurrency.

Assumptions and Their Risks

The assistant made several assumptions in this message that deserve scrutiny. First, it assumed that the FlashInfer autotune had selected optimal tactics for SM120. In reality, the autotune log later revealed that some tactics were skipped due to initialization failures in the CUTLASS TMA (Tensor Memory Accelerator) grouped GEMM kernel ([msg 697]). The assistant was not aware of these failures at the time of [msg 694], but the benchmark results suggest that the autotune still found sufficiently good tactics to deliver the performance gain.

Second, the assistant assumed that 256 concurrency would be safe. It was — but the very next benchmark at 512 concurrency caused a server crash due to a 'PrefillMetadata' object has no attribute 'page_table_1_flattened' error ([msg 696]). This crash was not an OOM but a code bug in the NSA (Native Sparse Attention) path, likely triggered by the combination of large batches and disabled CUDA graphs. The assistant's assumption was reasonable but incomplete — it had not anticipated this particular code path issue.

Third, the assistant implicitly assumed that the performance gain would scale linearly with concurrency. It did not — the gain from 64 to 256 was dramatic, but further gains required additional tuning (like --disable-radix-cache and --num-continuous-decode-steps). The relationship between concurrency and throughput is bounded by GPU compute capacity, memory bandwidth, and communication overhead.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

Output Knowledge Created

This message produced several pieces of actionable knowledge:

  1. A validated optimization strategy: The combination of FlashInfer CUTLASS MoE autotune + high max_running_requests + disabled CUDA graphs works for GLM-5 on SM120.
  2. A new throughput baseline: 1,950 tok/s at 256 concurrency, more than doubling the previous best.
  3. Confirmation of the bottleneck diagnosis: The previous 64-request limit was confirmed as the primary bottleneck.
  4. A stability data point: The server handled 256 concurrent requests without issues (though 512 later proved problematic).
  5. Evidence for the trade-off analysis: The 64-concurrency regression confirmed that --disable-cuda-graph has a cost at low batch sizes, validating the assistant's understanding of the system.

Conclusion

Message [msg 694] is a masterclass in hypothesis-driven optimization. The assistant did not panic at a slightly worse initial result. Instead, it used domain knowledge about CUDA graphs and batch-size scaling to design a more revealing test, and that test delivered a breakthrough. The 1,950 tok/s result was not the end of the optimization journey — the assistant would go on to push throughput to nearly 4,000 tok/s and then grapple with the harder problem of GPU underutilization. But this message marks the moment when the optimization direction was validated, the bottleneck was confirmed, and the path forward became clear. It is a reminder that in performance engineering, the most important skill is not knowing the answers — it is knowing which questions to ask next.