Breaking Through the Throughput Ceiling: 3,740 tok/s on GLM-5-NVFP4 with 8 Blackwell GPUs

Introduction

In the high-stakes world of large language model inference, every tok/s matters. When deploying a massive Mixture-of-Experts model like GLM-5-NVFP4 across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the difference between a system that delivers 800 tok/s and one that delivers 3,740 tok/s is the difference between a curiosity and a production-ready service. Message 715 of this opencode session captures the precise moment when that breakthrough was achieved—a milestone that represents the culmination of hours of iterative debugging, kernel patching, server configuration tuning, and architectural problem-solving.

This message, though brief in its surface appearance, is dense with meaning. It is simultaneously a victory announcement, a task management update, and a springboard for further optimization. To understand its full significance, we must examine not only what the message says, but the entire trajectory that led to this point and the decisions embedded within its concise prose.

The Message Itself

The message opens with an emphatic declaration:

3,740 tok/s at 1024 concurrency! Peak output 3,945 tok/s!

>

This is a massive improvement. Let me update the todo list and run one more test at 2048 to see where we plateau:

The assistant then invokes the todowrite tool to update its task tracking system, marking four high-priority items as completed:

  1. "Check current server status and GPU utilization in LXC container" — completed
  2. "Run higher concurrency benchmarks (256, 512, 1024) to find throughput ceiling" — completed
  3. "Enable FlashInfer CUTLASS MoE autotune by patching model_runner.py" — completed
  4. "Restart server with autotune, higher..." — completed (the full description is truncated in the output) At first glance, this appears to be a simple status update. But every element of this message encodes a deliberate decision and a specific piece of knowledge gained through trial and error.

The Journey to 3,740 tok/s

To appreciate the significance of this result, we must understand where the project started. Earlier in the session, the assistant had been struggling with throughput numbers around 689–879 tok/s ([msg 693]). The system was underperforming dramatically relative to the hardware's theoretical capacity—eight RTX PRO 6000 GPUs, each with substantial compute and memory bandwidth, should have been capable of far more.

The breakthrough came through a series of interconnected optimizations. First, the assistant enabled FlashInfer CUTLASS MoE autotune for SM120 by patching model_runner.py. This was a non-trivial modification: the autotuner needed to be told to include flashinfer_cutlass in its search space for the SM120 architecture (the compute capability identifier for Blackwell consumer GPUs). Without this patch, the MoE kernel selection was suboptimal, leaving GPU compute units idle.

Second, the assistant raised --max-running-requests from 64 to 1024. This parameter controls how many concurrent requests the scheduler will admit into the active batch. At low values, the GPU is underutilized because there aren't enough requests to fill its compute capacity. At higher values, the scheduler can maintain a full pipeline, keeping all eight GPUs busy. The jump from 64 to 1024 was aggressive but deliberate—the assistant had observed that the system was nowhere near memory capacity limits and could sustain much larger batches.

Third, the assistant disabled CUDA graphs (--disable-cuda-graph) and the radix cache (--disable-radix-cache). These decisions were born from necessity rather than optimization theory. CUDA graphs had been causing crashes with the NSA (Native Sparse Attention) backend, and the radix cache triggered a bug where PrefillMetadata lacked the page_table_1_flattened attribute (<msg id=705-708>). Rather than spending time debugging these issues in a complex codebase, the assistant chose to disable the problematic features—a pragmatic trade-off that sacrificed some theoretical performance for stability.

Technical Decisions and Trade-offs

The message reveals several implicit assumptions and decisions. The assistant assumes that the 3,740 tok/s result is stable and reproducible, not a one-time spike. This is a reasonable assumption given that the benchmark (sglang.bench_serving) runs multiple requests and reports aggregate statistics, but it is still an assumption that would need verification under sustained load.

The decision to push to 2048 concurrency next is also revealing. The assistant is probing for the throughput ceiling—the point where adding more concurrent requests no longer increases total throughput because the system becomes bottlenecked by something else (memory bandwidth, PCIe transfers, or scheduler overhead). The fact that throughput scaled almost linearly from 256 to 512 to 1024 (1,846 → 2,800 → 3,740 tok/s) suggests the ceiling hasn't been reached yet, but it must exist somewhere.

A critical assumption embedded in this message is that the NSA bug with page_table_1_flattened is fully circumvented by disabling the radix cache. The assistant had earlier discovered that this attribute was only set when prefix sharing was enabled ([msg 708]), and disabling the radix cache eliminates prefix sharing entirely. This is a correct diagnosis, but it means the system cannot benefit from prefix caching—a feature that would significantly improve latency for requests with shared prefixes (e.g., system prompts in chat applications).

Knowledge Created by This Message

This message creates several pieces of valuable output knowledge:

Empirical throughput bounds: The GLM-5-NVFP4 model on 8x RTX PRO 6000 GPUs can achieve at least 3,740 tok/s total throughput and 3,945 tok/s peak output throughput at 1024 concurrent requests. These are real, measured numbers that can guide capacity planning and deployment decisions.

Scaling behavior confirmation: The system scales with concurrency, at least up to 1024 requests. The ratio of total token throughput to output token throughput (roughly 3:1) suggests that input processing (prefill) and output generation (decode) are both scaling, with prefill contributing about two-thirds of the total throughput.

Validation of the optimization stack: The combination of FlashInfer CUTLASS MoE autotune, the flashinfer_cutlass MoE runner backend, the trtllm NSA backends, and the --disable-cuda-graph --disable-radix-cache flags forms a working, high-performance configuration for this specific hardware-software combination.

Task tracking as process documentation: The todo list update serves as a form of process documentation. By marking items as completed with specific descriptions, the assistant creates an audit trail of what was done and in what order. This is valuable for reproducibility and for onboarding other engineers to the project.

The Thinking Process

The message's structure reveals the assistant's thinking process. It leads with the result—the most important information—then immediately pivots to the next action. This "result-first, action-next" pattern is characteristic of effective engineering communication: celebrate briefly, then move on to the next experiment.

The phrase "Let me update the todo list and run one more test at 2048 to see where we plateau" is particularly telling. The assistant is managing its own cognitive load by externalizing task state into the todo system. It knows that the next logical step is to probe the throughput ceiling, but it also knows it needs to close the loop on completed tasks to maintain a clean working context.

The choice to use todowrite rather than a simple note is also significant. Throughout this session, the assistant has been maintaining a structured todo list with priorities and statuses. This systematic approach allows it to track progress across multiple rounds of tool calls and server restarts without losing sight of the overall objective.

Conclusion

Message 715 is a milestone in the optimization of GLM-5-NVFP4 inference on Blackwell GPUs. The 3,740 tok/s result represents roughly a 4x improvement over the starting point of ~880 tok/s, achieved through a combination of kernel autotuning, batch size scaling, and pragmatic workarounds for software bugs. But the message is more than just a number—it is a snapshot of an engineering process that values empirical measurement, systematic experimentation, and disciplined task management.

The next step—probing at 2048 concurrency—would reveal whether the system can scale further or whether the throughput ceiling has been reached. But regardless of what that test shows, message 715 stands as a testament to the power of iterative optimization: start with a working baseline, measure carefully, fix what's broken, tune what's working, and keep pushing until you find the limit.