The 3,740 tok/s Milestone: Pushing GLM-5-NVFP4 Inference to Its Limit on 8 Blackwell GPUs

Introduction

In the long arc of optimizing GLM-5-NVFP4 inference on eight RTX PRO 6000 Blackwell GPUs, there comes a moment where all the pieces finally click into place. Message 714 in this coding session is that moment — a brief, almost telegraphic report from the assistant that captures the culmination of dozens of prior iterations, bug fixes, kernel patches, and configuration tweaks. The message itself is deceptively short, but it represents the payoff for an enormous amount of work spanning multiple segments of the conversation. This article examines that single message in depth: what it says, what it means, and what it reveals about the process of squeezing peak performance out of a cutting-edge MoE inference stack on non-datacenter hardware.

The Message

The assistant writes:

2,800 tok/s at 512 concurrency! Peak output hit 2,348 tok/s! Let me push to 1024:

>

`` Output token throughput (tok/s): 1239.81 Peak output token throughput (tok/s): 3945.00 Peak concurrent requests: 1024 Total token throughput (tok/s): 3740.43 Concurrency: 774.37 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 40066.14 Median E2E Latency (ms): 41055.09 P90 E2E Latency (ms): 50844.35 P99 E2E Latency (ms): ... ``

The message is structured as a progress report: a celebratory headline ("2,800 tok/s at 512 concurrency!"), a decision ("Let me push to 1024"), and the results of that decision. The benchmark output is partially truncated — the P99 latency line cuts off with an ellipsis — but the critical numbers are all visible.

Why This Message Matters

To understand the significance of this message, one must appreciate the journey that led here. The assistant had been working for hours across multiple sessions to deploy and optimize GLM-5-NVFP4, a large Mixture-of-Experts model quantized to FP4, on eight RTX PRO 6000 GPUs in an LXC container running on a Proxmox host. The path was strewn with obstacles:

• NaN crashes during decode that required selecting specific NSA backends (trtllm instead of flashinfer)
• FlashInfer CUTLASS MoE autotune not running on SM120 (consumer Blackwell), requiring a manual patch to model_runner.py to include flashinfer_cutlass in the autotune list
• A hard-coded cap on max_running_requests that limited the server to 64 concurrent requests, later raised to 1024
• A page_table_1_flattened bug in the NSA attention path that caused crashes during chunked prefill with prefix sharing, eventually worked around by disabling the radix cache
• CUDA graph instability that led to disabling CUDA graphs entirely
• GPU underutilization with power draw stuck at ~250W out of a 600W TDP Each of these issues had been systematically addressed in the messages leading up to msg 714. The assistant had restarted the server multiple times, patched source code, tweaked launch parameters, and run incremental benchmarks. The progression tells the story: 689 tok/s at 64 concurrency (msg 693), then 1,950 tok/s at 256 concurrency (msg 695), then 2,800 tok/s at 512 concurrency (msg 714, first line), and finally 3,740 tok/s at 1024 concurrency (msg 714, benchmark output). This message is the climax. It answers the question that had been driving the entire optimization effort: how fast can this model go on this hardware?

Decisions Made in This Message

The message contains two key decisions, one explicit and one implicit.

Explicit decision: push concurrency to 1024. The assistant had just seen 2,800 tok/s at 512 concurrency and decided to double the concurrency to 1024. This was not a reckless move — it was a deliberate exploration of the throughput ceiling. The assistant had already set --max-running-requests 1024 in the server launch parameters (msg 700), so the server was configured to handle this load. The question was whether the system would remain stable and whether throughput would continue to scale.

Implicit decision: accept the latency trade-off. The assistant does not comment on the latency numbers, but they are striking: a mean end-to-end latency of 40 seconds, with P99 at over 50 seconds. At 1024 concurrency, individual requests take nearly a minute to complete. The assistant implicitly accepts this as the price of peak throughput — a classic batch-processing trade-off where latency is sacrificed for throughput. This is appropriate for a serving benchmark focused on finding the maximum throughput, but it would be unacceptable for interactive use cases.

Assumptions and Potential Issues

The assistant makes several assumptions in this message, some more justified than others.

The server will remain stable at 1024 concurrency. This assumption proved correct — the benchmark completed and produced results. However, earlier attempts at 512 concurrency had crashed with the page_table_1_flattened error (msg 696), and the fix (disabling radix cache) was only applied in msg 710. The assistant was right to be cautious, and the successful 512 run in msg 713 gave confidence to push further.

The benchmark tool will produce valid results at this scale. The bench_serving.py script generates random prompts and measures throughput. At 1024 concurrent requests with 256 input tokens and 128 output tokens each, the server is handling a substantial load. The assistant assumes the tool's measurements are accurate and that the random-prompt dataset is representative.

GPU utilization is adequate. Notably, the assistant does not check GPU power draw or utilization in this message. Earlier in the chunk, the assistant had observed that GPUs were drawing only ~250W out of 600W TDP, indicating significant underutilization. The 3,740 tok/s result, while impressive, might still leave headroom on the table. The assistant would later investigate this and discover that FlashInfer's allreduce fusion was unsupported on SM120, limiting GPU utilization.

The benchmark output is complete. The truncated P99 latency line suggests the output was cut off. The assistant does not re-run to get the full data, which means the complete latency distribution is unknown. This is a minor oversight — the key throughput numbers are present, but the tail latency story is incomplete.

Input Knowledge Required

To fully understand this message, a reader needs several pieces of context:

1. The hardware setup: 8x RTX PRO 6000 Blackwell GPUs (SM120 architecture) running in an LXC container on a Proxmox host, with PCIe P2P limitations due to virtualization.
2. The model: GLM-5-NVFP4, a large Mixture-of-Experts model quantized to FP4 using NVIDIA's ModelOpt framework. The model uses tensor parallelism across 8 GPUs.
3. The serving stack: SGLang with FlashInfer attention backend, TRT-LLM NSA backends for decode and prefill, and FlashInfer CUTLASS MoE runner for the expert computation.
4. The optimization history: The progression from ~880 tok/s through multiple configuration changes — enabling autotune, raising max-running-requests, disabling CUDA graphs, disabling radix cache — each step unlocking more throughput.
5. The benchmark methodology: sglang.bench_serving with random prompts (256 input tokens, 128 output tokens) at infinite request rate, measuring total token throughput, output token throughput, peak concurrency, and latency percentiles.
6. The known limitations: GPU power draw stuck at ~250W, allreduce fusion unsupported on SM120, PCIe bandwidth constraints from virtualization. Without this context, the numbers in msg 714 are just numbers. With it, they tell a story of systematic optimization against real hardware constraints.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

1. Peak throughput ceiling: 3,740 tok/s total throughput with 1,240 tok/s output throughput at 1024 concurrency. This is the highest throughput achieved in the entire session.
2. Peak output burst: 3,945 tok/s peak output token throughput, suggesting the system can briefly sustain nearly 4,000 output tokens per second under heavy load.
3. Scaling behavior: Throughput scales well from 256 to 1024 concurrency (1,950 → 2,800 → 3,740 tok/s), with no signs of saturation. This suggests further gains might be possible with even higher concurrency or more aggressive batching.
4. Latency at scale: Mean E2E latency of 40 seconds at 1024 concurrency, with P90 at 50.8 seconds. This establishes the latency profile for batch workloads.
5. Configuration validity: The combination of --disable-cuda-graph, --disable-radix-cache, --max-running-requests 1024, and the patched autotune is a working configuration for this model and hardware.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure and tone of the message. The exclamation marks and bold formatting convey genuine excitement — the assistant has been working toward this moment and the numbers are paying off. The progression from reporting the 512 result to immediately launching the 1024 test shows a methodical, iterative approach: verify stability at one level, then push to the next.

The decision to include the full benchmark output rather than just a summary suggests the assistant values transparency and precision. The numbers speak for themselves, and the assistant lets them do so.

The truncated P99 line is interesting. The assistant likely noticed it but chose not to re-run the benchmark — the key throughput numbers were already captured, and the latency story was clear enough from the mean, median, and P90 values. This is a pragmatic trade-off between completeness and speed.

The Aftermath

What follows msg 714 is a shift in focus. The assistant has found the throughput ceiling with the current configuration, but the GPU utilization problem remains. In subsequent messages, the assistant investigates why GPU power draw is stuck at ~250W, discovers that FlashInfer's allreduce fusion is gated on SM90/SM100 and doesn't support SM120, attempts to patch the kernel to add SM120 support (which fails with worse performance), and explores alternative MoE backends. The 3,740 tok/s result becomes the baseline against which further optimizations are measured.

Conclusion

Message 714 is a milestone in the optimization journey — the moment where all the accumulated fixes and tuning converge to produce the peak throughput. It is a testament to the power of systematic debugging and incremental improvement. The assistant's approach — identify a bottleneck, fix it, measure, repeat — transformed a server that could barely manage 689 tok/s into one delivering 3,740 tok/s, a 5.4x improvement. The message itself is brief, but it encapsulates hours of work, dozens of decisions, and a deep understanding of the interplay between model architecture, hardware constraints, and software configuration. For anyone working on large-scale model inference, it is a case study in how far you can push a system when you are willing to dig into every layer of the stack.