The Plateau at 806 Tok/s: Diagnosing Inference Throughput Limits on Blackwell GPUs

Introduction

In the high-stakes world of large language model inference, every token per second counts. When you have invested in eight NVIDIA RTX PRO 6000 Blackwell GPUs, migrated from a KVM virtual machine to an LXC container to unlock true bare-metal GPU topology with peer-to-peer (P2P) memory access at 53 GB/s, and finally gotten the GLM-5-NVFP4 model to serve requests without crashing, the natural expectation is a dramatic leap in throughput. But the numbers tell a more complicated story. In message [msg 647], the assistant running an opencode coding session confronts a sobering reality: despite all the infrastructure improvements, the throughput numbers are plateauing, and the root cause remains elusive.

This message captures a pivotal moment in the session — the transition from infrastructure validation to performance optimization. After spending hours resolving CUDA initialization blockers, installing missing build tools, and wrestling with NVIDIA driver compatibility, the assistant has finally achieved a working inference server. But the benchmark results raise more questions than they answer. Why does 128 concurrent requests yield only 806 total tokens per second? Why does the throughput curve flatten so early? And what does a cryptic warning about "DeepGemm scale format" mean for the Blackwell architecture?

The Message in Full

The subject message contains the assistant's analysis and two parallel tool calls:

757 tok/s at 64 concurrency. The throughput is actually lower than the KVM VM at 64 concurrency (~485 tok/s output, but wait — the VM numbers included input tokens differently). Let me try higher concurrency and also check the server log for any warnings about missing MoE kernel configs: [bash] ssh root@10.1.230.174 "grep -i 'moe\|kernel config\|default\|sub-optimal\|warning' /root/sglang-server.log 2>/dev/null | head -20" Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [2026-02-19 05:21:47] WARNING _http.py:779: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. [2026-02-19 05:21:47] WARNING model_config.py:955: DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell. [bash] ssh root@10.1.230.174 "/root/ml-env/bin/python3 -m sglang.bench_serving --backend sglang-oai --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 --base-url http://localhost:8000 --dataset-name random --num-prompts 128 --random-input-len 256 --random-output-len 128 --request-rate inf 2>&1 | tail -30" Total input text tokens: 16225 Total generated tokens: 8292 Total generated tokens (retokenized): 8286 Request throughput (req/s): 4.21 Input token throughput (tok/s): 533.42 Output token throughput (tok/s): 272.61 Peak output token throughput (tok/s): 453.00 Peak concurrent requests: 128 Total token throughput (tok/s): 806.04 Concurrency: ...

The message is deceptively simple: two bash commands and their outputs. But the reasoning embedded in the assistant's commentary reveals a sophisticated diagnostic process at work.

Context: The Long Road to a Working Server

To understand the significance of message [msg 647], we must appreciate what preceded it. The session had been a grueling multi-hour effort spanning several segments. The assistant had:

1. Set up the environment on Ubuntu 24.04 with NVIDIA drivers and CUDA Toolkit 13.1, creating a Python virtual environment with PyTorch using uv (segment 0).
2. Resolved flash-attn build issues by reducing parallel compilation jobs from 128 to 20 to avoid memory exhaustion, and rebuilding against the correct PyTorch version (segment 0).
3. Deployed GLM-5-NVFP4 on 8 GPUs using SGLang, but encountered persistent NaN crashes during decode (segment 1).
4. Resolved the NaN crash by selecting working NSA backends (trtllm) and established baseline throughput (segment 2).
5. Investigated P2P DMA limitations in the Proxmox VM environment, discovering that virtualization-induced PCIe latency was a key bottleneck (segment 3).
6. Attempted to bypass VFIO/IOMMU limitations by migrating to an LXC container on the Proxmox host, which showed correct bare-metal GPU topology (segment 4).
7. Resolved CUDA initialization in LXC by disabling the HMM feature in nvidia_uvm, then successfully launched the SGLang server (segment 5, chunk 0). By the time we reach message [msg 647], the assistant has just completed the first successful benchmark runs. The 32-concurrency test yielded 438 total tok/s, and the 64-concurrency test hit 757 total tok/s. These numbers are comparable to the KVM VM baseline (~485 tok/s peak output at 64 concurrency), which is puzzling because the LXC container should provide significantly better GPU-to-GPU communication.

The Reasoning Behind the Message

The assistant's thought process in [msg 647] reveals a careful diagnostic approach. The message begins with a comparison: "757 tok/s at 64 concurrency. The throughput is actually lower than the KVM VM at 64 concurrency (~485 tok/s output, but wait — the VM numbers included input tokens differently)." This parenthetical self-correction is telling. The assistant is realizing that the comparison isn't apples-to-apples — the KVM VM numbers may have been measuring output tokens only, while the current benchmarks report total tokens (input + output). The 757 tok/s total at 64 concurrency might actually be an improvement when accounting for the different measurement methodology.

But the assistant doesn't dwell on this ambiguity. Instead, it pivots to two parallel investigations: (1) checking the server log for warnings about MoE kernel configurations, and (2) running a higher-concurrency benchmark at 128 requests. This dual-pronged approach is characteristic of the session's methodology — gather more data while simultaneously inspecting system state.

The choice to grep for moe\|kernel config\|default\|sub-optimal\|warning is not arbitrary. The assistant has prior knowledge (from the user's hints in earlier messages) that Blackwell GPU architecture (SM120) requires custom MoE kernel tuning. The PR #14311 (already merged into SGLang) and user comments about kernel configuration suggest that the default MoE kernels may be suboptimal for Blackwell. The assistant is checking whether the server log contains any warnings about using default configurations that should be overridden.

The DeepGemm Warning: A Critical Clue

The grep output reveals two warnings. The first is a mundane HF Hub authentication warning. The second is far more interesting: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell."

This warning is a window into the complex quantization landscape of modern LLM inference. DeepGemm is a specialized library for efficient matrix multiplication on NVIDIA hardware, and it uses a specific scale format (ue8m0) that the GLM-5-NVFP4 checkpoint may not be using. The "NVFP4" in the model name refers to NVIDIA's FP4 quantization format, and the checkpoint's scale factors may be stored in a different format than what DeepGemm expects. The assistant notes this warning but doesn't act on it immediately — the immediate concern is throughput, not accuracy, and the warning is about "accuracy degradation," not crashes or performance issues.

However, this warning is a red flag that the quantization pipeline may not be fully optimized for the Blackwell architecture. If the scale format mismatch causes suboptimal kernel selection, it could contribute to the throughput plateau. The assistant files this information away for later investigation.

The 128-Concurrency Benchmark: Diminishing Returns

The second bash command runs the benchmark at 128 concurrent requests. The results show:

• Total token throughput: 806.04 tok/s (up from 757 at 64 concurrency)
• Output token throughput: 272.61 tok/s (up from 219 at 64 concurrency)
• Peak output token throughput: 453.00 tok/s (up from 417 at 64 concurrency) The scaling from 64 to 128 concurrency yields only a 6.5% improvement in total throughput (757 → 806 tok/s). This is a classic sign of diminishing returns — the system is hitting some bottleneck that prevents linear scaling with concurrency. The assistant doesn't explicitly state the conclusion in this message, but the implication is clear: something is limiting throughput, and it's not request concurrency. The bottleneck could be in the GPU kernels themselves (compute-bound), in the PCIe interconnect (communication-bound), or in the SGLang server's scheduling logic (software-bound). The fact that the throughput plateau occurs at roughly the same level as the KVM VM suggests that the P2P topology improvement may not be the dominant factor — or that the improvement is being masked by another bottleneck.

Assumptions and Potential Misconceptions

Several assumptions underpin the assistant's reasoning in this message:

1. The comparison to KVM VM throughput is meaningful. The assistant implicitly assumes that the KVM VM numbers (from segment 2) are directly comparable to the LXC numbers. But as the assistant itself notes, the VM numbers may have used different measurement methodology (output-only vs. total tokens). This makes the comparison unreliable.
2. Higher concurrency should yield higher throughput. The assistant expects that increasing from 64 to 128 concurrent requests will proportionally increase throughput. The fact that it doesn't is treated as anomalous. But in practice, many inference servers hit a saturation point where adding more requests increases queueing delay without improving throughput.
3. The MoE kernel configuration is the likely culprit. The assistant's decision to grep for MoE-related warnings reveals an assumption that the MoE kernel configuration is suboptimal. This is informed by the user's earlier hints about PR #14311 and Blackwell tuning, but it may not be the only — or even primary — bottleneck.
4. The server is correctly configured. The assistant assumes that the command-line flags passed to SGLang (e.g., --attention-backend flashinfer, --nsa-decode-backend trtllm, --moe-runner-backend flashinfer_cutlass) are the optimal combination for this model and hardware. This may not be true — the interaction between these backends is complex and poorly documented.

Input Knowledge Required

To fully understand this message, the reader needs:

• Knowledge of the GLM-5-NVFP4 model: It's a Mixture-of-Experts (MoE) model with FP4 quantization, requiring specialized kernel support.
• Understanding of SGLang architecture: The server uses tensor parallelism across 8 GPUs, with separate backends for attention, NSA (speculative decoding), and MoE routing.
• Familiarity with Blackwell GPU architecture (SM120): The RTX PRO 6000 Blackwell GPUs have specific kernel requirements that differ from previous generations.
• Knowledge of P2P/PCIe topology: The LXC container provides bare-metal GPU access, but the underlying PCIe topology (each GPU on its own root complex) may still limit cross-GPU communication.
• Understanding of inference benchmarking: The distinction between total token throughput and output token throughput, and the significance of concurrency scaling.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

1. Throughput baseline for GLM-5-NVFP4 on 8 Blackwell GPUs: 806 total tok/s at 128 concurrency, with 453 peak output tok/s.
2. Scaling characteristics: Throughput plateaus between 64 and 128 concurrency, suggesting a bottleneck unrelated to request volume.
3. DeepGemm scale format mismatch: The checkpoint uses a different scale format than what DeepGemm expects, which may affect accuracy on Blackwell.
4. No MoE kernel config warnings: The server log doesn't contain warnings about missing or suboptimal MoE configurations, suggesting the default configs are being applied without error (though they may still be suboptimal).

The Thinking Process: A Window into Diagnostic Reasoning

The assistant's reasoning in [msg 647] is a textbook example of iterative hypothesis testing. The sequence is:

1. Observe anomaly: Throughput at 64 concurrency (757 tok/s) seems lower than expected compared to the KVM VM baseline.
2. Self-correct: Realize the comparison may be apples-to-oranges due to different measurement methodologies.
3. Form hypotheses: The bottleneck could be MoE kernel configuration, or it could be a concurrency ceiling.
4. Test in parallel: Run a higher-concurrency benchmark while simultaneously inspecting server logs for configuration warnings.
5. Evaluate results: The 128-concurrency test shows diminishing returns, and the log reveals a DeepGemm scale format warning but no MoE config issues. The assistant doesn't jump to conclusions. It collects data and lets the numbers speak. The plateau at 806 tok/s is not explained in this message — it becomes the puzzle to solve in subsequent messages. But the groundwork for that investigation is laid here: the assistant has identified that something is wrong, narrowed down the potential causes, and set the stage for deeper analysis.

Conclusion

Message [msg 647] is a turning point in the opencode session. It marks the moment when the infrastructure work is complete and the real performance optimization begins. The assistant has successfully deployed GLM-5-NVFP4 on 8 Blackwell GPUs with P2P access, but the throughput numbers — 806 total tok/s at 128 concurrency — are underwhelming given the hardware investment.

The message is notable for its disciplined diagnostic approach: parallel investigation of system logs and benchmark results, careful attention to measurement methodology, and a willingness to question assumptions. The DeepGemm scale format warning is a breadcrumb that will lead to deeper investigation in subsequent messages, and the throughput plateau frames the central question of the session: what is limiting inference performance on Blackwell, and how can it be overcome?

For anyone working with large-scale LLM inference on cutting-edge hardware, this message offers a valuable lesson: even after solving the hard infrastructure problems (driver compatibility, CUDA initialization, P2P topology), the performance optimization phase can be equally challenging. The numbers don't lie, but they don't always tell the whole story — and finding the missing pieces requires systematic investigation, domain knowledge, and a willingness to challenge your own assumptions.