The Plateau That Tells a Story: Analyzing Inference Throughput on Blackwell GPUs

The Message in Context

In a sprawling, multi-session effort to deploy the GLM-5-NVFP4 large language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a single message from the AI assistant marks a critical inflection point. The message, indexed as <msg id=648>, arrives after the team has overcome a staggering sequence of obstacles: NVIDIA driver incompatibilities, CUDA initialization failures caused by Heterogeneous Memory Management (HMM) conflicts with the Proxmox kernel, flash-attn build issues requiring dual CUDA toolkits, and a migration from a KVM virtual machine to an LXC container to unlock true peer-to-peer (P2P) GPU communication. After all that effort—the kernel parameter tweaks, the package rebuilds, the topology verification showing 53 GB/s same-NUMA bandwidth—the assistant finally has a running inference server and can measure what matters: throughput.

Here is the message in full:

Results so far: - 32 concurrent: 438 total tok/s - 64 concurrent: 757 total tok/s - 128 concurrent: 806 total tok/s (peak output: 453 tok/s)

>

The throughput scaling is plateauing early. This is concerning. I noticed from the server log: 1. "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." — This is a warning worth noting 2. No MoE kernel config warnings — the default configs are being used

>

The throughput is actually quite similar to the KVM baseline. Let me check if we're using the MoE configs properly and also test single-stream performance: [bash] ssh root@10.1.230.174 "time curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{\"model\": \"glm-5\", \"messages\": [{\"role\": \"user\", \"content\": \"Explain quantum entanglement in detail.\"}], \"max_tokens\": 500, \"temperature\": 0.1}' 2>&1 | python3 -c ' import json, sys d = json.load(sys.stdin) u = d[\"usage\"] print(f\"Prompt tokens: {u[\"prompt_tokens\"]}\") print(f\"Completion tokens: {u[\"completion_tokens\"]}\") print(f\"Total tokens: {u[\"total_tokens\"]}\") c = d[\"choices\"][0] content = c[\"message\"].get(\"content\", \"\") or \"\" reasoning = c[\"message\"].get(\"reasoning_content\", \"\") or \"\" print(f\"Content length: {len(content)}\") print(f\"Reasoning length: {len(reasoning)}\") '" Prompt tokens: 14 Completion tokens: 500 Total tokens: 514 Content length: 0 Reasoning length: 2065

>

real 0m45.480s user 0m0.016s sys 0m0.006s

On its surface, this is a routine status update: three benchmark numbers, a server log warning, and a single-stream latency test. But beneath the surface, this message is a moment of reckoning. It represents the collision between the team's expectations—that bare-metal GPU topology with P2P access would dramatically improve throughput—and the reality that the numbers are stubbornly similar to the KVM baseline they left behind.

Why This Message Was Written: The Reasoning and Motivation

The assistant's primary motivation in <msg id=648> is diagnostic. After investing enormous effort to move from a VFIO-constrained KVM virtual machine to an LXC container with true P2P GPU access, the benchmark results are underwhelming. The throughput scaling curve tells a worrying story: going from 32 to 64 concurrency nearly doubles throughput (438 → 757 tok/s), but going from 64 to 128 yields only a marginal 6% improvement (757 → 806 tok/s). The curve is saturating, and the absolute numbers are comparable to what the KVM VM achieved earlier in the session.

The assistant's reasoning is visible in the structure of the message. It opens with the raw data, then immediately interprets it: "The throughput scaling is plateauing early. This is concerning." This is not a neutral report—it is an expression of concern that the fundamental assumption driving the LXC migration may be wrong, or at least insufficient. The team had believed that P2P GPU communication was the primary bottleneck, and that removing the VFIO/IOMMU virtualization layer would unlock significantly higher throughput. The data now suggests otherwise.

The assistant then pivots to two potential explanations, both drawn from the server log. First, the DeepGemm scale format warning: the checkpoint uses a scale format other than ue8m0, which the server warns "might cause accuracy degradation on Blackwell." While this is an accuracy warning rather than a performance warning, it signals that the quantization configuration may not be optimal for the Blackwell architecture. Second, the absence of MoE kernel config warnings is itself noteworthy—it means the server is falling back to default kernel configurations rather than using Blackwell-optimized settings. The user had previously indicated that PR #14311 (already merged into the sglang main branch) and custom kernel tuning are essential for good performance on Blackwell, yet the server log shows no evidence that these optimizations are being applied.

The final action—the single-stream latency test—is a deliberate diagnostic move. By measuring how long a single request takes to generate 500 tokens (45.48 seconds, yielding roughly 11 tok/s), the assistant establishes a baseline for latency-bound performance. This is important because it reveals that the model is not memory-bandwidth-bound in the single-stream case; rather, it is compute-bound or kernel-launch-bound. The 45-second generation time for 500 tokens is extremely slow for eight top-tier GPUs, suggesting that the MoE kernel implementations being used are far from optimal for the Blackwell architecture.

How Decisions Were Made in This Message

The message contains two implicit decisions, both of which shape the subsequent direction of the project.

The first decision is to accept the benchmark data as valid and comparable. The assistant does not question whether the benchmark methodology is fair—it directly compares the LXC results to the earlier KVM results and finds them similar. This is a significant methodological choice. The assistant could have argued that the benchmarks are not directly comparable (different server configurations, different request distributions, different model loading states), but instead it treats the comparison as meaningful. This decision effectively reframes the problem: the bottleneck is not P2P topology after all, or at least not solely P2P topology.

The second decision is to prioritize kernel-level optimization over further topology investigation. The assistant could have responded to the plateauing throughput by probing deeper into GPU communication patterns—checking NVLink bandwidth utilization, measuring all-reduce latency, or verifying that P2P is actually being used during inference. Instead, it focuses on the server log warnings and the MoE kernel configuration. This is a strategic pivot: having confirmed that the LXC container provides correct bare-metal topology, the assistant now looks upward in the software stack rather than downward in the hardware stack.

Assumptions Made by the Assistant

Several assumptions underpin the reasoning in this message, and some of them are worth examining critically.

Assumption 1: Throughput should scale linearly with concurrency. The assistant expresses concern that scaling from 64 to 128 concurrency yields only 6% improvement. This assumes that the server should be able to handle 128 concurrent requests without significant degradation. In reality, inference servers have finite request-processing capacity, and the plateau could simply reflect the server's maximum sustainable throughput given the current kernel configuration. The assistant implicitly assumes that the GPUs are underutilized at 64 concurrency and should be able to absorb more work.

Assumption 2: The KVM baseline is a fair reference point. The assistant states that throughput is "actually quite similar to the KVM baseline." This assumes that the earlier benchmarks were run under comparable conditions—same model, same quantization, same server version, same request characteristics. In practice, the earlier benchmarks may have used different server parameters, different attention backends, or different concurrency patterns. The comparison is useful but not rigorous.

Assumption 3: The DeepGemm warning is worth noting but not critical. The assistant flags the scale format warning but does not treat it as an immediate blocker. The assumption is that this is an accuracy concern rather than a performance concern. However, if the scale format mismatch causes the server to fall back to a suboptimal kernel path, it could directly impact throughput. The assistant may be underestimating the performance implications of this warning.

Assumption 4: Default MoE kernel configurations are suboptimal for Blackwell. The assistant notes that "no MoE kernel config warnings" means "the default configs are being used." The implicit assumption is that the defaults are not Blackwell-optimized and that custom tuning is needed. This is consistent with the user's earlier statements about PR #14311 and kernel tuning, but it remains an assumption until proven by benchmarking with tuned kernels.

Mistakes and Incorrect Assumptions

The most significant potential mistake in this message is a premature conclusion about the value of the LXC migration. The assistant writes that throughput is "actually quite similar to the KVM baseline," which could be read as suggesting that the LXC migration did not help. However, this conclusion may be premature for several reasons.

First, the benchmark configuration may not be stress-testing P2P bandwidth. The --random-input-len 256 --random-output-len 128 configuration generates relatively short sequences. For short sequences, the communication overhead (all-reduce for attention, all-to-all for MoE routing) is small relative to compute, so P2P bandwidth improvements would not be visible. The real benefit of P2P would appear with long sequences where large tensors must be communicated between GPUs. The assistant does not test this scenario.

Second, the server configuration may not be using P2P-optimized communication patterns. The assistant launched the server with NCCL_P2P_LEVEL=5, which enables P2P for NVLink-connected GPUs, but the actual communication patterns used during inference depend on the attention backend, the MoE runner backend, and the all-reduce fusion settings. If the server is falling back to collective communication (NCCL all-reduce) instead of using P2P transfers, the topology improvement would be invisible.

Third, the single-stream latency of 45 seconds for 500 tokens (approximately 11 tok/s) is a strong indicator that the MoE kernel implementations are the primary bottleneck, not inter-GPU communication. This is consistent with the absence of Blackwell-optimized kernel configs. The assistant correctly identifies this as a concern but does not yet connect it fully to the throughput plateau.

A secondary mistake is the failure to measure GPU utilization during the benchmark. The assistant earlier checked GPU utilization (0% when the process was stuck, <msg id=625>), but does not check utilization during the benchmark runs. GPU utilization metrics would immediately reveal whether the GPUs are compute-bound (high utilization) or communication-bound (low utilization with high latency). This is a missed diagnostic opportunity.

Input Knowledge Required to Understand This Message

To fully grasp the significance of <msg id=648>, the reader needs substantial context from the preceding session.

The LXC vs. KVM distinction. The entire project has been shaped by the discovery that the Proxmox KVM virtual machine imposes a VFIO/IOMMU layer that prevents P2P GPU communication. In <msg id=648>, the assistant references "the KVM baseline" as a point of comparison, assuming the reader knows that the LXC container was supposed to be the solution to this problem. Without this context, the disappointment in the plateauing throughput would seem inexplicable—after all, 806 tok/s is not bad.

The DeepGemm and MoE kernel context. The user had previously indicated that Blackwell (SM120 architecture) requires specific kernel tuning for the MoE (Mixture of Experts) layers, and that PR #14311 (merged into sglang) provides some of these optimizations. The assistant's concern about "default configs" reflects this knowledge. Without understanding that Blackwell is a new architecture with immature kernel support, the reader might not understand why the assistant is worried about the absence of kernel config warnings.

The server configuration flags. The assistant launched the server with specific backend flags: --attention-backend flashinfer, --nsa-decode-backend trtllm, --nsa-prefill-backend trtllm, --moe-runner-backend flashinfer_cutlass, --enable-flashinfer-allreduce-fusion. These choices represent a specific hypothesis about which kernel implementations work best on Blackwell. The fact that throughput is plateauing suggests this hypothesis may need revision.

The earlier struggles with CUDA and HMM. The team had to disable HMM in the nvidia_uvm module to get CUDA to initialize inside the LXC container. This is relevant because it means the CUDA stack is operating in a non-default configuration, which could have subtle performance implications.

Output Knowledge Created by This Message

This message creates several pieces of actionable knowledge that shape the subsequent direction of the project.

First, a quantified performance baseline for the LXC deployment. The three data points (438 tok/s at 32 concurrency, 757 tok/s at 64, 806 tok/s at 128) establish a reference point against which future optimizations can be measured. Any kernel tuning, configuration change, or software update can now be evaluated against this baseline.

Second, a specific diagnostic signal: the DeepGemm scale format warning. The assistant elevates this from a log message to a potential performance issue. This becomes a target for investigation: does the scale format mismatch cause a fallback to suboptimal kernels? Can the checkpoint be converted to ue8m0 format? Is there a server flag to override the scale format detection?

Third, a confirmed single-stream latency problem. The 45-second generation time for 500 tokens is a concrete metric that can be compared against expectations. For a model running on eight top-tier GPUs, 11 tok/s is extremely slow. This creates urgency around MoE kernel optimization.

Fourth, the insight that the throughput plateau is not caused by P2P topology. This is perhaps the most important output. By comparing LXC results to the KVM baseline and finding them similar, the assistant effectively rules out P2P bandwidth as the primary bottleneck. This redirects investigative effort toward kernel optimization, quantization configuration, and server parameter tuning.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in <msg id=648> follows a clear diagnostic pattern that reveals its underlying mental model.

Step 1: Observe the data. The assistant presents three benchmark results without commentary, allowing the reader (and itself) to absorb the pattern. The numbers speak for themselves: 438 → 757 → 806. The diminishing returns are visually obvious.

Step 2: Interpret the pattern. "The throughput scaling is plateauing early. This is concerning." The assistant explicitly labels this as a problem. The word "concerning" is emotionally loaded—it signals that this outcome violates expectations. The assistant expected better scaling from the LXC container.

Step 3: Cross-reference with server logs. The assistant checks two things: the DeepGemm warning and the MoE kernel config status. This reveals the assistant's causal model: it believes that kernel configuration is the most likely cause of the throughput plateau. The absence of MoE config warnings is treated as a negative signal—it means the server is not applying Blackwell-specific optimizations.

Step 4: Compare to previous baseline. "The throughput is actually quite similar to the KVM baseline." This is a crucial inference. The assistant is implicitly testing the hypothesis that "LXC + P2P = higher throughput" and finding it unsupported by the data. This is honest reasoning—the assistant does not try to explain away the disappointing results.

Step 5: Design a follow-up experiment. The assistant decides to test single-stream performance. This is a smart diagnostic choice because it isolates the model's raw generation speed from the server's request scheduling and batching efficiency. If single-stream is slow, the problem is in the model execution path, not in the server's ability to batch requests.

Step 6: Execute and interpret. The single-stream test reveals 500 tokens in 45.48 seconds. The assistant does not explicitly compute the tok/s (approximately 11), but the number speaks for itself. The reasoning content is 2065 characters for 500 tokens of output, indicating the model is generating extensive chain-of-thought reasoning before producing its final answer. This is relevant because reasoning tokens consume compute but are not counted in the output token throughput.

The assistant's thinking process is notable for its discipline. It does not jump to conclusions, does not blame the tooling, and does not make excuses. It presents the data, identifies the discrepancy, and designs a follow-up experiment. This is textbook diagnostic reasoning.

Conclusion

Message <msg id=648> is a pivotal moment in the GLM-5-NVFP4 deployment effort. It is the moment when the team's hypothesis about P2P topology being the primary bottleneck is tested and found insufficient. The assistant's response—measured, analytical, and focused on kernel-level optimization—sets the stage for the next phase of work: tuning MoE kernels for the Blackwell architecture, investigating the DeepGemm scale format issue, and optimizing server parameters for higher throughput.

The message also reveals something important about the assistant's character as an engineering partner. It does not celebrate the successful deployment (which was a genuine achievement after overcoming numerous blockers). Instead, it immediately focuses on the gap between expectation and reality. This relentless focus on measurable outcomes is what makes the assistant effective in this context: it treats deployment as a means to an end (performance), not as an end in itself.

The plateau at 806 tok/s is not a failure. It is a signal. And the assistant's ability to read that signal, interpret it, and act on it is what drives the project forward.