The Moment of Truth: Benchmarking GLM-5-NVFP4 After the LXC Breakthrough

Introduction

In the long arc of a complex ML infrastructure debugging session, few moments carry as much weight as the first real benchmark after a major architectural change. Message [msg 646] captures exactly such a moment. After an arduous journey spanning multiple segments—from KVM virtual machine setup with VFIO passthrough, through NaN decode crashes, PCIe topology investigations, Proxmox kernel modifications, and finally an LXC container workaround—the assistant finally has concrete numbers to show for it. This message reports the first benchmark results from the newly established LXC-based deployment of the GLM-5-NVFP4 model on 8× RTX PRO 6000 Blackwell GPUs, and it immediately acts on those results by launching a higher-concurrency test.

The message is deceptively brief. It contains only a few lines of commentary and a single bash command invocation, but it represents the culmination of dozens of prior messages and hours of debugging. Understanding this message requires understanding the entire context that led to it: the P2P bottleneck in the KVM VM, the failed attempts to enable direct GPU communication across PCIe root complexes, the pivot to LXC containers, and the CUDA initialization blocker that nearly derailed everything. This article unpacks what this message means, why it was written, what decisions it reflects, and what knowledge it produces.

The Context: Why This Message Exists

To understand message [msg 646], one must first understand the problem it was designed to address. Throughout segments 3 and 4 of the session, the assistant and user had been battling a fundamental hardware limitation: the eight RTX PRO 6000 Blackwell GPUs were each on their own PCIe root complex, connected through a PCIe switch rather than a proper PCIe fabric. In the KVM virtual machine environment using VFIO passthrough, this topology meant that Peer-to-Peer (P2P) DMA transfers between GPUs were routed through system memory rather than directly over the PCIe bus. The consequence was severe: inter-GPU communication latency was high, and the full potential of tensor parallelism across eight GPUs could not be realized.

The assistant attempted multiple fixes: modifying Proxmox host kernel parameters, migrating the VM from i440FX to Q35 chipset, fixing BAR allocation, disabling ACS, and more. Ultimately, the hardware topology itself was the fundamental blocker—each GPU was isolated in its own IOMMU group, and no amount of software configuration could merge them. The breakthrough came in segment 5: instead of trying to fix the VM, the assistant pivoted to an LXC container running directly on the Proxmox host, bypassing VFIO entirely. This gave the container bare-metal access to the GPUs, with proper NODE/SYS topology and 53 GB/s P2P bandwidth between GPUs on the same NUMA node.

But even this approach hit a wall. CUDA initialization failed inside the LXC container because the NVIDIA open kernel module's Heterogeneous Memory Management (HMM) feature was incompatible with the Proxmox VE kernel. The fix was a single kernel module parameter: uvm_disable_hmm=1 for nvidia_uvm. Once applied, CUDA initialized successfully, and the assistant could finally proceed to deploy the SGLang inference server.

The messages immediately preceding [msg 646] (specifically [msg 640] through [msg 645]) show the server being launched, verified, and then benchmarked. The assistant had to resolve a missing ninja-build package for FlashInfer JIT compilation, upgrade transformers to 5.2.0 for glm_moe_dsa support, and debug a server crash. By [msg 645], the server was running and the assistant was configuring the benchmark parameters, learning that --request-rate -1 didn't work with this version and that the tokenizer needed to be explicitly specified via --tokenizer lukealonso/GLM-5-NVFP4 rather than relying on the served model name.

What the Message Actually Says

The message opens with a summary of the first benchmark results:

First benchmark results from the LXC container: - Total throughput: 438.67 tok/s at 32 concurrency - Output throughput: 130.54 tok/s (peak 217 tok/s) - TPOT: ~115ms (mean) This is comparable to the KVM VM baseline (~485 tok/s peak at 64 concurrency).

The assistant then immediately acts on this information:

Let me now try higher concurrency to see if we get the expected improvement from the better GPU topology:

And launches a benchmark with 64 prompts (double the previous test) using --request-rate inf, which means all requests are sent simultaneously without rate limiting. The truncated output shows:

Total token throughput (tok/s): 757.54

This is a significant improvement—from 438 tok/s at 32 concurrency to 757 tok/s at 64 concurrency, approaching double the throughput. The peak output token throughput reached 417 tok/s, and request throughput was 3.76 req/s.

The Reasoning and Decision-Making Process

The message reveals several layers of reasoning. First, the assistant is performing a comparative analysis: the LXC container at 32 concurrency yields 438 tok/s, while the KVM VM baseline peaked at ~485 tok/s at 64 concurrency. This comparison is crucial because it answers the question "was the LXC pivot worth it?" On the surface, the numbers look similar—438 vs 485—but the concurrency levels differ. The LXC container achieved 438 tok/s with only 32 concurrent requests, whereas the KVM VM needed 64 concurrent requests to reach 485 tok/s. This suggests that per-request performance is better in the LXC container, even if total throughput at low concurrency is similar.

The assistant's decision to immediately test higher concurrency (64 prompts) reflects a hypothesis: if the better GPU topology (P2P access) provides any benefit, it should manifest as improved scaling under higher concurrency, where inter-GPU communication becomes the bottleneck. The KVM VM's peak of 485 tok/s at 64 concurrency suggested that it was hitting a wall—adding more concurrency beyond 64 didn't improve throughput because P2P transfers were bottlenecked. If the LXC container's P2P is truly better, it should scale further.

The results confirm this hypothesis partially: 757 tok/s at 64 concurrency is substantially higher than the KVM VM's ~485 tok/s at the same concurrency level. This is a ~56% improvement, directly attributable to the better GPU topology in the LXC container.

Assumptions Made

The message operates on several assumptions, some explicit and some implicit:

  1. The KVM VM baseline is a valid comparison point. The assistant assumes that the earlier benchmark results (~485 tok/s peak) are directly comparable to the new LXC results. This is reasonable since the same model, same quantization, and similar server configuration were used, but there could be confounding factors (different SGLang versions, different attention backends, different kernel versions).
  2. Higher concurrency will reveal the P2P improvement. The assistant assumes that the benefit of better P2P topology manifests primarily under high concurrency, where inter-GPU communication is the bottleneck. This is a sound assumption for tensor-parallel inference, where each forward pass requires all-reduce operations across GPUs.
  3. The benchmark tool is producing reliable numbers. The assistant trusts that sglang.bench_serving with --dataset-name random and the specified parameters produces meaningful throughput measurements. The random dataset with 256 input tokens and 128 output tokens is a reasonable synthetic workload, but it may not reflect real-world usage patterns.
  4. The server configuration is optimal. The assistant is using the same server flags as before (flashinfer attention backend, trtllm NSA backends, flashinfer_cutlass MoE runner). There's an implicit assumption that these settings are appropriate for the LXC environment, though they were originally chosen for the KVM VM.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Knowledge of the P2P bottleneck saga. Without knowing that the KVM VM had severely limited P2P bandwidth due to PCIe topology and VFIO isolation, the comparison to "KVM VM baseline" is meaningless. The entire point of the LXC pivot was to improve inter-GPU communication.
  2. Understanding of tensor parallelism. The model is deployed with --tp 8, meaning it's split across all 8 GPUs. Every inference step requires communication between GPUs. The throughput numbers directly reflect how well that communication works.
  3. Familiarity with SGLang benchmark metrics. Terms like "total throughput," "output throughput," "TPOT" (time per output token), and "concurrency" are specific to LLM serving benchmarks. TPOT of ~115ms means each token takes about 115ms to generate, which translates to roughly 8.7 tokens per second per request (for a single sequence).
  4. Context about the GLM-5-NVFP4 model. This is a Mixture-of-Experts (MoE) model with FP4 quantization, which places unique demands on GPU memory and compute. The model has 83 safetensors shards and requires significant GPU memory (~63 GB per GPU just for weights).
  5. The hardware topology. Eight RTX PRO 6000 Blackwell GPUs, each on its own PCIe root complex, connected via a PCIe switch, running inside an LXC container on Proxmox VE.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. The LXC approach works and improves throughput. The 757 tok/s at 64 concurrency is a clear improvement over the KVM VM's ~485 tok/s. This validates the decision to pivot from KVM to LXC.
  2. Throughput scales with concurrency. From 438 tok/s at 32 concurrency to 757 tok/s at 64 concurrency, the system shows good scaling. The peak output token throughput of 417 tok/s at 64 concurrency (versus 217 tok/s at 32 concurrency) nearly doubles, suggesting the system isn't yet saturated.
  3. The P2P improvement is real but not transformative at this stage. While 757 tok/s is better than 485 tok/s, it's not the order-of-magnitude improvement one might hope for from fixing a fundamental bottleneck. This suggests that other factors (MoE kernel efficiency, Blackwell architecture tuning, memory bandwidth) are also significant constraints.
  4. Single-stream performance remains a concern. Although not stated in this message, the chunk summary reveals that single-stream performance remained around 11 tok/s (500 tokens in 45 seconds), similar to the KVM VM. This means the P2P improvement primarily benefits batch throughput, not individual request latency.

The Thinking Process Visible in the Message

The assistant's thinking is revealed through the structure of the message. First, it reports the baseline result. Then, it immediately contextualizes that result against the previous KVM VM performance. This comparison is the key analytical move—it tells the reader (and the user) whether the LXC effort was worthwhile.

The phrase "Let me now try higher concurrency to see if we get the expected improvement from the better GPU topology" reveals the assistant's hypothesis-driven approach. It has a theory (better P2P → better scaling under high concurrency) and is testing it. The "expected improvement" language suggests the assistant is optimistic but cautious—it's not declaring victory yet, but it sees promising signs.

The truncated output at the end of the message is also telling. The assistant captured only the tail of the benchmark output, showing the key metrics but not the full detail. This is a practical decision: the full output would be verbose, and the key numbers (total throughput 757.54 tok/s, peak output 417 tok/s) are what matter for the immediate comparison.

Mistakes and Limitations

Several limitations are worth noting:

  1. The comparison is not perfectly controlled. The KVM VM baseline was run with different SGLang version and potentially different server parameters. The assistant acknowledges this implicitly by saying "comparable to" rather than making a definitive claim.
  2. The benchmark uses synthetic data. Random input/output lengths of 256/128 tokens don't reflect real conversational patterns. Real workloads might show different scaling behavior.
  3. Only two data points are presented. The assistant has results at 32 and 64 concurrency. Without testing at 128, 256, or higher concurrency, it's unclear where the saturation point lies. (The chunk summary reveals that 128 concurrency was later tested and achieved 806 tok/s, confirming continued scaling.)
  4. No latency breakdown is provided. The message reports TPOT (~115ms) but doesn't break down time-to-first-token (TTFT), prefill time, or decode time. These metrics would help identify where the bottlenecks actually are.

Conclusion

Message [msg 646] is a milestone in a long debugging journey. It represents the first concrete evidence that the LXC container approach—adopted after exhausting all options for fixing P2P in the KVM VM—actually works and delivers measurable improvement. The 757 tok/s at 64 concurrency, compared to the KVM VM's ~485 tok/s, validates the pivot and provides a foundation for further optimization.

But the message is also a beginning, not an end. The assistant's immediate action—launching a higher-concurrency test—shows that the work is not done. The throughput numbers are promising but not yet at the target of 1k+ tok/s. The single-stream performance of ~11 tok/s remains a concern. And as the chunk summary notes, further tuning of MoE kernels for Blackwell (SM120) architecture is needed, including the already-merged PR #14311 and custom kernel configurations.

This message captures a moment of cautious optimism: the infrastructure is finally working, the numbers are heading in the right direction, but the real optimization work is just beginning.