The State of the Union: A Critical Inflection Point in Blackwell Inference Optimization

Introduction

In the complex world of large language model deployment, there comes a moment in every optimization journey where the low-hanging fruit has been picked, the obvious bottlenecks have been addressed, and the team must confront a sobering reality: the problem is deeper than expected. Message 650 in this opencode session represents precisely such a moment. It is a comprehensive status document—a "state of the union"—written by the AI assistant after successfully deploying the massive GLM-5-NVFP4 model (744B parameters, 40B active) across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, only to discover that a major infrastructure migration from KVM virtualization to LXC containers had yielded disappointingly similar performance to the original setup.

This message is not merely a progress report. It is a diagnostic document, a planning artifact, and a knowledge consolidation exercise all rolled into one. It marks the transition from the "getting it working" phase to the "making it fast" phase—and the recognition that the latter will require fundamentally different approaches than anticipated. At over 2,000 words of dense technical content, it represents the assistant's complete mental model of the deployment at a critical inflection point.

The Context: A Long Journey to This Point

To understand why message 650 was written, we must understand the journey that preceded it. The conversation leading up to this message spans multiple segments covering days of work. The team had been attempting to deploy the GLM-5-NVFP4 model—a 744B-parameter Mixture-of-Experts (MoE) model with 256 experts and DeepSeek Sparse Attention—on a Proxmox-hosted machine with 8 NVIDIA RTX PRO 6000 Blackwell GPUs.

The initial deployment was on a KVM virtual machine (VM 128, "llm-one"). This setup achieved a peak throughput of approximately 485 tokens per second at 64 concurrent requests, with a painfully slow single-stream performance of about 11 tokens per second. The root cause was identified as GPU-to-GPU communication latency: the VFIO/IOMMU virtualization layer prevented Peer-to-Peer (P2P) Direct Memory Access between GPUs that were in separate IOMMU groups, imposing a floor of approximately 13 microseconds on inter-GPU latency.

The team made a strategic decision to migrate from the KVM VM to an LXC (Linux Container) container (CT 129, "llm-two") on the same Proxmox host. The theory was that LXC, being a lighter-weight virtualization technology that shares the host kernel, would eliminate the VFIO overhead and provide bare-metal GPU topology, enabling true P2P DMA between GPUs. This migration was a significant undertaking: it required installing NVIDIA drivers on the Proxmox host itself (a non-trivial operation), fixing a CUDA initialization bug caused by the NVIDIA open kernel module's HMM feature, configuring the LXC container with GPU device mounts, and setting up the entire software stack from scratch.

The migration succeeded. GPU topology was confirmed as NODE/SYS (bare-metal) rather than PHB (virtualized). P2P bandwidth was measured at 53.76 GB/s within the same NUMA node and 40.24 GB/s across NUMA nodes—a dramatic improvement over the virtualized setup. The server was launched, and initial benchmarks were run.

And then came the disappointing results.

What the Message Actually Says

Message 650 opens with a clear statement of purpose: "Deploy GLM-5-NVFP4... on a remote machine with 8x NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs using sglang for inference serving, optimized for maximum throughput." The user's targets are stated: "1k+ total tok/s, >100 tok/s single/dual stream."

The message then presents benchmark results that tell a sobering story:

| Concurrency | Total tok/s | Output tok/s | Peak Output | TPOT (ms) | |-------------|-------------|--------------|-------------|-----------| | 32 | 438.67 | 130.54 | 217 | 114.58 | | 64 | 757.54 | 219.15 | 417 | 131.18 | | 128 | 806.04 | 272.61 | 453 | 171.79 |

And the single-stream result: "500 tokens in 45.5s ≈ ~11 tok/s (same as KVM — no improvement)."

The message's own analysis is telling: "The throughput is disappointingly similar to the KVM VM baseline (~485 tok/s). Despite better P2P bandwidth, performance isn't scaling."

This is the central tension of the entire document. The team invested significant effort in migrating to LXC, fixed a critical CUDA initialization bug, confirmed bare-metal GPU topology, and achieved dramatically better P2P bandwidth—yet the throughput is essentially unchanged. The message is the assistant's attempt to make sense of this paradox and chart a path forward.

The Structure of the Message: A Deliberate Knowledge Artifact

Message 650 is meticulously structured. It is not a stream-of-consciousness reflection but a carefully organized document with clear sections. Let us examine its architecture:

  1. Goal: A one-paragraph restatement of the mission, including the user's performance targets and a brief summary of the previous approach's limitations.
  2. Instructions: Practical access information for the three machines involved (Proxmox host, LXC container, KVM VM), plus conventions for building CUDA extensions, package management, and critical references to prior research.
  3. Discoveries: A comprehensive section covering hardware specifications, bare-metal GPU topology, P2P performance measurements, the critical CUDA initialization fix, SM120 shared memory constraints, and model details.
  4. Current Benchmark Results: The disappointing numbers that motivated the message.
  5. Server Warnings: Notable log messages that may indicate issues.
  6. Working Launch Command: The exact command used to start the server, preserved for reproducibility.
  7. Key Prior Research: References to the local research repository containing findings from prior Kimi K2 deployment work on the same hardware.
  8. Status Tracking: A tri-state breakdown of Completed, In Progress, and Not Yet Done items.
  9. Relevant Files/Directories: A comprehensive file inventory across all machines. This structure reveals the message's dual purpose. First, it serves as a knowledge consolidation document—capturing everything learned so far in a single reference that can be consulted by both the user and the assistant in future work. Second, it serves as a decision support document—laying out the evidence that the current approach is insufficient and identifying the most promising paths forward.

The Reasoning Behind the Message

Why was this message written at this precise moment? Several factors converged:

1. The Need to Recalibrate Strategy

The most obvious reason is that the expected performance improvement from the LXC migration did not materialize. The assistant needed to acknowledge this failure of prediction, analyze why it happened, and propose a new direction. The message's "Analysis" bullet points list three possible reasons:

2. The Need to Document Critical Fixes

The CUDA initialization fix (uvm_disable_hmm=1) was a critical breakthrough that took significant debugging to discover. Documenting it properly ensures that:

3. The Need to Establish a Shared Mental Model

The message contains an enormous amount of detailed information—GPU bus IDs, NUMA topology, P2P bandwidth measurements, model architecture details, software versions, file paths. This serves to align the assistant and user on a shared understanding of the current state. In a complex deployment with many interacting components, such alignment is essential for effective collaboration.

4. The Need to Prioritize Next Steps

The "Not Yet Done" section lists seven items, but the first is clearly identified as the most important: "Tune MoE kernels for GLM-5-NVFP4 on SM120 — This is the most likely path to significant improvement." The message provides a specific action: run tuning_fused_moe_triton.py from the sglang benchmark suite. This transforms the document from a passive status report into an active planning tool.

Assumptions Made and Their Consequences

The message reveals several assumptions that shaped the work, some of which proved incorrect.

Assumption 1: P2P Bandwidth Was the Primary Bottleneck

The entire migration from KVM to LXC was predicated on the assumption that VFIO/IOMMU-induced P2P latency was the main performance limiter. The message's own history states: "The bottleneck was GPU-to-GPU communication latency (~13µs floor) caused by VFIO/IOMMU preventing P2P DMA between GPUs in separate IOMMU groups."

This assumption was reasonable—P2P bandwidth is critical for tensor-parallel inference where GPUs must communicate activations and gradients at every layer. However, the benchmark results prove that eliminating this bottleneck had minimal impact on overall throughput. The assistant's own analysis in the message acknowledges this implicitly by listing other potential bottlenecks.

Assumption 2: LXC Would Provide Near-Native Performance

LXC containers share the host kernel and avoid the overhead of a full hypervisor, so the expectation was that GPU performance would be essentially identical to bare metal. While this assumption was technically correct (the GPU topology was confirmed as NODE/SYS rather than PHB), it failed to account for the fact that other bottlenecks—particularly MoE kernel efficiency—were more significant than inter-GPU communication.

Assumption 3: Existing MoE Configs Would Be Transferable

The local research repository contained tuned MoE configurations from prior work deploying Kimi K2-Thinking NVFP4 on the same hardware. The message notes that these configs were for GLM-4.7 with E=161 experts, while GLM-5 has E=256 experts. The assumption that the general approach would transfer was correct, but the specific configurations needed regeneration.

Assumption 4: The SM120 Shared Memory Fix Would Be Sufficient

PR #14311, which fixes extend_attention.py block sizes for SM120's smaller shared memory (100KB vs 228KB on SM100), was identified as a "must-have fix" and was already merged into the sglang source. The message assumes this fix addresses the most critical SM120 compatibility issue, but the benchmark results suggest that MoE kernel tuning is an equally important concern.

Mistakes and Incorrect Assumptions

Beyond the general assumptions, the message reveals several specific mistakes or miscalculations.

The Performance Plateau Mystery

The benchmark results show a clear plateau: throughput goes from 438 tok/s at 32 concurrency to 757 tok/s at 64 concurrency, but only increases to 806 tok/s at 128 concurrency. This plateau suggests that some resource is saturated—likely GPU compute capacity or memory bandwidth—but the message doesn't definitively identify which. The assistant's analysis focuses on MoE kernels and allreduce, but the plateau could also be caused by the CPU-side scheduler being unable to feed requests fast enough, or by the KV cache management becoming a bottleneck at higher concurrency.

The Single-Stream Disappointment

The single-stream performance of ~11 tok/s is particularly striking because it's identical to the KVM baseline. Single-stream throughput is primarily determined by the latency of a single forward pass through the model, which depends on GPU compute speed and memory bandwidth rather than inter-GPU communication. The fact that this didn't improve at all suggests that the GPU compute kernels themselves are the bottleneck—not P2P communication. This is a strong signal that MoE kernel optimization is indeed the right path forward.

The Missing Diagnostic Data

The message doesn't include GPU utilization metrics during the benchmark runs. Were the GPUs fully utilized? Were they memory-bandwidth-bound or compute-bound? What was the power draw? The message mentions server warnings but doesn't include profiling data that would pinpoint the bottleneck. This omission is notable because the assistant had access to tools like nvidia-smi and could have collected this data.

The NCCL Configuration

The launch command includes NCCL_IB_DISABLE=1 (disabling InfiniBand, which doesn't exist on these GPUs) and NCCL_P2P_LEVEL=5 (setting P2P to NVLink/NVSwitch level). The NCCL_MIN_NCHANNELS=8 setting requests 8 communication channels. These NCCL settings were carried over from the KVM configuration and may not be optimal for the LXC environment. The message doesn't question whether these settings are appropriate for the new topology.

Input Knowledge Required

To fully understand message 650, the reader needs substantial domain knowledge spanning multiple areas:

GPU Architecture and Topology

Model Architecture

Inference Serving

Virtualization and Systems

Benchmarking Methodology

Output Knowledge Created

Message 650 creates substantial output knowledge that serves as a foundation for future work:

Documented Infrastructure State

The message provides a complete inventory of the deployment: machine addresses, software versions, configuration files, model cache locations, and running processes. This documentation ensures that anyone returning to this work can quickly understand the current state and resume optimization.

Validated (and Invalidated) Hypotheses

The message explicitly validates that LXC provides bare-metal GPU topology with P2P access, and that the CUDA initialization fix works. It implicitly invalidates the hypothesis that P2P bandwidth was the primary bottleneck—a significant finding that reshapes the optimization strategy.

Prioritized Optimization Path

The message clearly identifies MoE kernel tuning as the most promising next step, with specific instructions for how to proceed. This prioritization is based on evidence (the performance plateau, the single-stream stagnation) and prior research (the Kimi K2 findings showing 5,816 tok/s peak after tuning).

Reproducible Configuration

The working launch command and benchmark command are preserved verbatim, enabling reproducible testing. This is critical for measuring the impact of future optimizations.

Knowledge of Limitations

The message documents what is NOT working: single-stream throughput is stuck at ~11 tok/s, total throughput plateaus around 800 tok/s, and the expected LXC benefit didn't materialize. This negative knowledge is valuable because it prevents wasted effort on dead-end optimization paths.

The Thinking Process Visible in the Message

While message 650 is presented as a factual summary, the assistant's reasoning process is visible in several places.

The Diagnostic Chain

The assistant's analysis of the disappointing benchmark results reveals a diagnostic thought process:

  1. Observation: Throughput is similar to KVM despite better P2P
  2. Hypothesis generation: Three possible reasons are listed
  3. Evidence gathering: Server warnings are checked (DeepGemm scale format warning, transformers version)
  4. Prior knowledge integration: The local research repo is referenced for MoE tuning instructions
  5. Prioritization: MoE kernel tuning is identified as "the most likely path to significant improvement" This is a textbook diagnostic reasoning chain: observe anomaly, generate hypotheses, gather evidence, integrate prior knowledge, prioritize next steps.

The Comparative Analysis

The message includes a performance comparison table that implicitly compares LXC results to KVM results (referenced in the "Previous approach" section). The assistant is clearly tracking the performance delta and evaluating whether the migration investment was worthwhile. The conclusion—that it wasn't, at least not yet—is stated plainly.

The Forward-Looking Reasoning

The "Not Yet Done" section is not just a to-do list; it's a prioritized research agenda. The items are ordered by likely impact:

  1. MoE kernel tuning (highest impact)
  2. TP4+PP2 configuration (alternative parallelism strategy)
  3. Higher concurrency testing (exploring the plateau)
  4. CUDA graph disabling (eliminating a potential overhead)
  5. NCCL tuning (communication optimization)
  6. Profiling (diagnostic)
  7. Kernel upgrade (system-level fix) This ordering reflects a sophisticated understanding of where the bottlenecks likely lie. The assistant is reasoning that compute kernel efficiency (MoE) is more likely to be the bottleneck than communication (NCCL, allreduce) or system-level issues (kernel version).

The Broader Implications

Message 650 illuminates several important truths about large-scale ML inference deployment:

The Complexity of Performance Optimization

Performance optimization in distributed ML inference is a multi-dimensional problem. Eliminating one bottleneck (P2P communication) does not guarantee overall improvement if other bottlenecks (MoE kernel efficiency) are more severe. The message is a case study in the law of diminishing returns and the importance of systematic bottleneck identification.

The Value of Negative Results

The message's most valuable content may be its negative results: the LXC migration didn't help as expected, single-stream performance is stuck, and the throughput plateau is unexplained. In many engineering contexts, negative results go undocumented, leading to repeated dead ends. The assistant's thorough documentation of these findings is a best practice worth emulating.

The Importance of Domain-Specific Tuning

The message reveals that generic GPU kernels are not optimal for the SM120 architecture. The SM120's smaller shared memory (100KB vs 228KB) requires kernel block size adjustments, and the MoE kernels need hardware-specific configuration. This highlights the growing importance of hardware-specific kernel tuning as GPU architectures diversify.

The Role of Prior Knowledge

The local research repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ is referenced multiple times. This prior work on deploying Kimi K2-Thinking NVFP4 on the same hardware provides crucial guidance: peak throughput of 5,816 tok/s was achieved, MoE kernel tuning was identified as critical, and TP4+PP2 was found to outperform TP8 for short-context workloads. This prior knowledge shapes the assistant's optimization strategy and prevents reinventing the wheel.

Conclusion

Message 650 is far more than a status update. It is a carefully crafted knowledge artifact that serves multiple functions simultaneously: a diagnostic document analyzing why expected improvements didn't materialize, a planning document prioritizing the next optimization steps, a reference document capturing the complete infrastructure state, and a decision support document helping the user understand the tradeoffs involved.

The message's greatest strength is its honesty. It does not gloss over the disappointing benchmark results or make excuses for them. It presents the data plainly, analyzes it systematically, and proposes a concrete path forward. In doing so, it embodies the scientific method applied to engineering: form hypotheses, test them, analyze the results, and iterate.

The message also reveals the assistant's sophisticated mental model of the system. The assistant understands GPU topology, model architecture, inference serving, and system administration well enough to integrate information across these domains and draw meaningful conclusions. The identification of MoE kernel tuning as the most promising path forward, despite the availability of many other optimization levers, demonstrates a nuanced understanding of where the true bottlenecks lie.

For anyone working on large-scale ML inference deployment, message 650 offers a masterclass in how to structure a complex optimization effort: document everything, test hypotheses systematically, acknowledge failures honestly, and let evidence guide the next steps. The message marks the end of one phase of the work (infrastructure migration and basic deployment) and the beginning of another (deep performance optimization). Whether the team will achieve the target of 1,000+ tokens per second remains to be seen, but with this document as their guide, they have a clear map of the territory ahead.