The HMM That Almost Stopped Blackwell: From CUDA Blocker to 806 tok/s on Eight RTX PRO 6000 GPUs

Introduction

The deployment of a 744-billion-parameter Mixture-of-Experts language model across eight NVIDIA RTX PRO 6000 Blackwell GPUs is a story of compounding obstacles, each layer of the software stack revealing a new failure mode that must be diagnosed and resolved before the next layer can be tested. This segment of the opencode session captures the most dramatic arc of that story: from a complete CUDA initialization failure that rendered all GPUs inaccessible, through a cascade of dependency and compatibility issues, to a working inference server serving 806 tokens per second at 128 concurrent requests.

The journey spans over 110 messages (indices 539 through 649) and encompasses several distinct phases: diagnosing and fixing the CUDA blocker, verifying the bare-metal GPU topology in an LXC container, resolving model type recognition failures, installing missing build tools, and finally benchmarking the deployed server. Each phase required the assistant to form hypotheses, test them against system behavior, and adapt its understanding of the problem. This article synthesizes the entire segment, tracing the thread that connects a kernel module parameter to a throughput plateau, and examining what it reveals about the state of Blackwell GPU inference in early 2026.

The Critical Blocker: CUDA Initialization Failure

The segment opens with the assistant at a standstill. Message 539 ([msg 539]) is a comprehensive status report documenting a frustrating paradox: the LXC container on the Proxmox VE host had confirmed true bare-metal GPU topology (NODE/SYS) with peer-to-peer (P2P) access potential — a major architectural win over the VFIO-limited KVM VM — but CUDA refused to initialize. The cuInit() call either hung indefinitely or returned error code 3 (CUDA_ERROR_NOT_INITIALIZED). This was a show-stopper: without CUDA compute capability, the GPUs were useless for inference.

The assistant's investigation was methodical. It checked driver module types (open vs. proprietary), confirmed that the open kernel module was required for Blackwell support, examined GSP firmware loading, compared kernel versions between the working KVM VM and the non-working host/container, and traced ioctl calls through strace. Each hypothesis was tested and discarded in turn. The host kernel was Proxmox VE's 6.8.12-9-pve, while the KVM VM had used Ubuntu's 6.8.0-100-generic — both based on the same upstream 6.8 kernel, but the PVE kernel might have lacked patches needed for Blackwell's open module support.

The breakthrough came when the assistant searched for the exact error pattern and discovered a GitHub issue in the NVIDIA/open-gpu-kernel-modules repository ([msg 559]). The issue described the identical symptom: cuInit() returning error code 3 with the open kernel module. The root cause was the Heterogeneous Memory Management (HMM) feature of the nvidia_uvm module, which was incompatible with certain kernel configurations — including the Proxmox VE kernel. The fix was elegantly simple: set uvm_disable_hmm=1 as a module parameter for nvidia_uvm.

The assistant's "aha!" moment is captured in message 559 ([msg 559]): "This is exactly our bug. The key fix mentioned is: Setting uvm_disable_hmm=1 on the open kernel module sometimes works around/solves the issue." The assistant immediately connected the external knowledge (the GitHub issue) with local observations — nvidia-smi showing "Addressing Mode: HMM" — to produce a confident diagnosis.

The fix was applied by creating /etc/modprobe.d/nvidia-uvm-hmm.conf with the single line options nvidia_uvm uvm_disable_hmm=1, then reloading the module. After this change, CUDA initialized successfully on both the host and inside the LXC container, confirming all eight GPUs were accessible ([msg 565]). A quick matrix multiply test on GPU 0 verified that compute was working correctly.

Confirming the P2P Breakthrough

With CUDA working, the assistant could finally run the NCCL bandwidth test that would quantify the topology improvement. Message 569 ([msg 569]) reports the results:

Same-NUMA P2P: 53.76 GB/s
Cross-NUMA P2P: 40.24 GB/s
All GPU pairs have P2P access — no VFIO barriers This was a massive improvement over the KVM VM, where VFIO had forced all cross-GPU traffic through host memory with a ~13 microsecond latency floor. The LXC container, sharing the host kernel, provided direct PCIe P2P communication. The assistant's conclusion was clear: "This is a massive improvement. In the KVM VM, P2P was likely going through host memory due to VFIO, causing the ~13us latency floor. Now we have direct PCIe P2P" ([msg 569]). The P2P verification marked the transition from infrastructure debugging to application deployment. The foundation was solid: eight Blackwell GPUs with direct P2P access in an LXC container, CUDA compute working, and the software stack ready for the inference server.

The Dependency Cascade: Launching SGLang

The next phase — launching the SGLang inference server for the GLM-5-NVFP4 model — proved to be a gauntlet of dependency and compatibility issues. The assistant's first launch attempt ([msg 583]) failed due to a torchvision compatibility issue with PyTorch 2.9.1. The attempted fix — upgrading torch to 2.10.0 — broke sgl_kernel due to ABI incompatibility. The assistant carefully rolled back torch to 2.9.1 and installed a matching torchvision 0.24.1.

With dependencies restored, the assistant prepared the definitive launch command. The command was a carefully constructed artifact encoding accumulated knowledge from hours of debugging:

NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=5 NCCL_MIN_NCHANNELS=8 OMP_NUM_THREADS=8
SAFETENSORS_FAST_GPU=1 CUDA_HOME=/usr/local/cuda-12.8
nohup /root/ml-env/bin/python3 -m sglang.launch_server \
  --model lukealonso/GLM-5-NVFP4 --served-model-name glm-5 \
  --trust-remote-code --tp 8 --mem-fraction-static 0.92 \
  --quantization modelopt_fp4 --attention-backend flashinfer \
  --nsa-decode-backend trtllm --nsa-prefill-backend trtllm \
  --moe-runner-backend flashinfer_cutlass \
  --enable-flashinfer-allreduce-fusion

Each flag represented a deliberate choice informed by earlier failures. The CUDA_HOME override pointed to CUDA 12.8 (not the system's primary 13.1) because sglang's compiled extensions were linked against that version. The NSA backend flags (trtllm) were the result of extensive NaN-debugging in earlier segments — other NSA backends produced NaN on the SM120 architecture. The NCCL environment variables reflected the confirmed P2P topology, with NCCL_P2P_LEVEL=5 enabling P2P transfers and NCCL_MIN_NCHANNELS=8 ensuring sufficient communication channels.

But the launch immediately failed with a KeyError: 'glm_moe_dsa' ([msg 585]). The transformers library (version 4.57.1) did not include a configuration mapping for the model type used by GLM-5-NVFP4. This triggered a lengthy investigation into whether custom Python files were missing, whether trust_remote_code was being passed correctly, and how SGLang registered its model types.

The resolution came through a synthesis of multiple threads. The assistant discovered that SGLang's source code did contain built-in support for GlmMoeDsaForCausalLM ([msg 595]), but the model loading pipeline called HuggingFace's AutoConfig.from_pretrained() before SGLang could register its custom model types. The model's config.json had model_type: glm_moe_dsa but no auto_map field — meaning trust_remote_code couldn't help because there was no custom code to load.

The fix was to upgrade transformers to version 5.2.0 ([msg 611]), which had been released on February 16, 2026 and included native support for the glm_moe_dsa architecture. After upgrading, the model type was recognized correctly ([msg 612]):

transformers: 5.2.0
model_type: glm_moe_dsa
architectures: ['GlmMoeDsaForCausalLM']

The Server That Wouldn't Start

Even after the transformers upgrade, the server refused to start cleanly. The model loading process began — downloading 83 safetensors shards across eight GPUs — but then went silent. GPU utilization dropped to zero. The process was alive (221 threads, 7.3 GB RSS) but sleeping ([msg 620]).

The assistant investigated process wait channels, finding that the main process (TP0) was not in any specific wait state while the tensor-parallel workers were waiting on futex_wait_queue — a synchronization primitive. This suggested a deadlock or resource starvation. The server log showed that the process was stuck during ModelRunner initialization, but the traceback was truncated.

The root cause, discovered in message 633 ([msg 633]), was that FlashInfer's Just-In-Time CUDA kernel compilation required the ninja-build tool, which was not installed on the system. The server was crashing silently during ModelRunner initialization because the JIT compilation pipeline couldn't find ninja. The error message was unambiguous: FileNotFoundError: [Errno 2] No such file or directory: 'ninja' ([msg 636]).

A single apt-get install -y ninja-build command resolved the issue ([msg 634]). After several more attempts involving shell quoting issues and silent nohup failures, the server finally started. Message 640 ([msg 640]) captures the triumphant moment:

"The server is fired up and ready to roll!"

Key metrics: max_total_num_tokens=495488 (KV cache capacity for ~495K tokens), available_gpu_mem=5.19 GB per GPU after loading. A quick curl test confirmed the model was generating responses with reasoning content — the model's chain-of-thought reasoning was working correctly.

Benchmarking: The Plateau at 806 Tok/s

With the server operational, the assistant ran benchmarks to measure throughput. The results were:

32 concurrent: 438 total tok/s (peak output: 217 tok/s)
64 concurrent: 757 total tok/s (peak output: 417 tok/s)
128 concurrent: 806 total tok/s (peak output: 453 tok/s) The throughput scaling curve told a concerning story: going from 32 to 64 concurrency nearly doubled throughput, but going from 64 to 128 yielded only a 6% improvement. The system was plateauing early. The assistant's diagnostic response was immediate and structured ([msg 648]). It checked the server log for warnings, finding a critical clue: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." This warning signaled that the quantization configuration might not be optimal for the Blackwell architecture. The absence of MoE kernel config warnings was itself noteworthy — it meant the server was using default kernel configurations rather than Blackwell-optimized settings. A single-stream latency test revealed the deeper problem: 500 tokens took 45.48 seconds to generate, yielding approximately 11 tok/s ([msg 648]). For eight top-tier GPUs, this was extremely slow. The model was generating extensive chain-of-thought reasoning (2065 characters of reasoning content for 500 output tokens), but the raw generation speed was bottlenecked by suboptimal MoE kernel implementations.

The Unfinished Story: Kernel Tuning for Blackwell

The throughput plateau at 806 tok/s, while respectable, fell short of the ambitious targets (1k+ total tok/s, >100 tok/s single-stream). The assistant identified the likely cause: the MoE kernels were not properly tuned for the Blackwell SM120 architecture. The user had indicated that PR #14311 (already merged into SGLang's main branch) and custom kernel tuning were essential for good performance. The PR addressed the shared memory constraint (100KB per SM on Blackwell vs. larger pools on previous architectures), but the default kernel configurations were still being applied.

The assistant's analysis in message 648 ([msg 648]) represents the transition from infrastructure validation to performance optimization. The P2P topology improvement had been confirmed, the software stack was complete, and the server was running. But the numbers made it clear that the next frontier was kernel-level tuning — configuring MoE dispatch, attention block sizes, and quantization parameters specifically for the Blackwell architecture.

The DeepGemm warning about scale_fmt not being ue8m0 was particularly telling. DeepGemm is a specialized kernel for Blackwell that leverages hardware support for FP4 and FP8 matrix multiplication. The warning indicated that the checkpoint's quantization format didn't match what DeepGemm expected, which could cause both performance degradation and potential accuracy issues. This is a known challenge with early Blackwell deployments: the hardware supports novel quantization formats, but the software ecosystem is still catching up.

Themes and Lessons

Several themes emerge from this segment:

The compounding nature of infrastructure debugging. Each resolved blocker revealed the next. CUDA initialization → model type recognition → missing build tools → kernel configuration. The assistant could not have predicted the full sequence in advance; each layer of the stack had to be validated before the next could be tested. This is a fundamental characteristic of deep learning infrastructure work: the software stack is so deep that failures at any layer block progress at all layers above it.

The importance of external knowledge. The HMM fix was discovered through a GitHub issue, not through first-principles debugging. The transformers version requirement was identified through package resolution and HuggingFace documentation. The PR #14311 verification came from the user's research repository. In complex deployments, community knowledge and prior art are often the fastest path to resolution. The assistant's ability to search for, find, and apply external knowledge was critical to every breakthrough in this segment.

The value of structured documentation. Message 539 ([msg 539]) served as a comprehensive status report that crystallized the system state at a critical juncture. The todo lists throughout the segment provided a shared cognitive framework for tracking progress across dozens of messages. This structured approach to documentation — maintaining a clear picture of what's accomplished, what's blocked, and what's next — is essential when navigating a multi-layered debugging process.

The gap between infrastructure and performance. The LXC container delivered on its promise of bare-metal GPU topology with P2P access at 53 GB/s. But the throughput numbers did not reflect this improvement because the bottleneck had shifted upward in the software stack — from inter-GPU communication to kernel execution efficiency. This is a crucial lesson: fixing a lower-level bottleneck (P2P latency) doesn't automatically improve end-to-end performance if a higher-level bottleneck (MoE kernel efficiency) dominates.

The bleeding edge of Blackwell support. The entire deployment was happening on a platform that had only been available for months. The NVIDIA driver (590.48.01) was the first to support Blackwell's compute capability 12.0. The transformers library had only added GLM-5 support in version 5.2.0, released days before this session. SGLang's PR #14311 for SM120 block sizes was merged but not yet battle-tested. Every component was at the frontier of what the software ecosystem supported, and the cracks between versions and configurations were where all the bugs lived.

The Technical Details: What Made the Difference

Several specific technical decisions deserve attention for their role in the eventual success:

The uvm_disable_hmm=1 parameter. This was the single most impactful fix in the entire segment. The Heterogeneous Memory Management feature in the NVIDIA open kernel module was designed to enable GPU memory access from CPU page faults, but it required specific kernel infrastructure that the PVE kernel didn't fully support. Disabling it fell back to a more traditional UVM (Unified Virtual Memory) model that worked reliably. This is a parameter that most CUDA users will never encounter, but it became the critical enabler for the entire deployment.

The transformers 5.2.0 upgrade. The GLM-5-NVFP4 model uses a custom model architecture (GlmMoeDsaForCausalLM) that was only added to HuggingFace transformers in version 5.2.0, released on February 16, 2026. The assistant's initial setup used transformers 4.57.1, which predated this support. The upgrade was non-trivial because it also pulled in a newer huggingface-hub library (0.36.2 → 1.4.1) and several other dependencies, but it was the only way to get the model type recognized.

The ninja-build installation. FlashInfer uses Just-In-Time compilation to generate optimized CUDA kernels for the specific model and hardware configuration. This requires ninja, a build system that can parallelize compilation. The absence of ninja caused a silent crash during ModelRunner initialization — the error was logged but the process died before producing a visible failure. This is a classic example of a "missing tool" bug that's easy to fix once identified but difficult to diagnose because the error message is buried in logs.

The NCCL environment variables. The NCCL_P2P_LEVEL=5 setting was critical for leveraging the P2P topology. In the KVM VM, P2P was disabled (NCCL_P2P_LEVEL=PHB), forcing all cross-GPU communication through host memory. In the LXC container, the NODE/SYS topology allowed direct GPU-to-GPU transfers. The NCCL_MIN_NCHANNELS=8 setting ensured that NCCL would use enough communication channels to saturate the P2P bandwidth.

The Benchmark Numbers in Context

The throughput results deserve careful interpretation. At 806 total tok/s (453 output tok/s peak), the system was performing respectably for a 744B-parameter MoE model, but not at the level expected from eight Blackwell GPUs. For comparison:

The KVM VM baseline was ~485 tok/s at 64 concurrency
The LXC container achieved 757 tok/s at 64 concurrency — a 56% improvement
But at 128 concurrency, the LXC container only reached 806 tok/s — a mere 6% gain over 64 This early plateau suggests that the bottleneck is not inter-GPU communication (which the P2P topology should have improved) but rather per-GPU compute efficiency. The single-stream result of ~11 tok/s confirms this: if the bottleneck were communication, single-stream performance would be much higher because there's no contention. Instead, the model is compute-bound on each GPU, likely due to suboptimal MoE kernel configurations for the SM120 architecture. The DeepGemm warning about scale_fmt is a strong signal that the quantization path is not optimal. DeepGemm is designed to exploit Blackwell's native FP4 support, but if the checkpoint's scale format doesn't match what DeepGemm expects, the kernel may fall back to a slower path or use suboptimal memory layouts.

Conclusion

This segment of the opencode session tells a story of systematic problem-solving across multiple layers of the ML inference stack. The assistant navigated from kernel module parameters to HuggingFace transformers versions to JIT compilation build tools, resolving each blocker with a combination of diagnostic rigor, external knowledge, and adaptive reasoning. The result — a working GLM-5-NVFP4 inference server on eight Blackwell GPUs achieving 806 tok/s — represents a genuine engineering achievement.

But the story is not finished. The throughput plateau signals that the Blackwell architecture's full potential remains untapped. The assistant's identification of MoE kernel tuning as the next frontier sets the stage for the work ahead: configuring kernel parameters for SM120, addressing the DeepGemm scale format warning, and optimizing the attention and MoE backends for this specific hardware-model combination. The foundation is solid; the optimization phase is just beginning.

The most important lesson from this segment is that deploying cutting-edge hardware requires navigating the ragged edge of software maturity. Every component — from the NVIDIA driver to HuggingFace transformers to SGLang — was at the boundary of what its current version supported. The bugs that emerged were not random; they were the inevitable consequence of combining components that had never been tested together on this specific hardware. The assistant's methodical approach to isolating, diagnosing, and resolving each issue, combined with the ability to find and apply external knowledge, turned a seemingly intractable blocker into a working deployment.

The HMM parameter that almost stopped the entire project — a single bit in a kernel module configuration — serves as a reminder that in complex systems, the smallest details can have the largest consequences. And the 806 tok/s benchmark, while not yet meeting the ambitious targets, represents a solid foundation for the kernel tuning work that will follow.