Chunk 5.0

In this chunk, the assistant resolved the critical CUDA initialization blocker that had halted progress. By investigating the issue, they discovered that the NVIDIA open kernel module's Heterogeneous Memory Management (HMM) feature was incompatible with the Proxmox VE kernel, causing `cuInit()` to hang or return error code 3. The fix was to set `uvm_disable_hmm=1` as a module parameter for `nvidia_uvm`, which immediately allowed CUDA to initialize successfully on both the host and inside the LXC container. This confirmed that the LXC approach provides true bare-metal GPU topology (NODE/SYS) with P2P access at 53 GB/s same-NUMA, a major improvement over the VFIO-limited KVM VM. With CUDA working, the assistant proceeded to launch the sglang inference server for the GLM-5-NVFP4 model. They encountered and resolved several additional issues: upgrading transformers from 4.57.1 to 5.2.0 to support the `glm_moe_dsa` model type, installing `ninja-build` for FlashInfer JIT compilation, and ensuring all dependencies were compatible. The server started successfully and benchmarks were run, achieving throughput of 438 tok/s at 32 concurrency, 757 tok/s at 64, and 806 tok/s at 128 concurrent requests. However, single-stream performance remained around 11 tok/s (500 tokens in 45 seconds), similar to the KVM VM baseline. The key theme of this chunk is overcoming software compatibility barriers to enable GPU compute in a containerized environment, followed by initial performance characterization. While the P2P topology improvement was confirmed, the throughput numbers did not yet reflect a significant gain over the previous approach. This suggests that further tuning of MoE kernel configurations for Blackwell (SM120) architecture is needed, as the user indicated that PR #14311 (already merged) and custom kernel tuning are essential for good performance. The foundation is now solid, but achieving the targets of 1k+ total tok/s and >100 tok/s single-stream will require deeper optimization of the inference kernels.

From Blocker to Benchmark: The Blackwell GPU Inference Breakthrough 2059 words

Message Articles