The NUMA Wall: A Build Failure That Exposes the Bleeding Edge of Blackwell AI

Introduction

In the high-stakes world of deploying trillion-parameter language models on cutting-edge hardware, the smallest missing system library can bring progress to a screeching halt. Message [msg 3107] captures exactly such a moment: a single bash command, issued with careful precision, that fails not because of any conceptual flaw, but because a Linux system library called libnuma is absent from the build environment. This message is a microcosm of the entire opencode session—a story of pushing against the boundaries of what's possible with NVIDIA's Blackwell GPUs (SM120 compute capability), where every step forward requires solving problems that don't exist on older, more mature hardware.

The Broader Context: Why This Build Matters

To understand message [msg 3107], we must first understand the journey that led to it. The team had spent days building a complete EAGLE-3 speculative decoding pipeline for Kimi-K2.5, a ~1 trillion parameter Mixture-of-Experts model deployed across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The EAGLE-3 training pipeline was a remarkable achievement: 10,000 synthetic samples generated over 5.3 hours, 828 GB of hidden state data extracted at 3,165 tokens per second, and a 5-epoch finetune completed in 2.6 hours. Everything worked—except the final integration.

When the trained drafter was plugged into vLLM's EAGLE-3 speculative decoding engine, the acceptance rate was a disastrous ~15%. Both the custom-trained drafter and the pre-trained AQ-MedAI baseline performed identically poorly, yielding a throughput of just 0.66x compared to the baseline of 82.5 tok/s without speculation. As the assistant diagnosed in [msg 3089]: "The problem is NOT our training... The problem is the EAGLE-3 + vLLM + DeepSeekV3/MLA integration itself."

This diagnosis led to a decisive pivot. The user directed the team to try SGLang, an inference engine that has first-class EAGLE-3 support and is explicitly tested with Kimi-K2 drafters, reporting 1.8x speedups. The assistant immediately began the migration: stopping the vLLM service, freeing the GPUs, and preparing to install SGLang.

The Specific Problem: SM120 and sgl-kernel

SGLang was already present on the system as a development checkout at /root/sglang/. However, a critical component was missing: sgl-kernel, the CUDA kernel library that provides optimized implementations for SGLang's operations. An earlier version of sgl-kernel was installed, but it had been compiled for SM100 (compute capability 10.0, corresponding to NVIDIA's Hopper architecture), not SM120 (compute capability 12.0, corresponding to Blackwell). The Blackwell GPUs in this system—two RTX PRO 6000 cards, later upgraded to eight—require SM120-compatible kernels.

Previous attempts to rebuild sgl-kernel had failed for various reasons. In [msg 3103], the build failed because scikit-build-core was missing. The assistant installed it in [msg 3104]. In [msg 3105], the build failed again with a more cryptic error. Each failure taught the assistant something about the build environment.

Anatomy of the Command

Message [msg 3107] contains a single bash command that represents the assistant's most carefully constructed attempt yet:

CUDA_HOME=/usr/local/cuda-12.8 \
CUDACXX=/usr/local/cuda-12.8/bin/nvcc \
PATH=/usr/local/cuda-12.8/bin:$PATH \
SGL_KERNEL_CUDA_ARCHS="120" \
~/.local/bin/uv pip install --python ~/ml-env/bin/python3 \
  -e /root/sglang/sgl-kernel/ --no-build-isolation 2>&1 | tail -20

Every environment variable tells a story:

The Error: NUMA Not Found

The build fails with:

CMake Error at .../FindPackageHandleStandardArgs.cmake:290 (message):
  Could NOT find NUMA (missing: NUMA_INCLUDE_DIRS NUMA_LIBRARIES)

NUMA (Non-Uniform Memory Access) is a Linux library that provides APIs for querying and controlling memory topology on multi-socket systems. On systems with multiple CPU sockets, NUMA awareness is critical for performance—it allows processes to bind memory and threads to specific NUMA nodes, reducing memory access latency. The sgl-kernel build depends on libnuma through its mscclpp (Microsoft Collective Communication Library) dependency, which uses NUMA for optimizing GPU-to-GPU communication.

The error is straightforward: the libnuma-dev package (which provides the headers and shared library) is not installed on this Ubuntu 24.04 system. The CMake build system searches for NUMA using FindPackageHandleStandardArgs and fails when it can't locate the include directory or library.

Assumptions and Knowledge Required

To understand this message, several pieces of background knowledge are necessary:

  1. CUDA toolkit versioning: The assistant assumes that CUDA 12.8 is the correct version for building sgl-kernel, based on earlier experience with flash-attn compatibility. This is a learned assumption from previous build failures.
  2. SM120 architecture: The assistant knows that Blackwell GPUs use compute capability 12.0 and that sgl-kernel source code supports it (confirmed in [msg 3102] by grepping the CMakeLists.txt). The SGL_KERNEL_CUDA_ARCHS variable is assumed to be the correct way to specify target architectures.
  3. Build system knowledge: The assistant understands that scikit-build-core is the build backend (installed in [msg 3104]), that --no-build-isolation is needed for editable installs, and that environment variables control CUDA toolchain selection.
  4. NUMA dependency: The assistant did not anticipate the NUMA dependency. This is a reasonable oversight—NUMA is typically installed on server systems, and its absence is unusual for a machine with 8 GPUs. However, this system was provisioned as a fresh Ubuntu 24.04 installation, and libnuma-dev was not included in the initial setup.
  5. The mscclpp dependency: The error trace points to mscclpp (Microsoft Collective Communication Library), which is a sub-dependency of sgl-kernel. Understanding this chain requires knowledge of the SGLang build architecture.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. NUMA is a build dependency for sgl-kernel on SM120: This was not previously known. The build system's CMake configuration requires libnuma for the mscclpp component.
  2. The build command structure is validated: The environment variables, CUDA paths, and build flags are all correct—the error is purely a missing system dependency, not a configuration mistake. This confirms the approach is sound.
  3. The sgl-kernel source supports SM120: The build progressed far enough to reach the NUMA check, meaning the SM120 architecture flags were accepted by the build system. This validates the earlier grep-based investigation.
  4. The build environment needs additional system packages: Beyond Python dependencies, the system needs libnuma-dev (and potentially other system libraries) installed via apt.

The Thinking Process

The reasoning visible in this message reveals a systematic debugging approach. The assistant has been iterating on the build command, adding environment variables one at a time as each previous attempt revealed a new missing piece. The progression is clear:

  1. [msg 3103]: Basic uv pip install → fails, missing scikit-build-core
  2. [msg 3104]: Install build dependencies → succeeds
  3. [msg 3105]: Add SGL_KERNEL_CUDA_ARCHS="120" → fails with CMake error (not shown in tail)
  4. [msg 3106]: Verify nvcc exists → confirmed
  5. [msg 3107]: Add explicit CUDA_HOME, CUDACXX, PATH → fails with NUMA error Each iteration adds precision. The assistant is methodically eliminating variables, ensuring each component of the build toolchain is correctly configured before moving to the next potential issue. This is classic debugging: isolate the failure, fix it, then re-run with more information. The choice to use tail -20 is also telling. The assistant expects the error to be at the end of the output, and truncating to the last 20 lines reduces noise while preserving the critical error message. This is a practical optimization for remote command execution where full build logs can be thousands of lines.

Broader Significance

This message represents a recurring theme in the entire opencode session: the bleeding edge is hard. Every component—Blackwell GPUs, CUDA 12.8, SGLang nightly builds, EAGLE-3 speculative decoding, Kimi-K2.5 INT4 quantization—is at the frontier of what's possible. At the frontier, nothing works out of the box. Every integration requires solving problems that the broader community hasn't encountered yet because nobody else has this exact combination of hardware and software.

The NUMA error is trivial to fix (apt-get install libnuma-dev), but it represents a class of problems that define this kind of work. The assistant couldn't have known about this dependency in advance—it's an emergent property of the specific build configuration. The only way to discover it is to try, fail, read the error, and adapt.

This message also demonstrates the value of persistent state across a long session. The assistant remembers that CUDA 12.8 is needed (from the flash-attn saga in segment 0), remembers that scikit-build-core was missing (from [msg 3103]), and remembers that SM120 support exists in the source (from [msg 3102]). Each piece of knowledge is earned through failure, and each failure narrows the path forward.

Conclusion

Message [msg 3107] is, on its surface, a failed build command. But in the context of the broader session, it's a testament to systematic debugging, the challenges of bleeding-edge hardware, and the incremental nature of progress in AI infrastructure. The NUMA error is a small wall—easily demolished with a single apt-get install—but it's representative of the dozens of small walls that must be broken through to deploy a trillion-parameter model on hardware that didn't exist six months ago. The assistant's methodical approach, building on knowledge from previous failures, is the only reliable way to navigate this terrain.