The Milestone Report: How Eight Blackwell GPUs Were Brought Online for SGLang Inference

Introduction

In the course of a sprawling machine learning engineering session spanning dozens of messages and thousands of lines of logs, one message stands out as a quiet but significant milestone. Message [msg 9546] is a brief status report from an AI assistant to its user, announcing that all eight SGLang inference servers are healthy and serving requests on a cluster of RTX PRO 6000 Blackwell GPUs. On its surface, the message is unremarkable: a few lines of summary, a checklist of environment fixes, and a todo update. But beneath this terse report lies the culmination of an intense debugging odyssey that involved linker errors, missing CUDA headers, architecture incompatibilities, and the delicate art of making bleeding-edge hardware work with software that barely supports it.

This article examines message [msg 9546] in depth: why it was written, what decisions it reflects, the assumptions and knowledge it encodes, and the thinking process that produced it. It is a story about the invisible work that goes into making AI infrastructure actually run.

The Message Itself

The assistant writes:

All 8 servers healthy, HTTP 200. 84.7 GB used per GPU (~54 GB model + ~30 GB KV cache). That's solid.

>

Summary of what it took to get SGLang running on RTX PRO 6000 Blackwell SM120:

>

Environment fixes needed: 1. sglang==0.5.12 + torch==2.11.0+cu130 (ABI match with sgl_kernel) 2. sglang-kernel==0.4.2.post2+cu130 (SM100 kernels work on SM120 with ABI match) 3. nvidia-cuda-nvcc==13.2.78 + nvidia-cuda-runtime>=13.2 (PTX 9.2 for SM120) 4. ninja-build (flashinfer JIT compilation) 5. LD_LIBRARY_PATH pointing to nvidia/cu13/lib (libnvrtc.so.13) 6. CUDA_HOME=nvidia/cu13 with lib64 symlink + libcudart.so symlink + libcuda.so stub 7. CCCL headers from flashinfer overlaid onto nvidia/cu13/include/ (nv/, cuda/std/) 8. --attention-backend flashinfer (no FA3/FA4 on SM120)

>

Now let me proceed with preparing the prompt datasets and generation script. Let me stop here for a status report since this was a significant milestone.

The message then updates the todo list to mark several items as completed, including reading the data expansion plan, stopping the previous training run, checking the environment, and installing SGLang for batch inference.

Why This Message Was Written: The Strategic Pivot

To understand why this message exists, we must understand the context that produced it. The session had been focused on training a DFlash drafter model—a speculative decoding architecture—using a DDTree-optimized pipeline. The training had been running on a Proxmox LXC container (CT200) equipped with 8× RTX PRO 6000 Blackwell GPUs. But the user made a strategic decision: halt the training run and pivot to data expansion instead.

This pivot, documented in the preceding chunk (chunk 0 of segment 54), was motivated by a discovered data skew: 77% of the training data was coding-related, which would bias the model. The user wanted to generate 193K diverse prompts from multiple datasets (Infinity-Instruct-0625, WebInstructSub, CodeFeedback, MetaMathQA, Hermes Function Calling, Agent Training) to create a more balanced training corpus.

But generating that much text required high-throughput batch inference—and that required SGLang to run on the Blackwell GPUs. The assistant's task was to set up SGLang for data-parallel (DP=8) inference across all eight GPUs, with each GPU running its own independent server instance.

This is where the real work began. The RTX PRO 6000 Blackwell GPUs use the SM120 architecture, which is so new that most ML software doesn't officially support it. The assistant had to cobble together a working environment from mismatched components, each requiring its own workaround.

The Debugging Journey: What the Summary Encodes

The eight-point summary in message [msg 9546] is a compressed record of dozens of failed attempts. Each bullet point represents a bug that was encountered, diagnosed, and fixed. Let us unpack what each one means.

Point 1: Version matching. The assistant had to pin sglang==0.5.12 and torch==2.11.0+cu130 to ensure ABI compatibility with sgl_kernel. This was not obvious—it required understanding that the sgl_kernel compiled shared objects had specific ABI expectations that would fail silently if linked against a different torch version.

Point 2: SM100 kernels on SM120. The sglang-kernel package compiled for SM100 (the previous Blackwell variant) actually worked on SM120, but only when the ABI matched. This is a lucky accident of NVIDIA's compute capability numbering—SM100 and SM120 share enough instruction set compatibility that SM100 binaries run on SM120 hardware.

Point 3: CUDA toolkit version. The assistant needed nvidia-cuda-nvcc==13.2.78 and nvidia-cuda-runtime>=13.2 because SM120 requires PTX version 9.2 for code generation. Older CUDA toolkits cannot generate code for this architecture.

Point 4: ninja-build. Flashinfer uses JIT compilation, which requires ninja to parallelize CUDA kernel builds. Without it, compilation would fail or be prohibitively slow.

Points 5-6: Library path surgery. The pip-installed CUDA toolkit (nvidia/cu13) had its libraries in lib/ but the linker expected them in lib64/. The assistant created a symlink from lib64 to lib. It also had to create a libcudart.so symlink (the linker needs the .so extension, not .so.13) and a libcuda.so stub for the driver API.

Point 7: CCCL header overlay. This was the most intricate fix. During CUDA graph capture, SGLang's kernel compilation failed because nv/target and related headers were missing from the pip-installed CUDA include tree. These headers are part of NVIDIA's CCCL (CUDA C++ Core Libraries), which flashinfer bundles in its own package. The assistant had to copy the entire nv/ and cuda/std/ directory trees from flashinfer's CCCL into the CUDA toolkit's include directory. This was a multi-step process that initially failed because individual symlinks couldn't handle the directory structure—the assistant had to switch to cp -rn to recursively copy the files.

Point 8: Attention backend selection. Flashinfer's attention backend was the only option that worked on SM120. FlashAttention 3 and 4 (FA3/FA4) are not supported on this architecture, so the assistant had to explicitly pass --attention-backend flashinfer to avoid crashes.

Assumptions and Mistakes

The debugging process reveals several assumptions, some correct and some not.

Correct assumption: The assistant correctly assumed that the pip-installed CUDA toolkit from nvidia-cu13 packages would be sufficient for compilation, even though it lacked some headers. This turned out to be true—the headers could be supplemented from flashinfer's CCCL bundle.

Correct assumption: The assistant assumed that SM100-compiled kernels would work on SM120 hardware. This was verified by the successful server startup and health check.

Incorrect initial assumption: The assistant initially assumed that symlinking individual files would be sufficient to fix the missing headers. When nv/target was symlinked, the next compilation failed because it needed nv/detail/__target_macros, which was in a subdirectory. The assistant had to pivot to copying the entire directory tree.

Incorrect initial approach: The assistant first tried ln -sf for individual files in a loop, but this failed because the target directories didn't exist. The shell escaping was also causing issues. The fix was to use cp -rn to recursively copy directories without overwriting existing files.

Tacit assumption about memory: The assistant assumed that 84.7 GB of GPU memory usage (54 GB model + ~30 GB KV cache) was acceptable and "solid." This turned out to be optimistic—later in the session (chunk 1), the same memory pressure would cause OOM failures during training ramp-up, leading to a torch version rollback.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A verified recipe for running SGLang on Blackwell SM120 GPUs. The eight-point checklist is a reproducible guide that could be used by anyone else attempting the same setup.
  2. A confirmation that SM100 kernels run on SM120. This is non-obvious and valuable information for the ML community working with early Blackwell hardware.
  3. A demonstration that pip-installed CUDA toolkits can be made to work with proper header overlays. This technique of supplementing nvidia/cu13/include/ with CCCL headers from flashinfer is a novel workaround.
  4. A memory budget baseline: 84.7 GB per GPU for the Qwen3.6-27B model with 8K context length, which informs future capacity planning.
  5. A decision point: The message marks the completion of the infrastructure setup phase and the transition to the data generation phase. It serves as a checkpoint that the user can refer back to if something breaks later.

The Thinking Process Visible in the Message

The message reveals a methodical, engineering-minded approach. The assistant does not simply declare success—it provides a structured summary of what was learned. The numbered list is not just documentation; it is a form of knowledge distillation. The assistant is saying: "Here is what I had to do. Here is the order of operations. Here are the constraints."

The tone is confident but not triumphal. "That's solid" is the only evaluative language. The rest is factual. This is characteristic of an engineer who has been burned by overconfidence before and knows that the real test is whether the system stays up under load.

The decision to "stop here for a status report since this was a significant milestone" is itself a thinking artifact. The assistant recognizes that the user needs to know the state of play before the next phase begins. It is managing attention and expectations.

The todo update is also significant. By marking items as completed, the assistant is creating a shared progress model with the user. The remaining items (presumably the data generation and merging steps) are implicitly queued for the next phase.

Conclusion

Message [msg 9546] is a milestone marker in a complex engineering journey. It captures the moment when infrastructure stops being a problem and starts being a tool. The eight-point summary is a testament to the debugging that preceded it—each point a scar from a battle with missing headers, wrong versions, and unsupported architectures.

But the message is also forward-looking. It says, in effect: "The hard part is done. Now we can do the real work." This pivot from infrastructure to data generation is the whole point of the exercise. The assistant has built a platform for high-throughput batch inference on eight of the most powerful consumer-grade GPUs available. The next step is to use that platform to generate the diverse training data that the DFlash model needs.

In the broader arc of the session, this message represents a transition from reactive debugging to proactive construction. The assistant has earned the right to be brief—the work speaks for itself.