The Custom Build Conundrum: Deploying a Modified SGLang Across Machine Boundaries

Introduction

In the high-stakes world of large language model deployment, the difference between a research prototype and a production-ready system often comes down to infrastructure plumbing. Message 11754 of this opencode session captures a pivotal moment in deploying Kimi K2.6—a 548 GB MoE model—with DDTree speculative decoding on a freshly provisioned 8× NVIDIA B300 SXM6 machine. The message is a masterclass in the kind of real-world systems thinking that separates successful ML deployments from stalled experiments. It reveals the assistant wrestling with a problem that is simultaneously mundane and profound: how to transfer a heavily customized software stack from one machine to another when the standard package management tools simply cannot deliver what is needed.

The message sits at the intersection of several converging workflows: a 548 GB model downloading at 190 MB/s in the background, a freshly inventoried B300 machine with 2.2 TB of VRAM and full NVLink, and a critical realization that the SGLang build powering the DDTree speculative decoding algorithm is not the public PyPI package but a custom research fork. The assistant's reasoning in this message—its evaluation of transfer strategies, its investigation of the CT200 environment, and its architectural understanding of what can be copied versus what must be rebuilt—forms the subject of this article.

Context: The B300 Deployment Mission

To understand why this message exists, we must trace the thread back through the conversation. The user and assistant have been engaged in an extended campaign to deploy and benchmark Kimi K2.6 with DFlash and DDTree speculative decoding across multiple hardware platforms. The journey began on CT200, a machine with 8× RTX PRO 6000 Blackwell GPUs connected via PCIe, where the assistant resolved CUDA toolkit incompatibilities, benchmarked parallelism strategies (TP8, PP8, EP8, EP4), deployed DFlash speculative decoding, and ultimately achieved a 1.3× speedup over the autoregressive baseline. The DDTree variant—a more sophisticated tree-based speculative decoding algorithm—was then implemented through a series of SGLang patches that modified eight core files including cuda_graph_runner.py, ddtree_utils.py, and dflash_info.py.

The user then provided access to a B300 machine ([msg 11746]): "B300 Machine ready: ssh root@86.38.182.109". This was the NVLink target all along—the hardware where tensor parallelism could finally shine without the PCIe AllReduce bottleneck that had plagued the earlier benchmarks. The assistant's reconnaissance ([msg 11747]) revealed an extraordinary machine: 8× NVIDIA B300 SXM6 GPUs with 275 GB each (2.2 TB total VRAM), compute capability 10.3 (SM103), CUDA 13.0 toolkit, 2 TB RAM, and full NVLink connectivity. This was the ideal testbed for DDTree.

The assistant immediately began setting up the environment: bootstrapping uv as a package manager, creating a Python 3.12 virtual environment, installing huggingface_hub with hf_transfer for accelerated downloads, and kicking off the 548 GB model download in the background ([msg 11752], [msg 11753]). By message 11754, the download was progressing at 190 MB/s with an estimated 48 minutes remaining. But a deeper problem was surfacing.

The Core Problem: A Custom Build That Can't Be Pip-Installed

The central insight that drives this message is the assistant's realization that the SGLang version running on CT200 is not the public release. The version string 0.4.3.post2 appears innocuous—a standard semantic versioning label with a post-release patch number. But the assistant knows better. It recalls the server arguments from the CT200 deployment:

Looking at the server arguments from CT200, there were tons of DDTree-specific flags and DFLASH parameters that definitely aren't in any public sglang release, which means the CT200 build was heavily customized—possibly a research fork or a newer version mislabeled as 0.4.3.post2.

This is a critical distinction. The DDTree algorithm requires modifications across the SGLang codebase: new command-line flags (--speculative-ddtree-budget, --speculative-ddtree-topk), modified CUDA graph runners, custom attention backends, and new utility modules for tree-path sampling. None of these exist in the public SGLang repository. The assistant's reproduction package (/data/dflash/k26-ddtree-repro/) contains eight patched files, but these are only the delta—the changes on top of a base that is itself already customized.

The assistant considers whether it can simply install the public SGLang 0.4.3.post2 from PyPI and overlay the patches:

The repro package's patches/full/ directory contains the complete working speculative files with DDTree already integrated, so if I install the base sglang 0.4.3.post2 and overlay all the full patch files on top, I should get the complete implementation. The key question is whether the public 0.4.3.post2 version exists or if that version string itself is a custom label.

This line of reasoning reveals the assistant's deep understanding of the software supply chain. It recognizes that the version string could be a deliberate deception—a custom build masquerading as a standard release. The public PyPI version of SGLang 0.4.3.post2 (from early 2025) predates the DDTree implementation entirely. Even if it existed, overlaying eight patched files would not be sufficient because the DDTree changes ripple across hundreds of files: server_args.py (which defines the flags), cuda_graph_runner.py (which sizes the graph for tree verification), triton_backend.py (which handles the custom attention mask), and the K2.6 model file itself.

The assistant correctly concludes: "I can't just pip install and patch; I need the entire custom sglang package from CT200."

The Reasoning Process: Evaluating Transfer Strategies

The message's reasoning section is a careful exploration of the options available for getting the custom SGLang build onto the B300 machine. The assistant considers four approaches, each with distinct trade-offs:

Approach 1: PyPI Install + Patch Overlay

This is the simplest option but also the most fragile. Install the public SGLang from PyPI, then copy the eight full patched files from the reproduction package. The assistant correctly identifies this as insufficient because the base package itself is missing DDTree infrastructure that exists in the CT200 build but is not captured by the eight patch files.

Approach 2: Full Venv Rsync

Copy the entire CT200 virtual environment (approximately 4.9 GB as later revealed) to B300. This would preserve every dependency at exactly the right version. However, it would also copy SM120-specific compiled binaries (.so files for sgl-kernel, flashinfer cubins) that might not work on SM103 architecture. The assistant notes: "the compiled wheels likely contain multiple SASS targets or PTX, and flashinfer JITs at runtime so it'll recompile for sm_103 on first use." This is a nuanced understanding of GPU binary compatibility—NVIDIA's SASS (Shader Assembly) is architecture-specific, but PTX (Parallel Thread Execution) is a portable intermediate representation that can be compiled just-in-time for the target architecture.

The obstacle here is network topology. CT200 is on an internal network (10.1.2.200 behind kpro6), while B300 is a public machine (86.38.182.109). Direct rsync between them is unlikely to work without routing. The local machine (where the assistant's reasoning is running) can reach both, making it a natural intermediary.

Approach 3: Tar + Transfer via Local Intermediary

This is the approach the assistant begins to implement. Tar the custom SGLang package directory on CT200, pull it to the local machine, then push it to B300. This isolates the custom component from the rest of the venv, allowing the assistant to install binary dependencies fresh on B300 (compiled for SM103) while preserving the custom Python source code.

Approach 4: Install Binary Stack Fresh + Copy Custom SGLang Source

This is the refinement of Approach 3 that the assistant settles on. The plan is:

  1. Install the binary stack (torch, sgl-kernel, flashinfer) on B300 matching the CT200 pip-freeze.txt versions
  2. Copy CT200's entire custom SGLang Python directory to replace the pip-installed version
  3. This ensures all DDTree-related code is present while the compiled extensions are built for the correct architecture The assistant articulates this clearly: "For the compiled dependencies (torch, sgl-kernel, flashinfer), I should install fresh on B300 via pip rather than copying the SM120 builds, since flashinfer JITs anyway and the versions matter more than the specific architecture builds."

The Investigation: Probing CT200's SGLang Package

The message culminates in a bash command that probes the CT200 environment for critical details about the SGLang package. This is not idle curiosity—every piece of information directly informs the transfer strategy:

Assumptions and Their Risks

The assistant's reasoning rests on several assumptions, each carrying its own risk profile:

Assumption 1: The public SGLang 0.4.3.post2 does not contain DDTree. This is almost certainly correct. DDTree is a research-stage feature that appeared in SGLang's development branch after the 0.4.3 release. The assistant's confidence is justified by having worked with both the public API and the CT200 custom build.

Assumption 2: FlashInfer will JIT-compile successfully for SM103. FlashInfer does support runtime JIT compilation, but the SM103 architecture is relatively new (Blackwell Ultra). There is a risk that the JIT compiler lacks SM103-specific kernel templates or that the CUDA 13.0 toolkit's nvcc has incomplete support. The earlier CT200 deployment required specific CUDA toolkit versions (13.0) to resolve SM120 compatibility issues—a similar headache could await on SM103.

Assumption 3: The local machine can reach both CT200 and B300 for file transfer. The assistant has already established SSH connections to both machines, so network reachability is confirmed. However, transfer bandwidth is unknown. The earlier HuggingFace download achieved 190 MB/s, suggesting the B300 machine has excellent internet connectivity. The CT200→local leg is internal and likely fast. The local→B300 leg over the public internet could be the bottleneck.

Assumption 4: The eight patched files in the reproduction package are the only customizations needed beyond the CT200 base. This is the riskiest assumption. The CT200 build may contain additional modifications beyond what the assistant has documented. The assistant acknowledges this implicitly by deciding to copy the entire SGLang package rather than just the eight files.

Assumption 5: Binary compatibility across architectures. The assistant plans to install torch, sgl-kernel, and flashinfer fresh on B300, which should produce SM103-compatible binaries. However, PyPI wheels for these packages may not include SM103 support yet. The assistant may need to build from source or use nightly builds.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Understanding of SGLang architecture: SGLang is a serving engine for LLMs that supports various speculative decoding algorithms. Its modular design separates the serving framework (Python) from kernel implementations (CUDA/C++). The DDTree algorithm extends the DFlash speculative decoding approach with tree-structured candidate generation.
  2. Knowledge of GPU architecture compatibility: NVIDIA GPUs have compute capabilities (e.g., SM100, SM103, SM120) that determine which compiled GPU code they can execute. PTX is portable, SASS is architecture-specific. FlashInfer uses JIT compilation to generate kernels at runtime for the target architecture.
  3. Familiarity with Python packaging and virtual environments: The distinction between pip-installed packages, editable installs, and custom builds. The role of dist-info directories, .so files, and the site-packages directory structure.
  4. Understanding of the deployment context: The earlier conversation segments established that DDTree is a tree-based speculative decoding algorithm that the assistant implemented through patches to SGLang. The B300 machine represents the NVLink target where tensor parallelism can achieve its full potential.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. CT200 SGLang package characteristics: 32 MB, 1,883 Python files, installed as a wheel (not editable), with a separate sglang-kernel package for compiled extensions. The total venv is 4.9 GB.
  2. The custom build hypothesis is confirmed: The presence of DDTree-specific server arguments in the CT200 deployment confirms that the SGLang build is heavily customized. The version string 0.4.3.post2 is misleading—this is not the public release.
  3. A viable transfer strategy is formulated: Install binary dependencies fresh on B300, copy the custom SGLang Python source from CT200 via a local intermediary. This separates the architecture-dependent compiled code from the architecture-independent Python source.
  4. A potential shortcut is discovered: The /root/sglang_env.sh file on CT200 may contain the original build or installation script, which could simplify the B300 deployment.

The Broader Significance

This message illuminates a fundamental tension in ML infrastructure: the gap between research code and deployable packages. The DDTree implementation exists as a set of modifications to a living codebase—patches applied to a fork of a fork. There is no clean release, no versioned wheel, no pip install sglang-ddtree. The only copy is the running environment on CT200.

This is the reality of cutting-edge ML deployment. The algorithms that produce the best benchmarks are often the ones that exist only as modifications to someone's working directory. Reproducing results across hardware requires not just copying files but understanding the entire dependency chain: which patches were applied, which versions of which libraries were used, which CUDA toolkit compiled the kernels, and which GPU architecture they target.

The assistant's response to this challenge is instructive. Rather than attempting to reconstruct the build from scratch (which would be error-prone and time-consuming), it treats the CT200 environment as the source of truth and devises a strategy to replicate it on B300 with minimal modification. The key insight is the separation of concerns: Python source code is architecture-independent and can be copied verbatim; compiled extensions must be rebuilt for the target GPU.

This is a pattern that appears repeatedly in ML engineering: the tension between reproducibility and performance optimization. A pure Python implementation would be trivially portable but slow. A heavily optimized CUDA implementation is fast but architecture-specific. The assistant navigates this tension by copying what can be copied and rebuilding what must be rebuilt—a pragmatic middle path.

Conclusion

Message 11754 captures a moment of architectural insight in a complex deployment workflow. The assistant recognizes that the standard package management tools are insufficient for the task at hand and devises a custom transfer strategy based on a deep understanding of the software stack's structure. The reasoning process reveals not just technical knowledge but strategic thinking: evaluating multiple approaches, identifying the critical unknowns, and investigating them systematically.

The message also serves as a case study in the challenges of deploying research ML systems across heterogeneous hardware. The custom build conundrum—where the software you need exists only as a running environment on another machine—is a recurring pattern in this domain. The assistant's response—copy the Python source, rebuild the binaries, use a local machine as transfer intermediary—is a template for solving it.

As the download of Kimi K2.6 continues in the background at 190 MB/s, and the B300 machine waits with its 2.2 TB of VRAM and full NVLink, the assistant has laid the groundwork for the next phase: getting the custom SGLang build onto the target hardware and finally benchmarking DDTree on the architecture it was designed for. The success of this mission depends on the decisions made in this message.