The Bridge Between Platforms: Planning K2.6+DDTree Deployment on B300 NVLink Hardware

In the arc of this coding session, message [msg 11750] marks a pivotal transition. For dozens of preceding messages, the assistant had been wrestling with PCIe-bound Blackwell GPUs — the RTX PRO 6000 cards — diagnosing CUDA toolkit incompatibilities, benchmarking parallelism strategies, and ultimately deploying Kimi K2.6 with DFlash speculative decoding on a machine where every AllReduce operation across MoE layers was bottlenecked by PCIe bandwidth. Expert parallelism (EP8) had won that battle, delivering 65 tok/s single-stream throughput by eliminating cross-GPU communication on the critical path. But the original goal had always been NVLink hardware, where tensor parallelism could finally realize its potential.

Now, in message [msg 11750], that hardware has arrived. The user announced "B300 Machine ready" in [msg 11746], and the assistant's reconnaissance across [msg 11747] and [msg 11749] revealed an extraordinary machine: 8× NVIDIA B300 SXM6 GPUs, each with 275 GB of VRAM (2.2 TB total), connected via NVLink at approximately 900 GB/s, running CUDA 13.0 on Ubuntu 24.04 with 240 CPU cores and 2 TB of RAM. This is the NVLink target — the platform where TP8 should shine without the PCIe AllReduce penalty that had dominated the earlier optimization work.

But between reconnaissance and deployment lies a critical planning gap. The assistant knows the hardware specs, but it doesn't yet know: Will the existing software stack (torch 2.11.0+cu130, sglang, flashinfer) work on sm_103, the B300's compute capability? Where exactly did the K2.6 model come from on CT200, so it can be downloaded on the fresh B300 machine? What's the most efficient deployment strategy given the 548 GB model download bottleneck?

Message [msg 11750] is where the assistant thinks through these questions and takes the first concrete step toward answering them. It is a planning and information-gathering message — not flashy, but structurally essential. This article unpacks the reasoning, assumptions, and knowledge boundaries visible in this single message, showing how it bridges the PCIe and NVLink phases of the project.

The Strategic Context

To understand why this message matters, we need to appreciate the journey that led here. The session had been working on two parallel tracks. First, DFlash speculative decoding deployment: getting Kimi K2.6 (a 548 GB INT4 model) to run with a custom drafter model (SubSir/Kimi-K2.6-DFlash-tmp-long) using SGLang's DFlash algorithm, which uses a small draft model to predict multiple tokens ahead, then verifies them against the full model in parallel. Second, cross-platform optimization: the assistant had benchmarked the K2.6+DFlash stack on PCIe Blackwell (RTX PRO 6000) across four parallelism strategies — TP8, PP8, EP8, EP4 — and found that EP8 dramatically outperformed TP8 because expert parallelism avoids the AllReduce bottleneck on MoE layers when GPUs are connected via PCIe. TP8 hit 26 tok/s at concurrency 1, while EP8 reached 65 tok/s.

The B300 machine changes this calculus fundamentally. With NVLink providing ~900 GB/s inter-GPU bandwidth (versus ~64 GB/s for PCIe Gen5 x16), the AllReduce bottleneck that crippled TP8 on PCIe hardware is largely eliminated. TP8 on NVLink should match or exceed EP8, and with CUDA graphs, could potentially double the single-stream throughput. The reproduction package the assistant had just finished creating at /data/dflash/k26-ddtree-repro/ ([msg 11736] through [msg 11745]) captured all the fixes, patches, and benchmark results from the PCIe work — but deploying on the B300 requires adapting that knowledge to a new architecture.

Before any of that can be tested, however, the assistant needs to get the software stack running on sm_103 — a compute capability that didn't exist when most of the current software was released. And it needs to figure out where the 548 GB model came from so it can be downloaded on the fresh machine.

Decoding the Reasoning: sm_103 and the Compatibility Puzzle

The assistant's reasoning section reveals a careful assessment of compatibility risks. The full reasoning text reads:

Looking at the B300 specifications — it has compute capability 10.3 which is newer than B200's 10.0, so I need to verify that torch, sglang, and flashinfer all support sm_103. The CUDA 13 toolkit is already present and torch 2.11.0 with cu130 should handle this architecture through forward compatibility or JIT compilation.

>

For the environment setup, I have Python 3.12.3 but no package manager, so I'll bootstrap pip using uv since it's faster than the traditional get-pip.py approach. The disk space is sufficient for both Llama 2.6 (548 GB) and the drafter model (6.5 GB).

>

The key decision is whether to replicate the exact pinned dependencies from the reproduction package or build fresh — I'll go with replicating the pinned stack since torch 2.11.0+cu130 already worked on SM120 (an even newer Blackwell variant), so it should work fine on sm_103 as well.

>

The real bottleneck is the model download: Llama 2.6 INT4 is 548 GB, which at typical HuggingFace speeds could take 30-90 minutes. I need to figure out the exact HF repo name (likely moonshotai/Kimi-K2.6 or a community quantized version) and kick off the download in parallel while setting up the environment, rather than waiting for one to finish before starting the other.

This reasoning operates on multiple layers simultaneously.

Layer 1: Architecture compatibility. The B300 uses compute capability 10.3 (sm_103), which is newer than the B200's 10.0 (sm_100). This matters because PyTorch, SGLang, and FlashInfer all ship pre-compiled CUDA kernels for specific compute capabilities. If sm_103 isn't supported, the software will fall back to JIT compilation or fail outright. The assistant's reasoning here is nuanced: it notes that torch 2.11.0+cu130 already worked on sm_120 (the RTX PRO 6000's compute capability, which is actually newer than sm_103 in the Blackwell family), so it should work on sm_103 through forward compatibility. This is a reasonable inference — CUDA binaries compiled for sm_90 (Hopper) or sm_100 (Blackwell) should run on sm_103 through binary compatibility, and any missing kernels can be JIT-compiled. However, the assistant is implicitly acknowledging uncertainty by saying "should handle" rather than "will handle."

Layer 2: The FlashInfer question. FlashInfer's attention kernels are architecture-specific. On the RTX PRO 6000 (sm_120), the assistant had to patch SGLang to use the Triton attention backend because FlashInfer rejected sm_120. On sm_103, the situation might be different — FlashInfer might support sm_103 natively, or it might not. The assistant doesn't have this information yet and is implicitly planning to discover it through trial. This is a critical unknown because if FlashInfer doesn't support sm_103, the assistant will need to either patch it (as was done for sm_120) or use the Triton backend exclusively.

Layer 3: Environment bootstrapping. The B300 machine has Python 3.12.3 but no package manager (no pip, no uv). The assistant decides to bootstrap pip using uv, which is faster than the traditional get-pip.py approach. This is a pragmatic choice — uv can install packages in parallel and has better dependency resolution — but it also introduces a new dependency that must itself be installed. The assistant doesn't discuss fallback options if uv installation fails.

Layer 4: The pinned-dependencies decision. The assistant considers whether to replicate the exact pinned dependencies from the reproduction package or build fresh. It chooses replication, reasoning that torch 2.11.0+cu130 worked on sm_120 (a newer Blackwell variant) so it should work on sm_103. This is a conservative, reproducibility-focused choice — it minimizes the risk of version incompatibilities by using the exact same stack that was validated on the PCIe machine.

Layer 5: The model download bottleneck. The assistant correctly identifies that downloading the 548 GB K2.6 model is the critical path. At typical Hugging Face download speeds (50–200 MB/s), this could take 30–90 minutes. The assistant wants to parallelize this with environment setup, but first needs to know the exact Hugging Face repository name. This last point is what drives the bash command in the message.

The Information-Gathering Command

The bash command in message [msg 11750] is deceptively simple. It SSHes into CT200 (the PCIe machine where the model is already deployed) and queries metadata about the K2.6 model:

ssh -o ConnectTimeout=5 root@10.1.2.200 "
cat /root/models/Kimi-K2.6/config.json 2>/dev/null | python3 -c 'import json,sys; c=json.load(sys.stdin); print(\"name_or_path:\", c.get(\"_name_or_path\",\"?\")); print(\"quant:\", c.get(\"quantization_config\",{}).get(\"quant_method\",\"?\"))' 2>/dev/null
echo '--- HF cache refs ---'
ls /root/models/Kimi-K2.6/ | head -5
cat /root/models/Kimi-K2.6/README.md 2>/dev/null | head -5
echo '--- dir size ---'
du -sh /root/models/Kimi-K2.6 2>/dev/null
"

This command is trying to answer a specific question: where did this model come from? The assistant needs the Hugging Face repository name so it can download the model on the B300 machine. It checks three sources of information:

  1. config.json's _name_or_path field — Hugging Face transformers models typically store their source repository in this field. If set, it directly gives the HF repo name, enabling a simple huggingface-cli download <repo> command.
  2. README.md — The model card often contains tags and metadata identifying the source. The compressed-tensors tag is particularly relevant as it identifies the quantization format.
  3. Directory listing — The file names can hint at the model's origin. The presence of THIRD_PARTY_NOTICES.md and chat_template.jinja alongside config.json is typical of Hugging Face model repositories. The results are revealing but incomplete:
name_or_path: ?
quant: ?
--- HF cache refs ---
LICENSE
README.md
THIRD_PARTY_NOTICES.md
chat_template.jinja
config.json
---
tags:
- compressed-tensors
license: other
license_name: modified-mit
--- dir size ---
548G	/root/models/Kimi-K2.6

The _name_or_path field is "?" — either not set, or set to a local path that wasn't captured by the Python query. The quantization_config.quant_method query also returned "?" — the Python one-liner may have failed because the config uses a non-standard quantization config structure, or because the quantization_config key doesn't exist in the expected format. The README reveals the model uses compressed-tensors quantization (a format for storing quantized models developed by Neural Magic/FM) and carries a modified-mit license.

The assistant now knows the model uses compressed-tensors quantization but doesn't have the exact HF repo name. This means the next step will involve searching Hugging Face for the correct repository, or finding another way to identify it — perhaps by examining CT200's Hugging Face cache directory or by searching for "Kimi-K2.6 compressed-tensors" on the Hugging Face hub.

Assumptions and Knowledge Boundaries

Message [msg 11750] reveals several assumptions the assistant is making, some of which deserve scrutiny.

Assumption 1: sm_103 compatibility is likely, not certain. The assistant assumes torch 2.11.0+cu130 will work on sm_103 because it worked on sm_120. This is reasonable — CUDA is generally forward-compatible within the same architecture family — but it's not guaranteed. The assistant is implicitly planning to discover and fix any incompatibilities as they arise, which is a pragmatic approach but one that could lead to significant debugging time if the assumption is wrong.

Assumption 2: uv is the best bootstrapping method. The assistant chooses uv over alternatives (get-pip.py, apt-installing python3-pip, conda). This is a judgment call based on speed and modern tooling preferences, but it assumes uv is available for installation on this system. If the machine has restricted internet access or if the uv installation script fails, the assistant will need a fallback plan.

Assumption 3: The model source can be identified from metadata. The assistant assumes that the model's config.json or README.md will contain enough information to identify the Hugging Face repository. When this fails (the _name_or_path field is empty), the assistant will need to fall back to other methods — examining CT200's Hugging Face cache, searching the HF hub, or checking the model's Git history.

Assumption 4: Parallel execution is optimal. The assistant wants to download the model and set up the environment simultaneously. This is correct in theory, but it assumes the download tool can be started independently of the environment setup. If huggingface-cli requires Python packages that aren't yet installed, or if the download depends on libraries that are part of the environment setup, the parallelism breaks.

Assumption 5: The model is "Llama 2.6." The assistant refers to the model as "Llama 2.6" in the reasoning, but the model is actually "Kimi K2.6" — a MoE architecture from Moonshot AI, not a Llama variant. This is a minor inaccuracy in naming, but it reflects the assistant's mental model of the architecture. The actual model architecture affects how it loads and runs, so this slip could matter if the assistant makes assumptions based on Llama-specific behaviors.

The message also reveals clear knowledge boundaries:

Knowledge Flow: Inputs and Outputs

Input knowledge required to understand this message:

Conclusion: The Bridge to NVLink Deployment

Message [msg 11750] is a planning and reconnaissance message that bridges two phases of the project. It closes the PCIe Blackwell chapter — where the reproduction package was created, all fixes were documented, and the stack was validated on sm_120 — and opens the B300 NVLink chapter, where the stack will be deployed on the target hardware for the first time.

The message's significance lies not in what it accomplishes (a single SSH command that returns partial information) but in what it represents: the shift from a known, working environment to an unknown, fresh one. The assistant must now replicate the entire stack — CUDA toolkit configuration, Python environment, patched SGLang installation, model downloads, service configuration — on a machine it has never touched before, with a compute capability (sm_103) that may introduce new compatibility issues.

The partial success of the information-gathering command — finding compressed-tensors but not the repo name — sets up the next challenge: identifying the exact model source so the 548 GB download can begin. This will likely involve searching Hugging Face for "Kimi-K2.6 compressed-tensors" or examining CT200's Hugging Face cache for the repository reference.

In the broader narrative of the session, message [msg 11750] is the moment when the assistant stops looking backward at the PCIe work and starts looking forward to the NVLink deployment. It is a small message with large consequences — the decisions made here (uv for bootstrapping, parallel download, pinned dependencies) will shape the next several hours of work. The reasoning visible in this message shows an assistant operating under uncertainty, making educated guesses about compatibility, and methodically gathering the information needed to proceed. It is, in essence, the kind of planning that separates a successful deployment from a failed one: not flashy, not glamorous, but absolutely essential.