The NVLink Pivot: Planning the B300 Deployment for Kimi K2.6 with DDTree

Introduction

In the sprawling, multi-week journey of deploying the Kimi K2.6 model with DFlash speculative decoding across diverse GPU architectures, few moments are as pivotal as the one captured in message 11748. This message, written by the AI assistant in an opencode coding session, represents a strategic inflection point: the transition from a PCIe-bound Blackwell setup (RTX PRO 6000, SM120) to a fully NVLink-connected B300 SXM6 machine (SM100). The message is not a tool execution, a bug fix, or a benchmark result—it is a planning artifact, a moment of reconnaissance and strategy formulation that sets the trajectory for the next phase of the project.

To understand why this message matters, one must appreciate the hardware landscape the assistant has been navigating. The previous several segments of the conversation were dominated by work on 8× RTX PRO 6000 Blackwell GPUs connected via PCIe. On that platform, the assistant discovered a critical bottleneck: AllReduce operations across PCIe were devastating for tensor parallelism (TP8), achieving only ~26 tok/s for single requests. The solution was expert parallelism (EP8), which eliminated the cross-GPU communication on MoE layers and boosted single-request throughput to ~65 tok/s. This was a hardware-imposed architectural decision—the PCIe bus simply could not sustain the bandwidth demands of TP8's AllReduce.

Now, the user has provided access to a B300 SXM6 machine, and the assistant's reconnaissance (in the message immediately preceding this one, msg 11747) reveals a dramatically different beast: 8× B300 GPUs with 275 GB each (2.2 TB total VRAM), full NVLink interconnect (NV18, ~900 GB/s), CUDA 13.0, 2 TB RAM, and 1.3 TB free disk. This is the NVLink target the project was always meant to run on. Message 11748 is the assistant's first strategic response to this new reality.

The Message: A Planning Manifesto

Let us examine the message in full. The assistant begins with an extended reasoning block that reveals its analytical process:

## Agent Reasoning This is an incredibly powerful machine — 8 GPUs with 2.2 TB of total VRAM and full NVLink connectivity between all of them for massive bandwidth, plus CUDA 13.0 already set up on Ubuntu with 240 cores.

The B300 uses SM100 compute capability, not SM120, so I need to verify the torch and sglang builds support that architecture properly. With 275 GB per GPU, K2.6 in INT4 will fit comfortably on just 2-3 GPUs with plenty of KV cache headroom across all 8, and the NVLink should make tensor parallelism across 8 GPUs extremely fast compared to the PCIe setup where expert parallelism won out.

My approach is to set up the environment with the right dependencies for SM100, download the model and drafter, apply the DDTree patches, then benchmark — especially comparing TP8 performance now that NVLink eliminates the AllReduce bottleneck. First I need to confirm whether the existing torch 2.11.0+cu130 and sglang stack will work on B300, since torch should have SM100 support but I want to verify FlashInfer compatibility too.

Let me start by checking the actual compute capability, whether there's an existing Python environment, and whether I need to download the 548 GB model on this fresh machine.

Then, in the response body, the assistant summarizes the machine's capabilities and lays out a structured plan via a todowrite directive:

Outstanding machine: 8× B300 SXM6, 275 GB each (2.2 TB total VRAM), full NVLink (NV18 ~900 GB/s), CUDA 13, 2 TB RAM, 1.3 TB free disk. This is the NVLink target — TP8 should shine here (no PCIe AllReduce penalty). B300 is SM100, not SM120, so FlashInfer attention may even work natively. Let me set up a deployment plan and check key details:

[todowrite] {"todos":[{"content":"Inventory B300 machine (done: 8xB300 275GB NVLink, CUDA13, SM100)","priority":"high","status":"completed"},{"content":"Check compute capability + existing python/venv/tooling on B300","priority":"high","status":"in_progress"},{"content":"Set up venv + install pinned stack (torch/sglang/sgl-kernel/flashinfer)","priority":"high","status":"pending"},{"content":"Deploy DDTree patches to B300 venv","status":"pending","priority":"high"}...}]}

The todowrite is truncated in the excerpt, but the visible tasks reveal a clear, methodical plan: inventory (done), check tooling (in progress), set up environment (pending), deploy patches (pending). This is the assistant's standard operating procedure for onboarding a new machine.

Why This Message Was Written

The message exists because the project has reached a hardware transition point. The user's simple statement—"B300 Machine ready: ssh root@86.38.182.109"—triggered a cascade of reconnaissance and planning. The assistant had just finished creating a comprehensive reproduction package at /data/dflash/k26-ddtree-repro/ designed specifically for NVLink hardware. Now, the target machine had arrived.

The deeper motivation is architectural validation. Throughout the PCIe phase, the assistant and user had been operating under a hypothesis: that NVLink would fundamentally change the parallelism calculus, making TP8 viable and potentially superior to EP8. The PCIe experiments had proven that EP8 was the only sensible choice on that hardware, but the entire DDTree speculative decoding work was designed with NVLink in mind. The reproduction package, the patched SGLang files, the benchmark harness—all of it was built for this moment. Message 11748 is the bridge between preparation and execution.

There is also an element of risk assessment. The B300 uses SM100 compute capability, whereas the RTX PRO 6000 uses SM120. This architectural difference means that the entire software stack—PyTorch, SGLang, FlashInfer, sgl-kernel—must be verified for SM100 compatibility. The assistant's reasoning explicitly flags this: "I need to verify the torch and sglang builds support that architecture properly." This is not a trivial concern; earlier in the project, FlashInfer had rejected SM120 outright, requiring a custom build. SM100 could present similar or different challenges.

How Decisions Were Made

The decision-making in this message is primarily about prioritization and sequencing. The assistant has a finite set of unknowns about the B300 machine, and it must decide what to check first and in what order to act.

The first decision is to prioritize architecture verification over everything else. Before downloading the 548 GB model, before deploying patches, before running benchmarks, the assistant needs to know whether the software stack even runs on SM100. This is sound engineering: validate the foundation before building on it.

The second decision is about parallelism strategy. The assistant immediately hypothesizes that TP8 will be the winner on NVLink: "TP8 should shine here (no PCIe AllReduce penalty)." This is a direct carryover from the PCIe analysis. On PCIe, TP8's AllReduce was the bottleneck. On NVLink with ~900 GB/s bandwidth, that bottleneck disappears. The assistant is implicitly deciding to prioritize TP8 benchmarking over EP8, though it will likely compare both.

The third decision is about model placement. The assistant notes that "K2.6 in INT4 will fit comfortably on just 2-3 GPUs with plenty of KV cache headroom across all 8." This suggests a decision to use all 8 GPUs for the deployment (TP8 or EP8), not just the minimum needed to hold the model weights. The reasoning is that the extra GPUs provide KV cache capacity for high-concurrency serving, which was a key metric in the PCIe benchmarks.

Assumptions Embedded in the Message

Every planning message rests on assumptions, and this one is no exception. The assistant makes several notable assumptions:

That SM100 is well-supported by the existing software stack. The assistant writes "torch should have SM100 support" and "FlashInfer attention may even work natively." These are optimistic assumptions. In practice, SM100 (Blackwell Ultra) is a relatively new architecture, and support in PyTorch, FlashInfer, and SGLang may be partial or buggy. The assistant's own history—having fought through SM120 FlashInfer issues—should caution against assuming smooth sailing.

That NVLink eliminates the TP8 bottleneck entirely. While NVLink dramatically reduces communication costs, it does not eliminate them. AllReduce still requires synchronization and data movement. The assistant's framing ("no PCIe AllReduce penalty") is slightly oversimplified. The real question is whether NVLink bandwidth is sufficient to make TP8's communication overhead negligible compared to compute time. This is an empirical question that only benchmarking can answer.

That the reproduction package is directly applicable. The assistant plans to "deploy DDTree patches to B300 venv" using the patches developed for SM120. But SM100 may require different kernel configurations, different CUDA graph capture behavior, or different attention backend settings. The patches may need adaptation.

That the model download is straightforward. The assistant flags the 548 GB model size but assumes it can be downloaded to the fresh machine. With 1.3 TB free disk, space is adequate, but download time (even at 575 MiB/s as seen on CT200) could be ~15 minutes. More importantly, the model must be compatible with the SM100 architecture—INT4 quantization schemes may differ between GPU generations.

Mistakes and Incorrect Assumptions

While the message is generally sound, there are potential issues worth examining:

The FlashInfer optimism may be misplaced. The assistant assumes FlashInfer attention "may even work natively" on SM100 because it's a different architecture from SM120. But FlashInfer's SM120 rejection was due to missing compute capability checks in the JIT compilation path, not a fundamental architectural incompatibility. SM100 could trigger similar or different JIT failures. In fact, later in the conversation (as seen in chunk 2 of the segment summary), the assistant encounters a "Triton JIT compilation issue (missing Python.h)" on the B300, confirming that the software stack was not entirely smooth.

The assumption that existing torch 2.11.0 supports SM100. At the time of this message, PyTorch 2.11.0's support for Blackwell Ultra (compute capability 10.0) was still evolving. The assistant would later need to verify this empirically. The message's phrasing—"torch should have SM100 support"—is a hypothesis, not a confirmed fact.

The model placement assumption. The assistant assumes K2.6 INT4 fits on 2-3 GPUs, but this depends on the exact quantization scheme and the memory overhead of SGLang's KV cache management. With 275 GB per GPU, even the full 548 GB model fits on 2 GPUs with room to spare, but the assistant's estimate of "2-3 GPUs" is conservative enough to be safe.

Input Knowledge Required

To fully understand this message, one needs substantial context from the preceding conversation:

The PCIe parallelism saga. The assistant had spent days benchmarking TP8, PP8, EP8, and EP4 on the RTX PRO 6000, discovering that EP8 was the only viable strategy due to PCIe AllReduce bottlenecks. This knowledge directly informs the B300 planning.

The DDTree implementation. The assistant had implemented DDTree (Dynamic Draft Tree) speculative decoding within SGLang, involving three bug fixes (cuda-graph sizing, triton custom-mask, temperature support) and extensive benchmarking. The reproduction package at /data/dflash/k26-ddtree-repro/ contains all these artifacts.

The SM120 vs SM100 distinction. Earlier in the project, the assistant discovered that the RTX PRO 6000 uses SM120 compute capability, which FlashInfer rejected. This led to a workaround involving the triton attention backend. The B300's SM100 architecture is a different beast, requiring separate verification.

The project's overall goal. The user is building a custom inference stack for the PRO6000 box, and the DDTree findings report (written later in the conversation) outlines a roadmap for a C/C++/CUDA stack. The B300 benchmarks are meant to validate the DDTree approach on ideal hardware before committing to the custom stack.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

A verified hardware inventory. The assistant confirms that the B300 machine has 8 GPUs with NVLink, CUDA 13.0, 2 TB RAM, and 1.3 TB free disk. This is the baseline for all subsequent work.

A prioritized action plan. The todowrite structure encodes a clear sequence: verify tooling → set up environment → deploy patches → download models → benchmark. This plan guides the next several rounds of the conversation.

An architectural hypothesis. The message explicitly states the hypothesis that TP8 will outperform EP8 on NVLink, which is a testable claim that will be validated (or refuted) by subsequent benchmarks.

A risk assessment. The assistant flags SM100 compatibility as the primary risk factor, setting expectations for potential issues.

The Thinking Process

The reasoning block in this message is particularly revealing of the assistant's cognitive process. Let us trace the logic step by step:

Assessment of raw capability: "8 GPUs with 2.2 TB of total VRAM and full NVLink connectivity... CUDA 13.0 already set up on Ubuntu with 240 cores." The assistant is impressed but grounded—it states the facts without hyperbole.
Architecture recognition: "B300 uses SM100 compute capability, not SM120." This is a critical distinction. The assistant immediately recognizes that the software stack must be verified for this architecture.
Model fit analysis: "K2.6 in INT4 will fit comfortably on just 2-3 GPUs with plenty of KV cache headroom across all 8." This is a quick mental calculation based on the 548 GB model size and 275 GB per GPU.
Parallelism re-evaluation: "NVLink should make tensor parallelism across 8 GPUs extremely fast compared to the PCIe setup where expert parallelism won out." This is the key insight—the assistant is carrying forward the lesson from PCIe and applying it to the new hardware.
Strategy formulation: "My approach is to set up the environment with the right dependencies for SM100, download the model and drafter, apply the DDTree patches, then benchmark." This is a classic plan-do-check-act cycle.
Risk identification: "First I need to confirm whether the existing torch 2.11.0+cu130 and sglang stack will work on B300." The assistant identifies the primary unknown and prioritizes it.
Optimistic hypothesis: "FlashInfer attention may even work natively." This is the most speculative statement in the reasoning. It reflects hope that SM100 will be better supported than SM120, but it is not yet evidence-based. The thinking process is methodical and structured, characteristic of an experienced systems engineer approaching a new hardware platform. The assistant does not jump to execution; it plans, prioritizes, and identifies risks before acting.

Conclusion

Message 11748 is a quiet but crucial moment in the opencode session. It contains no dramatic bug fix, no breakthrough benchmark result, no elegant code change. Instead, it captures the moment when a project pivots from one hardware platform to another, carrying forward months of learning and applying it to a new context. The assistant's reasoning reveals a deep understanding of the interplay between hardware architecture and inference performance—the PCIe lessons, the SM120 struggles, the DDTree implementation—all synthesized into a coherent plan for the B300.

This message exemplifies the kind of strategic thinking that separates effective AI-assisted development from mere task execution. The assistant does not simply react to the user's provision of a new machine; it contextualizes it within the project's history, identifies risks, formulates hypotheses, and creates a structured plan. The todowrite is not just a to-do list; it is a decision record that encodes the assistant's best understanding of what needs to happen and in what order.

For a reader following the project, this message is the signpost that says: "We are about to find out if everything we've built works on the hardware it was designed for." The answer, as the subsequent chunks reveal, is a complex mix of triumph and tribulation—CUDA graph bugs, Triton JIT issues, and ultimately, 2.15× speedups over the autoregressive baseline. But all of that flows from the planning laid down in this single, unassuming message.