The Throughput Lever: Quantifying the MoE GEMM and Mapping the Road to B300

The previous chunk had delivered a complete native C/C++/CUDA DDTree inference engine, validated token-for-token against a numpy golden reference on an RTX 5070 Ti. But an inference engine validated on a development GPU is a proof-of-concept, not a product. The work in this chunk bridges that gap with a vengeance: deploying to the actual 8× RTX PRO 6000 production target, running the full validation suite on real Blackwell silicon, benchmarking every component at K2.6-relevant shapes, and then — crucially — quantifying the single most important performance lever in the entire stack: the INT4 Marlin MoE GEMM.

What emerges is a story of honest engineering measurement, Amdahl's law in action, and a clear roadmap to B300-class hardware that reveals the native engine's true value proposition.

Deployment and Validation on the PRO 6000

The first task was getting the entire stack onto the CT200 production box — an 8× RTX PRO 6000 Blackwell system running CUDA 13.0. This was non-trivial: CT200 had no cmake, so the build had to use the nvcc-direct pipeline developed earlier ([msg 11973]). The rsync, build, and test sequence completed cleanly ([msg 11984]), with all 27 kernel unit tests passing bit-exact on the PRO 6000's sm_120 architecture ([msg 11985]). The native engine's autoregressive and DDTree greedy-exact invariant held with a maximum logit difference of ~6e-6 — the same precision as on the dev card.

But the deployment uncovered a real bug. The tree_accept kernel crashed during its benchmark run with an illegal memory access ([msg 11980]). The root cause was subtle: the benchmark generated random next_token and next_sibling arrays, which could form cycles. When the kernel's tree-walk followed these cyclic pointers, the path[] array grew unbounded, eventually writing past its allocation. The fix was a defensive safety bound — a valid accepted path can never visit more nodes than the tree contains, so capping the walk at q_len prevents the overrun without affecting correctness on real trees ([msg 11980]). This is a textbook example of why benchmarks need to be as carefully validated as the code they measure.

The Benchmarks That Matter

With the crash fixed and the suite green, the assistant ran three benchmark suites on the PRO 6000, each answering a specific question.

GPU Tree Builder vs. SGLang's CPU Heapq

The headline comparison was the GPU tree builder against SGLang's actual build_ddtree_tree_from_topk + compile_ddtree_retrieve — the real CPU code path used in production. The results were dramatic ([msg 11989]):

Kernel Microbenchmarks at K2.6 Shapes

The kernel microbenchmarks confirmed that tree building (12–144 µs) and tree acceptance (2–7 µs) are both cheap and batch-independent ([msg 11986]). The verify_attn kernel told a different story: the naive implementation was memory-bound on long prefixes, taking 0.2–8.7 ms depending on prefix length and query count. This was expected — the production architecture documented in the plan explicitly calls for reusing FlashMLA for the long prefix attention and applying the custom masked kernel only to the small tree tail. The benchmark confirmed the design decision rather than revealing a flaw.

Live SGLang DDTree Baseline

The service baseline measurement established the number to beat ([msg 11992]):

The Honest Finding

The most important result was also the most sobering. At the target 1–10 streams, a verify step is dominated by the 1T INT4 MoE forward pass (~80–90 ms/step, HBM-bound). The CPU tree build is only ~1% of the step time at these concurrency levels. So the 474× tree-build win — while impressive as a kernel benchmark — would translate to only a ~1–2% end-to-end improvement if patched into the current SGLang service ([msg 11993]).

This is why the assistant deliberately chose not to do an in-place SGLang integration. The risky service patch would have yielded negligible benefit. Instead, the assistant quantified the bottleneck honestly and documented that the real throughput lever is the INT4 Marlin MoE GEMM fed a dense (streams × q_len) batch — the documented Phase-3 priority.

Phase 3: The INT4 Marlin MoE Throughput Lever

With the baseline established, the assistant turned to the single component that determines whether the native engine can beat the baseline: the INT4 Marlin MoE grouped GEMM — the kernel K2.6 actually uses (moe_wna16_marlin_gemm via fused_marlin_moe).

Building the Benchmark

Constructing valid Marlin-packed weights for benchmarking was non-trivial. The K2.6 model uses INT4 quantization with group size 32 via the compressed-tensors format ([msg 12003]). Rather than hand-crafting the packed layout — error-prone given the fiddly Marlin permutation requirements — the assistant used SGLang's own gptq_marlin_moe_repack function to transform randomly-generated INT4 GPTQ-format weights into the correct Marlin layout ([msg 12005]). This guaranteed the kernel would see exactly the same weight format it processes in production.

The benchmark required stopping the live service to free a GPU — each of the 384 experts at K2.6 scale (hidden 7168, intermediate 2048) requires ~8.5 GB of Marlin-repacked weights, with peak memory during repacking reaching ~16 GB ([msg 12006]). The service was stopped, the benchmark run on a freed GPU with 97 GB available, and the service restarted afterward ([msg 12007], [msg 12011]).

The Result: Two Regimes

The benchmark produced a clean, interpretable curve ([msg 12010], [msg 12023]):

| M (tokens/layer) | ms/call | µs/token | Regime | |---|---|---|---| | 1 (AR C=1) | 0.151 | 150.6 | ~8/384 experts active | | 9 (verify b8) | 1.18 | 131 | Weight-streaming-bound | | 90 (10× b8) | 5.19 | 57.7 | | | 256 | 7.02 | 27.4 | ~all experts active | | 512 | 7.01 | 13.7 | Plateau | | 1024 | 7.02 | 6.85 | Plateau |

Two distinct regimes emerge:

  1. Weight-streaming-bound (M < ~256): Time grows with the number of active experts. Each token activates ~8 of 384 experts via top-8 routing, and the kernel must stream those experts' weights from HBM. The per-token cost is high because each token pays the weight-streaming cost for its small expert subset.
  2. Compute-bound plateau (M ≥ ~256): All 384 experts' weights are streamed once per layer call. Additional tokens within that call are nearly free — they reuse the already-loaded weights. The per-token cost drops 4× (27 → 6.85 µs) while the total layer time stays flat at ~7 ms. The bandwidth utilization analysis confirmed the kernel is already efficient: the MoE GEMM achieves ~67–73% of the PRO 6000's 1.8 TB/s GDDR7 peak bandwidth ([msg 12029]). There is little headroom left in the kernel itself.

What It Means for the Engine

The verify batch is M = streams × q_len (where q_len = budget + 1). To amortize the dominant expert-weight read, the engine must push M into the plateau region (≥256). The sweet spot is big trees (budget 32–64) + 8–16 streams — for example, budget 32 (q_len=33) × 8 streams = M≈264 at plateau entry, or ×16 streams = M≈528 well into the plateau.

Critically, single-stream (C=1) can't reach the plateau — it stays weight-streaming-bound regardless of budget. This refines the plan: speculation's win at C=1 comes from committing many tokens per sparse forward (acceptance rate), while the bandwidth amortization win is a multi-stream story.

The B300 Analysis: When the Bottleneck Flips

The user's question about B300 hardware — with its ~8 TB/s HBM3e bandwidth, ~4.5× the PRO 6000's 1.8 TB/s GDDR7 — triggered a deeper analysis grounded in the measured data ([msg 12028]).

Amdahl's Law in Action

The PRO 6000 MoE GEMM runs at ~67–73% of peak bandwidth. On B300, the same kernel would achieve similar utilization, compressing the MoE forward pass from ~7 ms to ~1.5 ms at the plateau. The sharp consequence: the fixed overheads become the dominant cost.

On the PRO 6000, the MoE forward is ~80% of the step time. On B300, it would be ~20%. The remaining 80% comes from CPU tree building, Python/launch overhead, AllReduce, attention, and draft model forward — all of which stay roughly constant regardless of HBM bandwidth.

This flips the optimization priority. On the PRO 6000, the lever is bandwidth amortization (reach the M≈256 plateau). On B300, the lever is overhead elimination — and everything the native engine already does becomes proportionally more valuable:

  1. Native C++ loop (no Python): Launch overhead is a bigger slice when the MoE is 1.5 ms.
  2. GPU tree build (6–474× vs CPU): The CPU heapq at 6–21 ms/step could be >50% of a fast B300 step.
  3. Owned verify-attn + tree kernels: These directly fix the two B300 blockers from the prior session — the sm_103 CUDA graph crash at budget >8 and the MLA-absorb CUBLAS limit — enabling budget 16–32 at graph speed.
  4. NVLink NV18 + NVLS: Make TP-8 AllReduce cheap (EP's PCIe edge disappears), and 275 GB HBM allows more concurrent streams.

The Throughput Estimate

Using a transparent per-step cost model calibrated to two measured B300 points (AR 133 tok/s, DDTree b8 303 tok/s at accept 4.4), the assistant built a reproducible estimator (estimate_b300.py) that projects the impact of a stronger drafter ([msg 12040], [msg 12043]).

For a stronger drafter achieving accept 12–16 with block=16 and budget=32:

| Scenario | Step Time | Tok/s | |---|---|---| | SGLang-style pipeline, accept 12/14/16 | 20.9 ms | ~570 / 670 / 765 | | Native engine (−40% overhead), accept 12/14/16 | 14.7 ms | ~815 / 950 / 1090 |

The key dynamics: at C=1, budget 32 pays a ~3 ms MoE tax (M=33 vs M=9: 5.3 vs 2.0 ms) to commit ~3× more tokens per step. Acceptance grows faster than cost. However, C=1 stays weight-streaming-bound (M=33 ≪ the M≈256 MoE plateau), so the win comes from acceptance rate, not bandwidth amortization. The plateau lever remains a multi-stream (C≥8) story.

The uncertainty is ~±30%, dominated by the draft model's cost scaling at block 16 and whether the drafter actually achieves 12–16 accept — that's a training outcome, not something the inference stack controls.

The Broader Picture: From Component Benchmarks to System Architecture

This chunk represents a maturation of the project from "does it work?" to "how fast is it, and why?" The component benchmarks — tree build vs. SGLang CPU, kernel microbenchmarks, live service baseline, INT4 Marlin MoE curve — form a coherent picture of where the bottlenecks are and what levers exist to address them.

The honest finding that the GPU tree build, despite its 474× speedup, would only yield ~1–2% end-to-end improvement in the current service is a textbook example of why system-level measurement matters. A less disciplined engineer might have patched the kernel in, declared victory, and moved on. Instead, the assistant quantified the actual bottleneck (the MoE forward pass), characterized its performance curve, and documented the path forward.

The B300 analysis reveals that the native engine's value proposition shifts with the hardware. On the PRO 6000, it's about bandwidth amortization — pushing M into the plateau. On B300, it's about overhead elimination — stripping away the Python/CPU costs that become dominant when the MoE shrinks to 1.5 ms. The same engine, the same kernels, but the reason they matter changes with the hardware's bandwidth.

What Was Committed

The work in this chunk produced several lasting artifacts:

Conclusion

This chunk transformed the native DDTree inference engine from a validated prototype into a characterized system with quantified performance bounds. The key findings — the MoE GEMM's two-regime curve, the bandwidth efficiency measurement, the Amdahl-flip on B300 — are not just numbers; they are design guidance for the remaining Phase-3 work. The engine's architecture is sound; the path to beating the 138/517 tok/s baseline is clear; and the next steps are integration plumbing, not architectural risk.

The work also demonstrates a philosophy of measurement that separates genuine system understanding from wishful thinking. Every claim is grounded in a benchmark run on the actual target hardware. Every optimization priority is backed by a quantified bottleneck. And every projection is accompanied by its uncertainty bounds. That is the discipline that turns a prototype into a production path.