Grounding Speculative Decoding Estimates in Measured Data: A Reproducible Performance Model for B300

Introduction

In the middle of an intensive speculative decoding optimization session for the Kimi K2.6 model on Blackwell GPUs, a seemingly simple question from the user triggered one of the most methodologically instructive moments in the conversation. The user asked: "Can you estimate how fast a B300 would be at C=1 now? What if we had a stronger drafter which with ddtree at block=16 and budget=32 would hit 12-16 accept lengths?" ([msg 12038]). This was not an idle hypothetical—the team had just lost access to the B300 hardware (it had been "released" per [msg 12032]), but the architectural insights from comparing the PRO 6000 and B300 platforms were central to their roadmap.

The assistant's response to this question, embodied in message [msg 12040], is a masterclass in how to build trustworthy performance estimates for complex ML inference systems. Rather than offering a hand-wavy projection, the assistant wrote a transparent, calibrated Python estimator script and executed it, producing a concrete set of numbers grounded in real measurements. This article examines that message in depth: why it was written, the reasoning that produced it, the assumptions it encodes, and what it reveals about the art of performance modeling for speculative decoding.

The Context: Two Platforms, One Bottleneck

To understand message [msg 12040], we need to appreciate the technical landscape the assistant was navigating. The team was deploying Kimi K2.6, a Mixture-of-Experts (MoE) model with 61 transformer layers, using speculative decoding via a Draft-and-Verify (DDTree) scheme. The target hardware was the NVIDIA RTX PRO 6000 Blackwell (featuring 1.8 TB/s GDDR7 memory bandwidth) and, until recently, the B300 Blackwell datacenter GPU (with approximately 8 TB/s HBM3e bandwidth—roughly 4.5× higher).

The assistant had just completed a thorough bandwidth analysis ([msg 12037]) showing that on the PRO 6000, the MoE forward pass was hitting 67–73% of the theoretical 1.8 TB/s peak—meaning the GPU was genuinely memory-bandwidth-bound and the Marlin INT4 kernel was already efficient. The key insight was Amdahl's Law: on the PRO 6000, the MoE GEMM dominated the step time (~7 ms), so optimization efforts focused on amortizing that cost across more tokens. But on the B300, where the MoE forward would shrink to ~1.5 ms, the fixed overheads—Python dispatch, CPU tree building, attention computation, AllReduce communication—would become the dominant cost. This flipped the optimization priority: overhead elimination became the primary lever.

The user's question cut to the heart of this analysis: given the B300's vastly higher bandwidth, what throughput could they actually expect, both with the current drafter and with a hypothetical stronger one?

Why This Message Was Written: The Commitment to Reproducibility

The assistant's reasoning in [msg 12039] reveals a deliberate methodological choice. The opening line of the reasoning block states: "I need to estimate throughput for B300 with C=1, and also project what it would look like with a stronger drafter using different parameters. This is fundamentally an estimation problem, so I should build a small calculator to ground my reasoning in actual data and make the numbers transparent and reproducible."

This is a critical decision point. The assistant could have simply reasoned aloud and produced approximate numbers—many engineers would have done exactly that. Instead, the assistant chose to write a Python script that explicitly models each component of the decode step, calibrates it against two measured B300 data points, and produces a table of estimates with clear assumptions. The reasoning block shows the assistant working through the math mentally first, then pivoting: "Writing the estimator script... Build mode. I have a measured B300 C=1 anchor from the prior session, plus the measured MoE-GEMM curve and bandwidth ratio — enough to build a transparent per-step cost model. Let me write a reproducible estimator rather than hand-wave."

The phrase "Build mode" is telling—it marks a shift from analytical reasoning to tool-building. The assistant recognized that the estimation problem had enough structure and enough empirical grounding to warrant a reusable artifact, not just a one-off calculation.

What the Message Contains

Message [msg 12040] is, on its surface, a simple bash command execution:

cd /home/theuser/glm-kimi-sm120-rtx6000bw/kdtree-engine && python3 python/estimate_b300.py

The output shows three sections of the estimator's results:

Calibration (B300, measured anchors): The script reports two measured data points from prior B300 sessions: autoregressive decoding at 133 tok/s (7.52 ms per step) and DDTree with block=8 at 303 tok/s with an acceptance rate of 4.4 tokens per step (14.52 ms per step). Critically, the script then decomposes the DDTree step time into three components: MoE(M=9) at 1.99 ms, ATTN_BASE at 7.26 ms, and SPEC(block8) at 5.27 ms. This decomposition is the core of the model—it separates the cost of the target model's MoE forward pass, the attention and base overhead, and the speculative (draft + tree) overhead.

Current drafter (block8/budget8, accept ~4.4): The script shows the measured baseline (303 tok/s) and then projects what happens if the native C++ engine (being built in the kdtree-engine/ repository) cuts approximately 40% of the Python launch overhead: the step time drops to 9.51 ms and throughput rises to 463 tok/s—a 53% improvement with the same drafter, purely from overhead reduction.

Stronger drafter (block16/budget32): The output is truncated with "...", but from the follow-up message ([msg 12043]) we know the complete projection: with acceptance rates of 12, 14, or 16 tokens per step, the SGLang-style pipeline would achieve approximately 570, 670, or 765 tok/s respectively, while the native engine would reach approximately 815, 950, or 1090 tok/s.

The Decomposition Model: MoE(M) + ATTN_BASE + SPEC

The estimator's architecture reflects a deep understanding of where time goes in a speculative decoding step. The three-component decomposition is:

1. MoE(M): The time to run the target model's Mixture-of-Experts forward pass for M tokens (where M = budget + 1, the verification batch size). This scales with M and is proportional to the number of active expert weights that must be streamed from HBM. The assistant calibrated this using the measured MoE GEMM curve from the PRO 6000, scaled by the bandwidth ratio between PRO 6000 and B300.
2. ATTN_BASE: The base attention cost plus lm_head and other fixed overhead. This was derived by subtracting the MoE cost from the measured autoregressive step time (7.52 ms - MoE(M=1)). On B300, this component is surprisingly large (~7.3 ms) relative to the MoE cost (~2.0 ms), confirming the Amdahl effect.
3. SPEC(block): The speculative overhead—draft model forward pass, tree construction, and other DDTree-specific costs. This was isolated by subtracting MoE(M=9) and ATTN_BASE from the measured DDTree step time (14.52 ms - 1.99 ms - 7.26 ms = 5.27 ms). This decomposition is powerful because it allows the assistant to scale each component independently when projecting to new configurations. For the stronger drafter with block=16 and budget=32, the MoE cost grows from M=9 to M=33 (scaled using the measured MoE scaling curve), the SPEC cost grows to account for a larger draft model and more tokens, and ATTN_BASE grows slightly for the larger attention computation.

Assumptions and Their Implications

The estimator rests on several key assumptions, each worth examining:

Calibration to two B300 points is sufficient. The assistant had exactly two measured data points from the prior B300 session: autoregressive throughput and DDTree b8 throughput. From these, the three-component model is fully determined (three unknowns, two equations, plus the MoE scaling curve from PRO 6000). This is a minimal calibration, and the assistant acknowledges ~±30% uncertainty.

The MoE cost scales linearly with M. The model assumes that running the MoE forward for M=33 tokens costs the same multiple of M=9 as it did on the PRO 6000. This is reasonable because the MoE is bandwidth-bound and the weights are read once per forward pass regardless of batch size (up to the plateau), but the exact scaling depends on the Marlin kernel's behavior at different batch sizes.

The native engine can cut ~40% of overhead. This assumption is based on the observation that Python dispatch, CPU-GPU synchronization, and SGLang's pipeline overhead constitute a significant fraction of the step time. The native C++ engine being built in the repository aims to eliminate this, but the exact savings depend on implementation quality.

The stronger drafter achieves 12-16 accept length. This is the most significant assumption and, as the assistant notes, "that's a training outcome, not something the inference stack controls." The estimator projects what would happen if such a drafter existed, but actually building it is a separate (and substantial) effort.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in [msg 12039] shows a meticulous, iterative refinement process. It begins with high-level decomposition: "I need to build a per-step cost model for DDTree decode that accounts for the draft phase, tree construction, target verification forward pass, and overhead—then divide the accept length by total step time to get throughput."

The assistant then works through the numbers mentally, cross-checking against multiple data points: the PRO 6000 baseline (138.7 tok/s), the autoregressive comparison (98 tok/s on PRO 6000, 133 on B300), and the measured DDTree b8 point (303 tok/s on B300). Each mental calculation is an opportunity to verify consistency—for example, checking that the MoE cost at M=9 on B300 (~2 ms) is consistent with the PRO 6000 measurement scaled by the bandwidth ratio.

The reasoning also reveals the assistant confronting uncertainty directly. When estimating the draft model cost for block16, it notes: "the draft forward roughly doubles since block=16 versus block=8 means twice the tokens to process, though the 1.5B model is partly launch-bound so the scaling won't be perfectly linear." This is not a hand-wave—it's a reasoned judgment about where the bottlenecks lie.

Perhaps most importantly, the assistant explicitly identifies the critical constraint that the user's hypothetical scenario must contend with: "the prior finding showed that budget>8 crashed the cuda graphs on sm_103." This means the stronger drafter configuration (budget=32) is not even feasible in the current SGLang-based pipeline—it requires the native engine with owned verify kernels. This constraint is carried through to the final answer, where the assistant separates "SGLang-style pipeline" from "native engine" scenarios.

The Output Knowledge Created

Message [msg 12040] produces several forms of knowledge:

Calibrated decomposition of B300 step time. Before this message, the team knew the aggregate throughput (303 tok/s) but not how that time was distributed across components. The estimator reveals that overhead (ATTN_BASE + SPEC) accounts for ~85% of the step time, with MoE only ~15%. This is a striking finding that validates the Amdahl analysis from [msg 12037].

Quantified native engine benefit. The projection that the native engine alone could boost throughput from 303 to 463 tok/s with the same drafter gives a concrete target for the engineering effort.

Projected range for stronger drafter. Although truncated in the message, the complete projection (570-765 tok/s for SGLang pipeline, 815-1090 tok/s for native engine) provides actionable targets for drafter training.

A reusable estimation tool. The script itself (estimate_b300.py) is committed to the repository and can be rerun with different assumptions, making the analysis reproducible and extensible.

The Follow-Up and Resolution

The assistant's next actions ([msg 12041], [msg 12042]) commit the estimator script and update the documentation. The final answer ([msg 12043]) presents the complete results in a clean table, reiterates the key dynamics, and explicitly bounds the uncertainty at ~±30%. The bottom line: a 12-16 accept drafter would roughly double to triple C=1 throughput versus the current 303 tok/s baseline, landing around 0.8-1.1k tok/s single-stream once the native engine removes the overhead that currently dominates the B300 step.

Broader Significance

Message [msg 12040] exemplifies a methodological principle that extends far beyond this specific project: when estimating the performance of complex systems, build a transparent, calibrated model rather than reasoning in isolation. The estimator script is not just a calculation tool—it's a communication device that makes assumptions explicit, enables sensitivity analysis, and can be updated as new measurements become available.

For the speculative decoding community, the decomposition itself is valuable. The finding that overhead dominates on high-bandwidth hardware (B300) while MoE dominates on lower-bandwidth hardware (PRO 6000) has implications for how inference engines should be designed for different GPU tiers. On consumer or workstation GPUs (like the PRO 6000), optimizing the MoE kernel itself matters most. On datacenter GPUs (like the B300), eliminating Python overhead, moving tree construction to the GPU, and owning the verification kernels become the critical paths.

The message also demonstrates the importance of grounding projections in measured data. The two calibration points (AR 133 tok/s and DDTree 303 tok/s) anchor the entire model, preventing the estimates from floating into pure speculation. When the assistant says the native engine could reach ~460 tok/s with the current drafter, that number is not pulled from thin air—it's derived from a decomposition calibrated to actual hardware measurements.

Conclusion

Message [msg 12040] is, at first glance, a simple bash command execution showing the output of a Python script. But it represents the culmination of a disciplined reasoning process that chose reproducibility over hand-waving, transparency over convenience. The assistant built a three-component model of speculative decoding step time, calibrated it to two measured data points, and used it to project performance across two scenarios (current drafter and stronger drafter) and two pipeline architectures (SGLang and native engine).

The result is a set of estimates that are not only useful for decision-making but also trustworthy—because the assumptions are explicit, the calibration is grounded, and the tool is available for anyone to inspect and modify. In a field where performance claims are often opaque and difficult to verify, this approach sets a standard worth emulating.