The Strategic Question: Estimating B300 Performance Under Speculative Decode

In the middle of a deep engineering session building a native C/C++/CUDA DDTree inference engine for the Kimi K2.6 model, the user asks a forward-looking question that shifts the conversation from what is to what could be:

Can you estimate how fast a B300 would be at C=1 now? What if we had a stronger drafter which with ddtree at block=16 and budget=32 would hit 12-16 accept lengths?

This single message, <msg id=12038>, is a pivotal moment. It is not a request for debugging, a status check, or a configuration change. It is a strategic estimation question that forces the assistant to synthesize everything it has learned across dozens of prior messages — measured kernel latencies, bandwidth utilization curves, Amdahl's law tradeoffs, drafter acceptance behavior, and the hardware specifications of two very different GPU architectures — into a grounded, reproducible projection. The question is deceptively simple, but answering it requires a rigorous cost model built from first principles.

Why This Message Was Written: The Motivation and Context

The user's question emerges from a specific moment in the conversation. Just two messages earlier, at <msg id=12036>, the assistant had committed an analysis document (docs/B300_HBM_NVLINK_NOTES.md) that established a critical finding: the PRO 6000's INT4 MoE GEMM runs at approximately 67–73% of its 1.8 TB/s GDDR7 peak bandwidth, meaning the kernel is already efficient and the remaining headroom lies in amortization and overhead removal rather than raw bandwidth. The assistant then explained that B300's HBM3e at roughly 8 TB/s would be about 4.5× faster for the MoE forward pass, which flips the bottleneck from memory bandwidth to fixed overhead — CPU tree building, Python dispatch, AllReduce communication.

The user absorbs this analysis and immediately asks the natural next question: given what we now know, what would the actual throughput be? This is not idle curiosity. The user is making a strategic decision about where to invest engineering effort. The question contains two scenarios that represent very different development paths:

  1. The "now" scenario: B300 at C=1 with the current drafter. This tells the user what they would get by simply porting the existing system to faster hardware — a pure hardware upgrade play.
  2. The "what if" scenario: B300 with a stronger drafter at block=16 and budget=32, achieving 12–16 accept lengths. This tells the user whether investing in drafter training and larger tree configurations would yield proportional returns on B300's faster memory subsystem. The user's assumptions are worth examining. They assume that a stronger drafter achieving 12–16 accept lengths is feasible — a roughly 3–4× improvement over the current ~4.4 accept length. They also assume that the DDTree framework can support block=16 and budget=32, which is not trivial: the prior session had documented that budget>8 crashed CUDA graphs on the sm_103 architecture. The user may be implicitly trusting that the native engine being built will resolve that limitation. They also assume that B300's performance characteristics are known well enough to estimate from PRO 6000 measurements, which is a reasonable engineering extrapolation but carries uncertainty.

The Thinking Process: Building a Reproducible Cost Model

The assistant's reasoning, visible in <msg id=12039>, is a masterclass in disciplined estimation. The first instinct is not to hand-wave but to build a calculator — a Python script that documents every assumption and makes the numbers reproducible. The reasoning proceeds through several phases.

Phase 1: Anchoring to measured data. The assistant starts with two concrete measurements: the PRO 6000 baseline at 138.7 tok/s with C=1 and block=8, and the prior session's B300 measurement at 303 tok/s with the same configuration. These are not theoretical — they are real numbers from real hardware. The B300 point (303 tok/s, ~4.4 accept length) gives a per-step time of roughly 14.5 ms, which becomes the calibration anchor.

Phase 2: Decomposing the step time. The assistant breaks the DDTree decode step into its constituent costs: the MoE GEMM forward pass (the dominant compute), the draft model forward pass, tree construction on CPU, attention computation, AllReduce communication, Python dispatch overhead, and sampling. Using the PRO 6000's measured per-layer MoE latencies and scaling them through 60 layers under TP-8, the assistant estimates that the MoE component for M=9 (budget=8, plus the target token) costs about 8.85 ms on PRO 6000. Scaling by the 4.5× bandwidth ratio gives ~1.97 ms on B300. The remaining ~12.5 ms of the 14.5 ms step time must be draft forward, attention, and overhead.

Phase 3: Scaling to the stronger drafter. For the hypothetical scenario (block=16, budget=32, accept length 12–16), the MoE cost grows because the verify forward pass now processes M=33 tokens instead of M=9. Using the measured MoE latency curve from PRO 6000 (M=9 at 8.85 ms, M=32 at 23.8 ms), the assistant projects M=33 on B300 at roughly 5.3 ms. The draft forward roughly doubles since it processes 15 predictions instead of 7. The attention cost grows modestly with the larger query batch. The assistant estimates the non-MoE portion grows from ~12.5 ms to ~15 ms, giving a total step time around 20.4 ms.

Phase 4: Computing throughput. With accept lengths of 12–16 tokens per step and a step time of ~20.4 ms, the projected throughput lands at 575–770 tok/s. If a native engine can cut overhead from 12.5 ms to 8 ms, the projection rises to 850–980 tok/s — a substantial jump from the 303 tok/s baseline.

What Makes This Message Interesting

The message reveals something important about how the user thinks. They are not asking "is B300 faster?" — they already know it is. They are asking for a quantified projection that accounts for the specific interaction between speculative decoding's unique compute pattern and B300's memory hierarchy. The DDTree verify step processes a batch of tokens (budget+1) through the full model, which is fundamentally different from autoregressive decode (batch size 1). On PRO 6000, the MoE forward dominates because the bandwidth ceiling is relatively low. On B300, the MoE forward shrinks dramatically, and the fixed costs that were previously negligible become the primary constraint. The user's question implicitly tests whether the assistant understands this inversion.

The question also reveals the user's mental model of the drafter's role. By specifying "block=16 and budget=32" alongside "12-16 accept lengths," the user shows they understand the relationship between tree structure and acceptance rate — larger trees can capture longer speculative paths, but only if the drafter is accurate enough to propose useful continuations. The 12-16 target is ambitious: it represents roughly 3-4× the current acceptance rate, which would require a substantially better-trained drafter.

Input Knowledge Required

To fully understand this message, the reader needs to know several things from the preceding conversation:

Output Knowledge Created

This message triggers the creation of a reproducible estimation framework. The assistant writes python/estimate_b300.py — a Python script that encodes the cost model with explicit parameters, measured anchor points, and scaling rules. This is not a one-off answer; it is a tool that can be re-run with different assumptions as more data becomes available. The output also includes a clear comparison table showing both scenarios with uncertainty bounds, and a discussion of the key levers: overhead reduction in the native engine, the cuda-graph budget limit, and the drafter's acceptance rate.

The deeper output is a refined understanding of where the DDTree system sits on the cost curve. The assistant's analysis confirms that the system is behaving correctly — there is no hidden bug, no missed optimization. The 303 tok/s on B300 is real, and the path to 600-800+ tok/s requires either a better drafter or a native engine that eliminates the overhead that Amdahl's law has exposed.

Mistakes and Uncertainties

The estimation necessarily carries uncertainty. The B300 measurement (303 tok/s) came from a prior session with a different software stack (SGLang with CUDA graphs), not the native engine under development. The overhead breakdown is inferred rather than directly measured — the assistant does not have a profile that separates draft forward time from attention from Python dispatch. The drafter cost at block=16 is estimated as roughly 1.5-2× the block=8 cost, but this depends on whether the drafter is launch-bound or compute-bound at these batch sizes. And the fundamental assumption — that a stronger drafter can achieve 12-16 accept lengths — is untested.

The assistant handles these uncertainties responsibly by building the estimator with explicit parameters and presenting ranges rather than single numbers. The script is designed to be updated when actual measurements become available.

Conclusion

The user's question at <msg id=12038> is a turning point in the conversation. It moves the discussion from "what have we built and how does it perform" to "what would it take to reach the next level." The answer requires synthesizing kernel microbenchmarks, bandwidth utilization curves, Amdahl's law, and speculative decoding theory into a coherent projection. The assistant's response — building a reproducible estimator rather than hand-waving — reflects the engineering discipline that characterizes the entire session. The numbers that emerge (575-770 tok/s with a stronger drafter, 850-980 tok/s with a native engine) become the target that the rest of the development effort will aim for.