The B300 Throughput Estimator: When Overhead Becomes the Bottleneck

In a coding session centered on speculative decoding for large language models, one message stands out as a masterclass in rigorous performance estimation. Message [msg 12043] is the assistant's response to a user question about how fast a B300 GPU would run the DDTree (Draft-Driven Tree) speculative decoding system, and what a hypothetical stronger drafter could achieve. Rather than offering a hand-wavy guess, the assistant built a transparent, calibrated estimator script, decomposed the per-step costs into measurable components, and delivered a nuanced answer that reveals a fundamental shift in bottleneck as hardware gets faster.

This article examines that message in depth: the reasoning that motivated it, the decisions made in constructing the estimator, the assumptions baked into the model, and the knowledge it creates. It is a standalone analysis that assumes no prior familiarity with the conversation.

The Context That Produced This Message

To understand why message [msg 12043] was written, one must understand the broader arc of the session. The assistant had been working for weeks on a custom C/C++/CUDA DDTree inference engine for the Kimi K2.6 language model, deployed on an 8× RTX PRO 6000 Blackwell workstation. DDTree is a form of speculative decoding: a small "drafter" model proposes multiple candidate token sequences organized as a tree, and the large "target" model verifies them in a single batched forward pass, accepting the longest valid prefix. This technique trades a small overhead in compute for potentially large reductions in the number of expensive target model calls.

The assistant had already built a complete native engine with custom CUDA kernels for tree building, tree-verify attention, and token acceptance. They had benchmarked the system extensively on the PRO 6000, measuring the MoE (Mixture-of-Experts) GEMM kernel at approximately 67–73% of the 1.8 TB/s GDDR7 memory bandwidth — confirming the workload was genuinely HBM-bound and the kernel was already efficient.

Then the conversation shifted. The user asked about the B300, a higher-end datacenter GPU that was no longer accessible (it had been "released," meaning deallocated). The assistant wrote an analytical document in [msg 12033] analyzing how B300's ~8 TB/s HBM3e bandwidth would change the bottleneck profile. The key insight was Amdahl's law: when the MoE forward pass shrinks from ~7 ms to ~1.5 ms, the fixed overhead costs (CPU tree building, Python dispatch, launch overhead, AllReduce) become the dominant fraction of each step. The native engine's overhead-elimination work becomes more valuable on faster hardware, not less.

That analysis set the stage. Then came the user's question that directly prompted message [msg 12043]:

"Can you estimate how fast a B300 would be at C=1 now? What if we had a stronger drafter which with ddtree at block=16 and budget=32 would hit 12-16 accept lengths?"

This is the question the assistant answers in the subject message. The user wants two things: (1) a concrete throughput number for the current system on B300, and (2) a projection for an improved system with a larger draft tree and a better-trained drafter.

Why the Estimator Was Built: The Reasoning Process

The assistant's thinking, visible in the reasoning block preceding the message, reveals a deliberate methodological choice. Rather than producing a single number, the assistant decided to "build a small calculator to ground my reasoning in actual data and make the numbers transparent and reproducible." This is a crucial decision that shapes everything that follows.

The reasoning traces through several steps. First, the assistant anchors to the measured B300 point from a prior session: 303 tok/s at C=1 with the block=8 drafter achieving approximately 4.4 tokens accepted per step. This is a real measurement, not an estimate. Second, the assistant decomposes the 14.5 ms per-step time into components: MoE GEMM at M=9 (the verify batch processes budget+1 tokens), target attention, draft model forward pass, tree building, and Python/launch overhead. Third, the assistant scales each component to the stronger-drafter scenario (block=16, budget=32, M=33 for the verify batch) using measured MoE scaling curves from the PRO 6000 and bandwidth ratios to project to B300.

The reasoning is explicit about uncertainty. The assistant notes that "the main uncertainties are the non-MoE overhead breakdown, draft cost at block16, and whether the drafter actually achieves those accept rates." This is not a weakness but a strength: the estimator is designed to be tweaked, and the assumptions are documented.

The Structure of the Answer

The message itself is concise and structured as a table-driven answer. It opens by stating that a calibrated estimator was built (python/estimate_b300.py) and that it models each decode step as MoE(M) + ATTN_BASE + SPEC. It then presents three scenarios.

Scenario 1: Current B300 performance. The measured 303 tok/s is the anchor. But the decomposition reveals something striking: "overhead is ~85% of the step." The MoE forward pass at M=9 takes only ~2.0 ms of the 14.5 ms step. The remaining ~12.5 ms is target attention, lm_head, base overhead, and the draft+tree+spec pipeline. This is the Amdahl effect predicted in the earlier analysis: on a GPU with ~4.5× the bandwidth of the PRO 6000, the MoE has become a minor cost, and everything else dominates.

The assistant then projects that simply porting to the native engine (cutting ~40% of the Python/launch overhead) would yield ~460 tok/s with the same b8 drafter — a 52% improvement from overhead reduction alone.

Scenario 2: Stronger drafter with SGLang-style pipeline. With block=16, budget=32, and acceptance rates of 12, 14, or 16 tokens per step, the step time grows to 20.9 ms (the MoE at M=33 costs ~5.3 ms instead of ~2.0 ms, and the draft+attention costs scale up). The resulting throughput ranges from 570 to 765 tok/s depending on the acceptance rate.

Scenario 3: Stronger drafter with native engine. If the native engine also strips the launch/Python overhead (the ~40% reduction), the step time drops to 14.7 ms, yielding 815 to 1090 tok/s.

The key dynamic is summarized succinctly: "at C=1, budget 32 pays a ~3 ms MoE tax (M=33 vs M=9: 5.3 vs 2.0 ms) to commit ~3× more tokens per step — acceptance grows faster than cost."

Assumptions and Their Implications

The estimator rests on several assumptions, and the message is transparent about them. The most critical are:

  1. The MoE cost scaling is linear in M (batch size). The estimator uses a measured scaling curve from the PRO 6000 and scales it to B300 by the bandwidth ratio. This assumes the kernel is bandwidth-bound at all batch sizes up to M=33, which is reasonable given that M=33 is still far below the M≈256 plateau where the MoE becomes compute-bound.
  2. The non-MoE overhead (ATTN_BASE + SPEC) scales modestly with budget. The estimator assumes that attention at M=33 costs more than at M=9, but not dramatically so, and that the draft forward pass roughly doubles when block size goes from 8 to 16. These are educated guesses, and the message flags them as the dominant source of uncertainty.
  3. The native engine can cut ~40% of overhead. This is based on the observation that Python dispatch, CPU tree building, and launch overhead constitute a large fraction of the current SGLang pipeline's step time. The native C++ engine eliminates the Python layer entirely and runs tree building on the GPU, so this assumption is plausible but unverified on B300 hardware.
  4. The drafter actually achieves 12–16 accept lengths. The message is careful to note that "that's a training outcome, not something the inference stack controls." The estimator can only project what would happen if such a drafter existed; it cannot guarantee that training one is feasible.
  5. budget>8 requires the native engine. This is a hard constraint from the prior session: SGLang's Triton-based verify kernels crashed CUDA graphs at budget>8 on sm_103 (the B300 architecture). The owned CUDA kernels in the native engine are the only path to running larger budgets at graph speed.

Input Knowledge Required

To fully understand this message, one needs familiarity with several concepts:

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. A calibrated, reproducible throughput model for DDTree on B300. The estimator script (estimate_b300.py) is a concrete artifact that anyone can run, modify assumptions, and explore sensitivity. It transforms the question from "how fast is B300?" into "how fast is B300 under these explicit assumptions?"
  2. The insight that overhead dominates on B300. The decomposition showing ~85% of step time is non-MoE is a sharp, quantitative articulation of the Amdahl effect. This is actionable: it tells the team exactly where to focus optimization effort.
  3. The native engine's value proposition is quantified. The projection that the native engine alone (without a better drafter) could boost throughput from 303 to ~460 tok/s gives a clear, measurable target for the engineering work.
  4. The strong-drafter projection bounds the opportunity. The 815–1090 tok/s range for the native engine with a 12–16 accept drafter gives a ceiling for what's possible. It answers the user's question concretely: "roughly doubles to triples C=1 throughput vs today's 303."
  5. The uncertainty is bounded. The ~±30% confidence interval is stated explicitly, and the dominant sources of uncertainty are identified. This prevents over-interpretation of the numbers.

The Thinking Process Visible in the Reasoning

The assistant's reasoning block reveals a sophisticated analytical approach. Let me trace the key steps:

Step 1: Anchor to measurements. The assistant immediately identifies the measured B300 point (303 tok/s at C=1, block=8, accept ~4.4) as the calibration anchor. This grounds the entire analysis in reality rather than speculation.

Step 2: Decompose the step time. The assistant breaks the 14.5 ms step into MoE(M=9) ≈ 2.0 ms and non-MoE ≈ 12.5 ms. This decomposition is critical because it reveals the overhead dominance.

Step 3: Scale the MoE component. Using the PRO 6000 MoE scaling curve (M=9 costs 8.85 ms, M=32 costs 23.8 ms under TP-8) and the bandwidth ratio between PRO 6000 and B300 (~4.5×), the assistant estimates M=33 on B300 at ~5.3 ms. This is a straightforward bandwidth-scaling calculation.

Step 4: Estimate the non-MoE scaling. The draft forward pass at block=16 is estimated to cost more than at block=8 (roughly 1.5–2×), and attention at M=33 costs more than at M=9. These are the weakest assumptions, and the assistant flags them accordingly.

Step 5: Compute the final numbers. With step time = MoE(M=33) + scaled non-MoE, and accept rate = 12–16, the throughput is accept / step_time.

Step 6: Add the native engine scenario. The assistant estimates that the native engine can cut ~40% of the non-MoE overhead by eliminating Python dispatch, CPU tree building, and launch overhead. This is applied to both the current and strong-drafter scenarios.

The reasoning is iterative and self-correcting. At one point the assistant considers whether the draft cost scaling is linear with block size and decides it's not (partly launch-bound), adjusting the estimate accordingly. This kind of real-time refinement is characteristic of deep technical reasoning.

Mistakes and Incorrect Assumptions

The message is careful about uncertainty, but there are potential issues worth examining:

  1. The 40% overhead reduction for the native engine is an estimate, not a measurement. It's plausible but could be off. If the actual overhead reduction is only 20%, the native engine numbers would be ~720–960 tok/s instead of 815–1090. If it's 60%, they'd be even higher.
  2. The MoE scaling from PRO 6000 to B300 assumes perfect bandwidth scaling. In practice, B300's HBM3e may have different latency characteristics, and the kernel may not achieve the same percentage of peak bandwidth at small batch sizes. The 67–73% efficiency measured on PRO 6000 may not transfer exactly.
  3. The estimator assumes the draft model's cost scales modestly with block size. If the stronger drafter is a significantly larger model (not just better trained), its forward pass could dominate the step time, reducing the projected gains.
  4. The acceptance rate of 12–16 for a stronger drafter is a hypothetical. The message is explicit about this, but it's worth emphasizing that achieving such acceptance rates on hard reasoning/analysis text (which the session had previously identified as problematic) may be significantly harder than on predictable text.

The Broader Significance

This message is a case study in how to answer a "what if" question about hardware performance rigorously. Rather than giving a single number, the assistant:

  1. Built a transparent model with documented assumptions
  2. Calibrated to real measurements rather than relying on theory alone
  3. Decomposed the problem into independently estimable components
  4. Quantified uncertainty and identified its sources
  5. Made the tool available for the user to explore sensitivity This approach transforms the conversation from speculation to engineering. The user can now ask "what if the native engine only cuts 30% overhead?" or "what if the drafter only achieves 10 accept?" and get answers by tweaking the script. The message also crystallizes a key insight for the project: on B300-class hardware, the bottleneck shifts from memory bandwidth to overhead. This has profound implications for where to invest engineering effort. The native C++ engine, GPU-resident tree building, and owned CUDA kernels are not just nice-to-haves — they are necessary to realize the hardware's potential. The message provides a quantitative justification for this architectural direction.

Conclusion

Message [msg 12043] is a tightly reasoned, methodologically sound answer to a complex performance estimation question. It demonstrates the value of anchoring projections in real measurements, decomposing systems into measurable components, and being transparent about assumptions and uncertainty. The key output — that a stronger drafter could roughly double to triple C=1 throughput on B300, reaching 0.8–1.1k tok/s — is grounded in a reproducible model rather than intuition. And the deeper insight — that overhead, not memory bandwidth, is the dominant constraint on fast hardware — provides strategic direction for the entire project.

The message is also a model for how to communicate technical estimates: present the calibrated baseline first, then the projection, then the caveats. Give the user a tool, not just a number. And always be clear about what is measured versus what is assumed.