The Art of Reproducible Estimation: Committing a Throughput Model for B300 Inference

Introduction

In the midst of a sprawling coding session dedicated to building a native C/C++/CUDA DDTree inference engine for the Kimi K2.6 model, a seemingly mundane git commit appears. Message 12042 is a single bash command that stages all changes and commits them with a detailed message. On its surface, it is a routine act of version control—the sort of thing that happens dozens of times in a session. But this particular commit crystallizes a critical moment of analytical reasoning: the transition from hand-waved speculation to reproducible, calibrated estimation.

The commit message reads:

Add B300 C=1 throughput estimator + estimate table

- python/estimate_b300.py: transparent per-step model (MoE(M)+ATTN_BASE+SPEC) calibrated to two
  measured B300 points (AR 133, DDTree b8 303 @ accept 4.4). Reproducible, ~+/-30%.
- Estimates: b8 now ~303 (measured); stronger drafter b16/budget32 @ accept 12-16 -> ~570-765
  (SGLang pipeline) or ~815-1090 tok/s (native engine removing launch/py overhead, required for
  budget>8 on sm_103). Recorded in docs/B300_HBM_NVLINK_NOTES.md s5.

This article unpacks why this commit matters, the reasoning that led to it, the assumptions it encodes, and the knowledge it creates.

Why This Message Was Written: The Context and Motivation

To understand why the assistant wrote this message, we must trace the conversation that preceded it. The user had just asked a pointed question: "Can you estimate how fast a B300 would be at C=1 now? What if we had a stronger drafter which with ddtree at block=16 and budget=32 would hit 12-16 accept lengths?"

This question landed in a specific context. The B300 machine—a high-bandwidth NVIDIA datacenter GPU with ~8 TB/s HBM3e memory bandwidth—had been released and was no longer accessible for live benchmarking. The assistant had previously measured the B300 achieving 303 tokens per second (tok/s) with the current block-8 DDTree drafter at C=1, and 133 tok/s for autoregressive decoding. But the user was now asking about a hypothetical stronger drafter with different parameters (block=16, budget=32) that could achieve significantly higher acceptance rates (12-16 tokens per step instead of ~4.4).

The assistant faced a choice. It could offer a quick back-of-the-envelope estimate, relying on intuition and rough scaling laws. Or it could build something more durable: a transparent, reproducible model that anyone could inspect, modify, and re-run. The assistant chose the latter path, and this commit is the record of that decision.

The deeper motivation is epistemological. Throughout this coding session, the assistant has been building a native inference engine precisely because it wants to replace guesswork with measurement. The DDTree engine's entire raison d'être is to eliminate the overhead and uncertainty of the SGLang pipeline. When faced with a question about future hardware, the assistant applied the same philosophy: rather than speculate, build a model grounded in actual measurements.

How Decisions Were Made: The Estimator Architecture

The commit message reveals the structure of the estimator. It implements a "transparent per-step model" with three components:

  1. MoE(M): The cost of the Mixture-of-Experts forward pass, which scales with the number of tokens M being verified. This is the dominant compute cost in the DDTree verify step.
  2. ATTN_BASE: The baseline attention cost, which is relatively fixed for a given context length.
  3. SPEC: The speculative decoding overhead, including the draft model forward pass, tree construction, and Python dispatch. The model is calibrated to two measured B300 data points: the autoregressive baseline (133 tok/s) and the DDTree block-8 configuration (303 tok/s with accept 4.4). This calibration anchors the absolute scale of the estimates to real hardware measurements, rather than relying on theoretical peak bandwidth numbers alone. The decision to use a three-component decomposition reflects a specific mental model of where time goes in the DDTree decode loop. The MoE component captures the bandwidth-bound work of streaming expert weights through the GPU's HBM. The ATTN_BASE captures the compute-bound attention work. The SPEC component captures everything else—the "overhead" that the native engine is designed to eliminate.

Assumptions and Their Implications

Every model makes assumptions, and this estimator is explicit about its uncertainties. The commit message notes "~+/-30%" uncertainty, which is honest about the limits of extrapolation.

The key assumptions embedded in this model include:

Linearity of the MoE cost with M: The estimator assumes that the MoE forward pass cost scales linearly with the number of tokens being verified. This is a reasonable first-order approximation for a bandwidth-bound workload, but it ignores potential sub-linear scaling from kernel launch amortization and L2 cache reuse.

Fixed overhead independence from M: The ATTN_BASE and SPEC components are treated as roughly constant regardless of tree size. In reality, larger trees mean more attention work and more draft tokens to process. The estimator accounts for this implicitly through the calibration to the block-8 point, but extrapolating to budget=32 introduces uncertainty.

The 40% overhead reduction from a native engine: The estimate for the native engine scenario assumes that replacing the SGLang pipeline with a C++/CUDA loop cuts launch and Python overhead by ~40%. This is a plausible engineering estimate, but it hasn't been measured on B300 hardware.

The stronger drafter's acceptance rate: The most speculative assumption is that a stronger drafter with block=16 and budget=32 could achieve acceptance rates of 12-16 tokens per step. This is the user's hypothetical, not a measured quantity. The estimator takes it as input and computes the consequences.

These assumptions are not hidden. The commit message and the script itself document them explicitly. This transparency is a deliberate choice—it allows anyone reading the estimate to understand where it might be wrong and to update the numbers as better measurements become available.

Input Knowledge Required

To understand this message, a reader needs knowledge spanning several domains:

Hardware architecture: Understanding what B300 is (a datacenter GPU with ~8 TB/s HBM3e), what PRO 6000 is (a workstation GPU with ~1.8 TB/s GDDR7), and how memory bandwidth dominates transformer inference.

Speculative decoding: Understanding the DDTree algorithm, where a small "drafter" model proposes multiple candidate tokens in a tree structure, and the large "target" model verifies them in a single batched forward pass. Key parameters include block size (how many tokens the drafter predicts ahead), budget (the maximum number of tokens in the tree), and acceptance rate (how many tokens the target model accepts per verify step).

The SGLang pipeline: Understanding that the current DDTree implementation runs within the SGLang inference framework, which adds Python dispatch overhead, CPU-side tree construction (using a Python heapq), and Triton-based kernels. The native engine aims to replace all of this with custom C++/CUDA code.

The project's history: Knowing that the assistant has been building a kdtree-engine repository over multiple segments, with custom CUDA kernels for tree building, verify attention, and token acceptance. The B300 analysis document (docs/B300_HBM_NVLINK_NOTES.md) was created in the immediately preceding messages.

Output Knowledge Created

This commit creates several forms of knowledge:

A reproducible estimator script: python/estimate_b300.py is now part of the repository. Anyone can run it, modify its assumptions, and see how the estimates change. This is fundamentally different from a one-off calculation in a chat message.

A documented estimate table: The results are recorded in docs/B300_HBM_NVLINK_NOTES.md section 5, creating a permanent reference that can be cited and updated.

A calibrated baseline: The estimator anchors future work. If someone later measures the B300 achieving different throughput numbers, they can update the calibration and re-run the projections.

A decision framework: The estimates clarify the relative importance of different levers. The native engine's overhead reduction is worth ~200-300 tok/s at the stronger drafter configuration. The stronger drafter itself is worth ~2-3× over the current block-8 drafter. This helps prioritize engineering effort.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the agent reasoning blocks of the preceding messages, shows a careful progression from question to answer.

First, the assistant resists the temptation to hand-wave. It explicitly says: "Let me write a reproducible estimator rather than hand-wave." This is a methodological commitment.

Second, it anchors to measured data. The B300 is no longer accessible, but the prior session's measurements (303 tok/s at block-8, 133 tok/s autoregressive) provide calibration points. The PRO 6000 measurements provide the MoE scaling curve. These anchors ground the estimates in reality.

Third, it decomposes the problem. Rather than estimating throughput directly, it builds a per-step cost model and divides the expected acceptance rate by the step time. This decomposition makes the reasoning transparent and the assumptions explicit.

Fourth, it acknowledges uncertainty. The ~±30% error bar is honest. The estimator doesn't pretend to be more precise than the data allows.

Fifth, it commits the work. The act of committing the estimator to the repository transforms a one-off calculation into a permanent project asset. This is a deliberate choice about knowledge management—the estimate becomes part of the project's shared understanding, not a transient artifact of a chat conversation.

Conclusion

Message 12042 is a git commit, but it is also a statement about how to do rigorous engineering estimation. Faced with a question about future hardware performance, the assistant chose to build a transparent, calibrated, reproducible model rather than offer an unsupported guess. The commit encodes assumptions explicitly, acknowledges uncertainty honestly, and creates a permanent artifact that the project can build upon.

In a session dominated by building custom CUDA kernels, debugging throughput regressions, and deploying inference services, this commit stands out as a moment of analytical clarity. It demonstrates that the same rigor applied to kernel development can and should be applied to performance estimation. The result is not just a number—it is a framework for understanding where the numbers come from and how they might change.