The Budget Ceiling: A User's Challenge to Underutilized Compute in Speculative Decoding

"b8 is really low, will cause us to underutilise compute and best case equal sequential accept len, no? Can we push 16/32/64?"

This single sentence, spoken by the user at a critical juncture in the conversation ([msg 11795]), is a masterclass in concise technical insight. It arrives after a cascade of infrastructure debugging, benchmarking, and a failed attempt to scale concurrency, and it fundamentally redirects the optimization strategy for deploying Kimi K2.6 with DDTree speculative decoding on 8× B300 SXM6 GPUs. To understand why this message matters, we must unpack the dense reasoning packed into its 21 words.

The Moment It Arrives

The immediate context is one of failure and uncertainty. In the preceding message ([msg 11794]), the assistant had attempted to push the TP8+NVLS DDTree service to higher concurrency by increasing max-running-requests from 128 to 256. The result was a complete service crash: the benchmark harness reported "Connection refused" errors at both C=128 and C=256, and NVIDIA power readings showed all eight GPUs sitting at 0% utilization drawing a mere ~185W — essentially idle power. The assistant's plan to "push to the ceiling" by scaling batch size had backfired spectacularly.

It is at this moment of setback that the user intervenes. But rather than debugging the crash or suggesting a rollback, the user pivots the conversation to an entirely different axis of optimization: the tree budget. This is not a reactive fix — it is a strategic redirection.

Deconstructing the Argument

The user's message makes three interconnected claims, each building on the last.

"b8 is really low" — The current DDTree configuration uses budget=8, meaning the speculative tree explores at most 8 candidate tokens per verification step. The user asserts this is conservative. This judgment is grounded in the hardware context: the B300 SXM6 GPUs have 275 GB of HBM each, connected via NVLink, and the assistant's own benchmarks showed power draw at only 360–450W out of a 1100W thermal ceiling. The GPUs were 100% utilized but power-limited — a classic sign of memory-bandwidth-bound operation where compute units are starved for data. If compute is idle, the tree verification step (which is compute-light relative to the base model forward pass) has headroom to evaluate more candidates.

"will cause us to underutilise compute" — This is the core of the user's argument. The B300 GPUs are not being pushed to their computational limits. The INT4 MoE decode workload is HBM-bandwidth-bound: the tensor cores spend most of their time waiting for expert weights to arrive from HBM. During this waiting, the GPU could be evaluating additional tree candidates essentially for free, since the tree verification kernel is typically much cheaper than the full model forward pass. By keeping the budget at 8, the user argues, we are leaving this spare compute capacity on the table.

"best case equal sequential accept len, no?" — Here the user demonstrates a deep understanding of speculative decoding theory. In sequential draft-then-verify, the acceptance length (number of tokens accepted per step) is bounded by the number of draft tokens. In tree-based speculative decoding (DDTree), the tree structure allows exploring multiple branches simultaneously, which can yield higher acceptance lengths than sequential for the same budget. However, the user correctly notes that with budget=8, even in the best case, the acceptance length cannot exceed 8. The earlier benchmarks showed mean acceptance lengths of 3.5–4.1 tokens per step with budget=8 — meaning the drafter was hitting only about half the ceiling. If the drafter quality is good enough, a larger budget should translate directly into higher acceptance lengths and thus higher throughput.

"Can we push 16/32/64?" — The final question is a call to action. The user is not asking whether larger budgets are possible, but to push them. This is an engineer's instinct: when you see headroom, you fill it.

The Assumptions at Play

The user's argument rests on several assumptions, some explicit and some implicit.

First, it assumes the drafter model quality is sufficient to benefit from a larger tree. If the drafter cannot predict good candidates beyond the first few tokens, a larger tree would add noise without improving acceptance length. The earlier benchmarks (acceptance length ~3.5–4.1 with budget=8) suggest the drafter is decent but not perfect — there is room to grow.

Second, it assumes that the tree verification kernel can scale to larger budgets without hitting memory or compute bottlenecks. The tree verification in DDTree involves attention over the tree structure, which scales with the number of nodes. At some point, the verification cost could eat into the gains from higher acceptance length.

Third, and most critically, the user may not be fully aware of the CUDA graph bug that the assistant had encountered earlier (documented in the segment summary: "any budget > 8 caused illegal memory accesses (b12/b16) or garbage output (b32) during the tree-verify graph capture"). This bug is sm_103-specific and had blocked earlier attempts to use larger budgets. The user's push for 16/32/64 implicitly assumes this bug can be worked around — perhaps by running in eager mode (without CUDA graphs) and accepting the performance penalty, or by fixing the graph capture issue.

Why This Message Matters

The user's intervention is significant for several reasons. It demonstrates the value of human intuition in the optimization loop: where the assistant was focused on debugging a failed concurrency experiment, the user saw a more fundamental lever to pull. The assistant had been working within the established configuration (budget=8) and trying to squeeze more throughput from batch size scaling. The user recognized that the configuration itself was the bottleneck.

Moreover, the message reveals the user's mental model of the system. They understand that on NVLink-connected hardware with memory-bandwidth-bound workloads, the bottleneck is not compute but data movement. The tree verification step is cheap relative to weight loading, so the marginal cost of evaluating more candidates is low. The optimal strategy is therefore to maximize the speculative gain per base model forward pass, which means maximizing the tree budget up to the point where verification cost begins to dominate.

This message also highlights a tension in the assistant's approach. The assistant had been treating budget=8 as a fixed parameter and optimizing around it (tuning NVLS, scaling concurrency, adjusting max-running-requests). The user's question forces a reconsideration of whether budget=8 is the right starting point at all. It is a reminder that in system optimization, the most impactful changes often come from questioning the parameters, not tuning around them.

The Knowledge Flow

The input knowledge required to understand this message includes: familiarity with speculative decoding and DDTree mechanics (budget, top-k, acceptance length), awareness of the hardware context (B300 SXM6 with NVLink, 1100W TDP), and understanding of the memory-bandwidth-bound nature of INT4 MoE decode. The user also draws on the immediately preceding context: the assistant's failed attempt to scale concurrency and the power readings showing idle GPUs.

The output knowledge created by this message is a new optimization direction. It reframes the problem from "how do we scale batch size without crashing the service?" to "how do we scale tree budget to utilize spare compute?" This pivot would lead the assistant to test larger budgets, eventually confirming the user's hypothesis (budget=16 in eager mode lifted acceptance to 5.3–6.4 tokens per step) while also discovering the sm_103 CUDA graph bug that blocks the full realization of these gains.

Conclusion

The user's message is a textbook example of effective technical intervention: concise, grounded in system understanding, and strategically redirectional. It identifies a fundamental ceiling in the current configuration, articulates why it matters, and proposes a concrete next step. In doing so, it transforms a moment of failure (the service crash) into an opportunity for deeper optimization. The question "Can we push 16/32/64?" would echo through the subsequent hours of debugging, benchmarking, and ultimately the writing of the DDTree findings report — a testament to the power of asking the right question at the right time.