The Budget Sweep: Pushing Speculative Decoding to Its Limits on NVIDIA B300

Introduction

In the high-stakes world of large language model inference, every token per second counts. When deploying a 275-billion-parameter model like Kimi K2.6 with speculative decoding, the configuration of the draft tree—specifically its budget (the number of candidate tokens explored per step) and top-k (the branching factor)—can make the difference between a mediocre speedup and a transformative one. This article examines a single message from an opencode coding session ([msg 11798]) in which an AI assistant executes a systematic budget sweep on an 8× NVIDIA B300 SXM6 machine, testing budget values of 16, 32, and 64 with a top-k of 8. The message is a deceptively simple bash script, but it sits at the convergence of several critical threads: hardware capability discovery, speculative decoding theory, benchmark methodology, and the practical art of wrangling distributed inference systems.

To understand this message, one must appreciate the journey that led to it. The session had been an extended exploration of deploying Kimi K2.6 with DFlash (draft-fast) speculative decoding across two hardware platforms: a PCIe-connected 8× RTX PRO 6000 system and an NVLink-connected 8× B300 SXM6 system. The B300 machine, with its 275 GB of HBM per GPU and NVLink interconnect, represented the high end of the hardware spectrum—a machine where compute was abundant and the primary bottleneck was memory bandwidth. Earlier benchmarks had established that the TP8+NVLS (tensor parallelism with NVIDIA's low-latency NVLink SHARP) configuration achieved 303 tok/s at concurrency 1 and scaled to 4723 tok/s at concurrency 128, with power consumption hovering around 450 W out of a 1100 W ceiling. The system was clearly compute-idle and memory-bandwidth-bound—the tensor cores had spare capacity, but the INT4 weight-only MoE decode was spending most of its time waiting for HBM reads.

It was in this context that the user posed a sharp question in [msg 11795]: "b8 is really low, will cause us to underutilise compute and best case equal sequential accept len, no? Can we push 16/32/64?" The user's intuition was correct: with a budget of 8 and top-k of 4, the draft tree was too small to exploit the B300's spare compute. Larger budgets would allow the tree to explore more candidate paths per step, increasing the probability of matching the target model's greedy trajectory and thereby lifting the acceptance length above the sequential chain. On a compute-rich system where verification is nearly free, bigger trees should translate directly to higher throughput.

The Message in Full

The subject message is a single bash command executed by the assistant in response to the user's challenge. Here it is in its entirety:

cd /home/theuser/glm-kimi-sm120-rtx6000bw
# Sweep budget 16/32/64 (topk=8) on B300, single-stream + concurrency, capture accept
CONFIGS=("16 8 2048 8" "32 8 2048 8" "64 8 2048 8")
for cfg in "${CONFIGS[@]}"; do
  read -r B T W BS <<< "$cfg"
  tag="B300_b${B}t${T}"
  echo ""
  echo "########## budget=$B topk=$T block=$BS ##########"
  if ! timeout 1000 ssh -o ConnectTimeout=15 root@86.38.182.109 "bash /root/reconfig_ddtree.sh $B $T $W $BS" 2>&1; then
    echo "SKIP $tag (failed to start)"
    timeout 15 ssh root@86.38.182.109 "journalctl -u sglang-k26-ddtree.service --no-pager -n 12|grep -iE 'error|cublas|None|fatal'|tail -5" 2>&1
    continue
  fi
  timeout 900 ssh -o ConnectTimeout=15 root@86.38.182.109 "cd /root && /root/venv/bin/python bench_ddtree_matrix.py --label '$tag' --contexts 60 1024 --concurrency 1 32 64 --max-tokens 512 --out /root/bench_results_opt/${tag}.json" 2>&1
  acc=$(timeout 15 ssh root@86.38.182.109 "journalctl -u sglang-k26-ddtree.service --no-pager --since '4 min ago'|grep -oE 'accept len: [0-9.]+'|tail -15|awk '{s+=\$3;n++}END{if(n)printf \"%.2f\",s/n}'" 2>&1)
  echo "  mean accept_len=$acc"
done
echo "########## BUDGET SWEEP DONE ##########"

At first glance, this is a straightforward loop over three configurations. But every line encodes a series of hard-won lessons from the preceding hours of debugging, benchmarking, and infrastructure recovery.

Why This Message Was Written: The Reasoning and Motivation

The immediate trigger for this message was the user's question in [msg 11795]. But the deeper motivation was the assistant's recognition that the B300 system was fundamentally underutilized. The earlier benchmarks had shown power consumption at only 450 W out of 1100 W, GPU utilization at 100% but with the tensor cores largely idle, and the workload confirmed to be HBM-bandwidth-bound. In such a regime, the cost of verifying additional draft tokens is nearly zero—the compute units are waiting on memory anyway, so larger draft trees can be verified without increasing the wall-clock time per step. The only downside is the risk of worse acceptance if the tree is poorly constructed, but with a well-trained drafter and appropriate top-k, larger budgets should monotonically improve acceptance length up to the drafter's depth limit.

The assistant's reasoning, visible in the preceding messages, shows a clear understanding of the tradeoff. In [msg 11797], the assistant notes: "With block_size=8 the tree depth caps at 7, so bigger budget adds width (more candidates/depth → higher accept up to ~7-8), and on compute-rich B300 the larger verify is nearly free." This is the key insight: the drafter was trained with a block size of 8, meaning it can predict at most 7 draft tokens ahead (the 8th being the first token of the next block). Increasing the budget from 8 to 64 doesn't make the tree deeper—it makes it wider, adding more alternative candidates at each depth. This increases the probability that at least one candidate matches the target model's greedy path, thereby extending the acceptance chain.

The assistant also had to navigate a recent failure: the attempt to increase max-running-requests to 256 had crashed with a CUBLAS_STATUS_EXECUTION_FAILED error in the MLA (Multi-head Latent Attention) absorb GEMM. This was a hard constraint—the cuBLAS strided batched GEMM implementation had a dimension limit that was exceeded when batch size × number of heads grew too large. The assistant correctly diagnosed this as independent of the EP/NVLS configuration and capped maxreq at 128, which was already delivering 4723 tok/s at C=128. The budget sweep would operate within this constraint, testing whether larger trees could improve single-stream performance without increasing the batch size.

How Decisions Were Made: The Architecture of the Sweep

The script embodies several deliberate design decisions. First, the choice of top-k=8 across all budgets. This is a significant increase from the previous top-k=4 used with budget=8. The assistant is implicitly betting that on the B300's abundant compute, a wider branching factor will be beneficial—more candidates per position means a higher chance of matching the target, and the verification cost is amortized over the larger tree.

Second, the test dimensions: context lengths of 60 and 1024 tokens, concurrency levels of 1, 32, and 64. The short context (60 tokens) represents a near-empty KV cache where memory pressure is minimal, giving the best-case throughput. The longer context (1024 tokens) tests a more realistic scenario where the KV cache consumes significant HBM. The concurrency sweep tests how the system scales from single-stream (where speculative decoding's benefits are most visible) to high-throughput serving (where the system is amortizing weight reads across many requests).

Third, the inclusion of acceptance length (accept_len) as a captured metric. This is crucial for understanding why a particular budget performs well or poorly. Throughput alone can be misleading—higher throughput might come from better acceptance or from some other system effect. By capturing the mean acceptance length from the service logs, the assistant can correlate throughput changes with the fundamental speculative decoding efficiency.

Fourth, the error handling: if the reconfiguration fails (the reconfig_ddtree.sh script returns non-zero), the script skips that configuration and dumps the last few error lines from the journal. This is a pragmatic choice—on a remote machine where a restart can fail for many reasons (OOM, CUDA errors, configuration parsing issues), it's better to skip and report than to let the entire sweep hang.

Assumptions Made by the Assistant

The message rests on several assumptions, some explicit and some implicit.

That the reconfiguration script works reliably. The reconfig_ddtree.sh script is called with four parameters (budget, top-k, window, block_size) and is expected to restart the SGLang service with the new configuration. The assistant assumes a 1000-second timeout is sufficient for the restart, which is generous given that the model takes about 6 minutes to load. However, earlier in the session ([msg 11791] and surrounding messages), the assistant had encountered readiness-check race conditions where the old process answered health checks briefly before being killed, and the new process was still loading weights. The script does not include a readiness check—it relies on reconfig_ddtree.sh to handle that internally, or assumes the benchmark script will fail gracefully if the service isn't ready.

That the benchmark script (bench_ddtree_matrix.py) produces meaningful results. This script had been developed and refined over the course of the session, and the assistant trusts that it correctly measures throughput, handles concurrency, and reports results in a parseable JSON format. The script's maturity is evident from its use in earlier benchmarks that produced clean, interpretable matrices.

That acceptance length from the journal is a reliable metric. The assistant greps for accept len: [0-9.]+ in the last 4 minutes of logs and averages the last 15 occurrences. This assumes that (a) the log format is stable, (b) the benchmark generates enough requests to produce at least 15 acceptance measurements, and (c) the 4-minute window captures the benchmark period without including noise from previous runs. This is a reasonable heuristic but not rigorous—if the benchmark finishes quickly, the acceptance measurements might be from the tail of the previous configuration's run.

That budget 64 is worth testing. With block_size=8 and a maximum depth of 7, a budget of 64 means the tree has 64 candidate tokens to distribute across 7 depth positions. This is a very wide tree—nearly 10 candidates per position on average. The assistant implicitly assumes that the drafter's probability distribution is diverse enough that additional candidates continue to add marginal value, and that the verification kernel can handle the increased tree width without performance degradation. This is not guaranteed—at some point, additional candidates will have near-zero probability and contribute nothing to acceptance.

That the B300's compute is truly "free" for verification. The earlier power measurements showed 450 W under load, well below the 1100 W ceiling. But this is an aggregate measurement—it doesn't reveal whether individual GPU tensor cores are idle or whether there are micro-architectural bottlenecks (e.g., instruction issue width, register pressure) that could make larger trees slower even when HBM bandwidth is the primary constraint. The assistant is making a reasonable inference from the data available, but it's an inference nonetheless.

Mistakes and Incorrect Assumptions

The most significant potential issue is the lack of a readiness check between reconfiguration and benchmarking. Earlier in the session ([msg 11791] and chunk 1 of the segment), the assistant had spent considerable effort debugging a race condition where the reconfig script's readiness check failed because the old process answered /v1/models briefly before being killed, while the new process was still loading weights for ~6 minutes. The current script does not include any explicit readiness polling—it relies on the 1000-second timeout of the reconfig_ddtree.sh call to cover the loading time, but if the script returns before the service is actually ready (e.g., if it exits after the old process is killed but before the new one is serving), the benchmark will fail with connection errors.

The script also does not capture power consumption during the benchmark. Given that one of the key motivations for the sweep was the observation that the B300 was underpowered (450 W vs 1100 W ceiling), it would have been valuable to measure whether larger budgets increase power draw. If budget=64 drives power to 800 W while budget=16 stays at 450 W, that changes the efficiency calculus. The omission is understandable—adding nvidia-smi polling to the benchmark loop would complicate the script—but it leaves an important question unanswered.

Another subtle issue is the acceptance length extraction. The command grep -oE &#39;accept len: [0-9.]+&#39; extracts the numeric value after "accept len:". But the --since &#39;4 min ago&#39; flag to journalctl is evaluated on the remote machine at the time the SSH command runs, which could be several minutes after the benchmark finished. If the benchmark took 10 minutes and the SSH command runs 2 minutes later, the --since window might miss the benchmark entirely or include noise from subsequent activity. A more robust approach would be to capture the acceptance metrics during the benchmark run itself, perhaps by having the benchmark script log them to a file.

Input Knowledge Required to Understand This Message

To fully grasp what this message is doing, one needs knowledge spanning several domains:

Hardware architecture: The B300 SXM6 is an NVIDIA GPU with NVLink interconnect, 275 GB of HBM, and sm_103 compute capability. NVLink provides significantly higher bandwidth than PCIe for inter-GPU communication, making tensor parallelism (TP) more attractive relative to expert parallelism (EP). The system has 8 such GPUs, and the assistant has configured TP8 (tensor parallelism across all 8 GPUs) with NVLS (NVLink SHARP, a hardware-accelerated collective communication feature).

Speculative decoding with DDTree: DDTree (Draft-Tree) is a form of speculative decoding where the draft model generates a tree of candidate tokens rather than a single chain. The "budget" is the total number of candidate tokens in the tree, and "top-k" is the branching factor—how many children each node can have. The tree is verified against the target model in parallel, allowing multiple tokens to be accepted per step. The "block size" parameter controls the drafter's lookahead window.

The DFlash architecture: DFlash is a draft-fast speculative decoding system where a lightweight drafter model predicts tokens that are then verified by the full target model. The drafter in this case is SubSir/Kimi-K2.6-DFlash-tmp-long with 6 draft layers and block_size=8. The drafter uses sliding window attention (window=2048) to limit its KV cache.

SGLang serving stack: The service is running under systemd with a configuration file that specifies model path, parallelism strategy, and speculative decoding parameters. The reconfig_ddtree.sh script modifies this configuration and restarts the service.

CUDA and cuBLAS constraints: The assistant has learned that max-running-requests=256 triggers a CUBLAS strided batched GEMM failure in the MLA attention layer. This is a hard constraint that limits the maximum batch size, independent of the speculative decoding configuration.

The benchmark methodology: The bench_ddtree_matrix.py script tests across a matrix of context lengths and concurrency levels, measuring aggregate throughput (tok/s), per-request throughput, wall time, and prompt processing time. It saves results to JSON for later analysis.

Output Knowledge Created by This Message

This message produces several forms of output knowledge:

  1. Benchmark results for budget=16, top-k=8 on B300 at context lengths 60 and 1024 with concurrency 1, 32, and 64, saved to /root/bench_results_opt/B300_b16t8.json.
  2. Benchmark results for budget=32, top-k=8 under the same conditions, saved to /root/bench_results_opt/B300_b32t8.json.
  3. Benchmark results for budget=64, top-k=8 under the same conditions, saved to /root/bench_results_opt/B300_b64t8.json.
  4. Mean acceptance length for each configuration, extracted from the service logs and printed to stdout.
  5. Failure diagnostics if any configuration fails to start, including the last few error lines from the journal. This data directly answers the user's question about whether larger budgets improve throughput on the B300. It also provides the foundation for the next phase of optimization: if budget=16 gives the best throughput, the assistant can fine-tune around that value; if budget=64 gives diminishing returns, it confirms that the drafter's depth limit (block_size=8) constrains the benefits of wider trees.

The Thinking Process Visible in the Message

The assistant's thinking process is visible not just in the script itself but in the structure of the sweep. Several design choices reveal the assistant's mental model:

The choice of top-k=8 across all budgets shows that the assistant is treating the branching factor as a secondary variable, holding it constant to isolate the effect of budget. This is a deliberate experimental design choice—vary one thing at a time. The assistant could have swept both budget and top-k simultaneously (a grid search), but that would multiply the number of configurations and risk hitting time or reliability constraints.

The inclusion of both short and long contexts (60 and 1024 tokens) shows that the assistant is thinking about how the KV cache size affects speculative decoding. At short context, the KV cache is small and memory bandwidth is primarily consumed by model weights. At longer context, the KV cache consumes significant HBM, potentially reducing the bandwidth available for weight reads and changing the optimal tree size.

The concurrency sweep (1, 32, 64) shows that the assistant is thinking about the system's behavior under different load regimes. At concurrency 1, speculative decoding's benefits are most visible because the system is doing one thing at a time. At high concurrency, the system is batching multiple requests, and the benefits of speculative decoding may be diluted by the increased batch size (which already improves throughput by amortizing weight reads).

The acceptance length capture shows that the assistant wants to understand the mechanism behind any throughput changes, not just the aggregate result. If budget=64 gives lower throughput than budget=16 despite higher acceptance length, that would indicate that the verification cost is growing faster than the acceptance benefit—a crucial diagnostic.

The error handling pattern (skip on failure, dump logs, continue) shows that the assistant has internalized the fragility of the remote deployment. Rather than letting one failure abort the entire sweep, the assistant treats failures as data points and moves on. This is a pragmatic approach born from hours of debugging CUDA errors, OOMs, and configuration mismatches.

Conclusion

The budget sweep message at [msg 11798] is a masterclass in systematic experimental design applied to large-scale ML inference. In a single bash script, the assistant encodes a hypothesis (larger budgets improve throughput on compute-rich hardware), a methodology (controlled variation of budget while holding other parameters constant), a measurement framework (throughput matrix plus acceptance length), and a failure handling strategy (skip and report). The script is the product of hours of prior work—debugging CUDA errors, benchmarking parallelism strategies, understanding the drafter's architecture, and learning the quirks of the SGLang serving stack.

What makes this message particularly interesting is what it doesn't show: the readiness-check race condition that plagued earlier reconfigurations, the CUBLAS dimension limit that caps batch size, the power measurements that revealed the B300's spare compute capacity, and the theoretical understanding that on an HBM-bandwidth-bound system, verification is nearly free. All of this context is invisible in the script itself but essential to understanding why the assistant chose these particular parameters and this particular experimental design.

The message also reveals the assistant's growth as a practitioner. Earlier in the session, the assistant might have run a single benchmark and declared victory. Here, the assistant runs a systematic sweep, captures mechanistic metrics (acceptance length) alongside aggregate metrics (throughput), and handles failures gracefully. This is the behavior of someone who has learned through experience that distributed inference systems are complex, that hardware constraints are real, and that the only way to find the optimal configuration is to measure, measure, and measure again.