The Benchmarking Gap: Understanding Speculative Decoding's Batch-Size Blind Spot

Introduction

In the middle of a high-stakes EAGLE-3 training pipeline for the Kimi-K2.5 model, a brief exchange between user and assistant reveals a critical moment of reflection. The user asks a simple follow-up question — "And at other batch sizes?" — and the assistant's response, while concise, exposes a significant blind spot in the benchmarking strategy that has guided weeks of work. This message (index 4294) is the assistant's candid acknowledgment that systematic throughput benchmarking across batch sizes was never performed, coupled with a nuanced explanation of why that decision made sense at the time, and a forward-looking proposal to fill the gap.

The message is modest in length but dense with reasoning about the fundamental tradeoffs of speculative decoding, the economics of PCIe-bound multi-GPU inference, and the implicit assumptions that shaped an entire development trajectory. To understand its significance, we must reconstruct the context: a multi-week effort to train an EAGLE-3 draft model for the Kimi-K2.5 architecture, running on 8 RTX PRO 6000 Blackwell GPUs connected via PCIe, with the goal of accelerating single-stream inference beyond the 90 tok/s SGLang baseline.

The Conversation That Preceded It

The exchange begins with the user at <msg id=4291> asking a straightforward question: "Remind me, what inference numbers were we getting in kimi k2.5 inference w/o eagle3?" The assistant responds at <msg id=4292> with a clean table of three benchmarks: vLLM at 82.5 tok/s, SGLang at 90.0 tok/s, and SGLang with the old 10K-sample EAGLE-3 drafter at 82.3 tok/s — actually slower than the no-speculation baseline. The assistant explains that the 10K drafter had an acceptance length of only ~2.1 tokens, meaning the overhead of running the drafter and verification exceeded the savings from speculation. The goal with the new 37K-sample, TTT=5 (tree-attention training steps) drafter is to push acceptance length to ~3.0+, where speculation becomes net positive, especially on a PCIe system where each accepted token saves a full 8-GPU allreduce round.

The user then asks at <msg id=4293>: "And at other batch sizes?" — a natural follow-up that probes whether the benchmarking picture is complete.

What the Subject Message Actually Says

The assistant's response at <msg id=4294> is worth quoting in full:

We didn't benchmark other batch sizes systematically. All the numbers above were single-stream (batch size 1, single concurrent request) — that's the latency-sensitive scenario where speculative decoding matters most.

>

For the record, the key single-stream numbers: - 90.0 tok/s — SGLang, no speculation, BS=1 - 82.5 tok/s — vLLM, no speculation, BS=1 - 82.3 tok/s — SGLang + EAGLE3 (10K drafter), BS=1

>

We didn't run throughput benchmarks at higher batch sizes (e.g., BS=4, 8, 16, 32). At higher batch sizes, the GPUs saturate on the MoE expert computation and PCIe allreduce, so speculative decoding typically helps less (the verification step batches well, but the drafter becomes the bottleneck). Speculative decoding's main value is in the low-batch/latency regime where GPUs are underutilized during decode.

>

That said, it would be worth benchmarking BS=1, 2, 4, 8 both with and without EAGLE-3 after training completes — especially since on a PCIe system, the allreduce cost per decode step is fixed regardless of batch size, so speculation could still help at moderate concurrency.

>

Want me to plan a full benchmark matrix for after training finishes?

The message has three distinct movements: an admission of a gap, a justification for that gap, and a proposal to close it.

The Reasoning: Why Single-Stream Was the Priority

The assistant's justification reveals a deep understanding of where speculative decoding provides value. The key insight is that speculative decoding is primarily a latency optimization, not a throughput optimization. At batch size 1, the GPUs are severely underutilized during autoregressive decoding — each step generates one token while the massive parallel compute units of the GPU sit mostly idle. Speculative decoding exploits this idle capacity by running a small draft model to propose multiple tokens per step, then verifying them in parallel. The verification step is a single forward pass that can process all draft tokens simultaneously, effectively multiplying the tokens generated per GPU step.

At higher batch sizes, however, the GPUs become saturated with the MoE (Mixture-of-Experts) expert computation and the PCIe allreduce communication. Each decode step already processes many tokens (one per request in the batch), so the GPU utilization is higher. Adding speculative decoding on top means the drafter must run for each request, and the verification step must process draft tokens for all requests — the drafter itself becomes the bottleneck. The assistant correctly identifies this regime change: "the verification step batches well, but the drafter becomes the bottleneck."

This reasoning is sound and reflects standard wisdom in the speculative decoding literature. However, the assistant also introduces a nuance that complicates the picture: on a PCIe system, the allreduce cost per decode step is fixed regardless of batch size. This means that at moderate concurrency (e.g., BS=2 or 4), the allreduce overhead per token is still significant, and speculation could help amortize it. The assistant acknowledges this caveat, showing an awareness that the conventional wisdom might not fully apply to their specific hardware configuration.## Assumptions Embedded in the Benchmarking Strategy

The message reveals several assumptions that shaped the project's trajectory. The first and most consequential assumption is that single-stream performance is the right optimization target. This assumption was never explicitly debated — it was inherited from the standard framing of speculative decoding as a latency-reduction technique. For a chatbot or interactive application, single-stream latency is indeed the primary user-facing metric. But the Kimi-K2.5 model deployment might serve batch workloads (e.g., offline evaluation, data generation) where throughput at higher concurrency matters more. The assistant's admission that "we didn't run throughput benchmarks at higher batch sizes" means the team lacks data to make this tradeoff quantitatively.

A second assumption is that the 10K-sample drafter's poor performance (82.3 tok/s vs 90.0 baseline) was purely a function of low acceptance length (~2.1 tokens), and that training on more data with deeper tree-attention (TTT=5) would fix it. The assistant's reasoning at <msg id=4292> frames the problem as: "The goal with this 37K-sample, TTT=5 trained drafter is to get accept_len significantly higher (target ~3.0+), at which point speculation becomes net positive." This is a plausible hypothesis, but it assumes that acceptance length is the only bottleneck — that the drafter's architecture, the verifier's alignment, and the system's overhead are all adequate. If the drafter has fundamental architectural issues (e.g., the hidden state concatenation bug that was fixed at <msg id=4260-4261>), more data might not help.

A third assumption is that the PCIe allreduce cost is the dominant overhead. The assistant states that "each accepted token saves a full 8-GPU allreduce round" — implying that allreduce is the bottleneck worth optimizing. This is likely true for 8 GPUs on PCIe with the MoE architecture of Kimi-K2.5, but it's an assumption that could be validated with profiling data. The assistant doesn't cite any profiling of allreduce vs compute time.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs several pieces of context from the broader session:

  1. The EAGLE-3 architecture: EAGLE-3 is a speculative decoding framework where a small "draft" model proposes multiple future tokens, and the main "verifier" model checks them in parallel. The draft model is trained on hidden states extracted from the verifier, using a technique called "tree-attention training" (TTT) that learns to predict multiple plausible continuations.
  2. The hardware topology: 8 RTX PRO 6000 Blackwell GPUs connected via PCIe, not NVLink. This means allreduce communication between GPUs goes over the PCIe bus, which has higher latency and lower bandwidth than NVLink. This is the key constraint that makes speculation potentially valuable — each decode step requires synchronizing all 8 GPUs, and the allreduce cost is a fixed overhead per step regardless of how many tokens are generated.
  3. The previous 10K drafter's failure: The old drafter trained on only 10,000 samples achieved an acceptance length of ~2.1 tokens, meaning on average it predicted only about 1 token correctly out of 5 draft tokens. This made speculation slower than no speculation — the overhead of running the drafter and the verification step exceeded the savings from generating multiple tokens per step.
  4. The training pipeline status: At the time of this message, a new drafter was training on 37,312 samples with TTT=5, running on 4 GPUs at ~0.57 steps/s with an estimated 10.8-hour runtime. The training had just been optimized by adding batch packing (batch_size=8 with max_seq_len=8192) after a series of OOM and Triton shared-memory issues.
  5. The SGLang vs vLLM baseline: SGLang achieved 90.0 tok/s vs vLLM's 82.5 tok/s at single-stream, establishing SGLang as the preferred serving framework. The SGLang configuration included NCCL tuning and --num-continuous-decode-steps 4.

Output Knowledge Created by This Message

This message creates several important pieces of knowledge for the conversation:

  1. A documented benchmarking gap: The message formally acknowledges that throughput at higher batch sizes was never measured. This creates a clear action item for after training completes.
  2. A nuanced thesis about speculation on PCIe systems: The assistant articulates a specific claim — that the fixed allreduce cost per step makes speculation potentially valuable even at moderate concurrency on PCIe. This is a testable hypothesis that could inform future deployment decisions.
  3. A proposed benchmark matrix: BS=1, 2, 4, 8 both with and without EAGLE-3. This is a concrete plan that could be executed after training.
  4. A clarification of the optimization target: The message reinforces that single-stream latency is the primary target, while acknowledging that throughput at higher batch sizes is an open question. This helps align expectations between user and assistant.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message reveals a structured analytical approach. First, it acknowledges the gap honestly — "We didn't benchmark other batch sizes systematically" — without defensiveness. Then it provides the rationale for why single-stream was prioritized, showing an understanding of where speculative decoding provides value. Then it introduces a countervailing consideration (the fixed allreduce cost on PCIe) that complicates the simple story. Finally, it proposes a concrete next step.

This pattern — acknowledge, explain, complicate, propose — is characteristic of good technical reasoning. The assistant doesn't just say "we didn't benchmark that" and move on; it explains why that decision was made, why it might need revisiting, and how to revisit it. The offer to "plan a full benchmark matrix" also serves as a natural handoff to the user, giving them agency over whether to pursue this line of investigation.

Mistakes and Incorrect Assumptions

The message itself contains no factual errors — the numbers it reports (90.0, 82.5, 82.3 tok/s) are consistent with earlier benchmarks. However, the reasoning reveals a potential blind spot: the assistant assumes that the new drafter will achieve significantly higher acceptance length, but this is an untested hypothesis. The training was still running at the time of this message, and no validation metrics were yet available for the 37K-sample model.

More subtly, the assistant's framing of the PCIe allreduce cost as "fixed regardless of batch size" is slightly imprecise. While the per-step allreduce cost is indeed independent of batch size, the number of steps to generate a fixed number of tokens decreases at higher batch sizes (because each step generates tokens for all requests in parallel). So the allreduce cost per token does decrease with batch size, even if the per-step cost is fixed. The assistant's intuition is directionally correct — speculation helps more at low batch sizes — but the precise tradeoff depends on the ratio of allreduce cost to compute cost, which varies with batch size.

Conclusion

Message 4294 is a moment of honest reckoning in a complex engineering project. The assistant acknowledges a gap in the benchmarking strategy, explains the reasoning behind it, and proposes a path forward. It reveals the implicit assumptions that guided weeks of work — that single-stream latency is the right target, that acceptance length is the key bottleneck, that PCIe allreduce is the dominant cost — and invites the user to reconsider them. In doing so, it demonstrates the kind of reflective engineering thinking that separates a well-executed project from one that simply follows a plan without questioning its foundations. The message is a reminder that even in the midst of a long training run, stepping back to ask "what are we optimizing for, and do we have the data to know?" is always worth the time.