The Benchmark That Broke the Camel's Back: DFlash Speculative Decoding Falls Short

Introduction

In the high-stakes world of large language model serving, speculative decoding promises a tantalizing proposition: use a small, fast "drafter" model to propose tokens that a larger "target" model can accept or reject, achieving the quality of the large model at a fraction of the latency. But when theory meets practice, the gap can be brutal. Message [msg 6958] captures the moment when that gap became undeniable—a throughput benchmark of the DFlash speculative decoding system on Qwen3.6-27B that was so disappointing the user aborted it mid-run.

The message is deceptively simple: a single bash command executing a Python benchmark script, followed by two rows of results and a user abort. But beneath this surface lies a pivotal moment in a much larger story—one about the chasm between research prototypes and production deployments, the hidden costs of architectural incompatibility, and the moment when a team realized they needed to go back to the drawing board.

The Message in Full

The assistant executes the following command:

python3 /home/theuser/glm-kimi-sm120-rtx6000bw/bench_qwen36_vllm.py --mode throughput --concurrency "1,2,4,8,16,32,64,100" --input-len 1000 --output-len 500

The output shows a warmup completing successfully (50 tokens, ok=True), then the throughput sweep begins:

=== Throughput Sweep: input=1000, output=500 ===
   C |  Agg tok/s |  Per-req tok/s |  TTFT(s) |   Tokens |  Wall(s) |   Ok | Fail
-------------------------------------------------------------------------------------
   1 |       17.7 |           18.1 |     0.64 |     2000 |    112.8 |    4 |    0
   2 |       30.4 |           16.0 |     1.49 |     2000 |     65.7 |    4 |    0

Then the <bash_metadata> tag records: "User aborted the command."

Context and Motivation: Why This Benchmark Was Run

To understand the significance of this message, we must reconstruct the chain of events that led to it. The assistant had spent the preceding dozens of messages—spanning multiple chunks and segments—building toward this moment.

The journey began with the migration of Qwen3.6-27B from a decommissioned host (kpro6) to a new one (kpro5). The initial deployment used SGLang 0.5.9, which produced degenerate output due to incompatible GDN hybrid attention handling. Upgrading to SGLang 0.5.11 resolved this, achieving a respectable 73.5 tok/s single-request throughput with MTP (Medusa-Tree-Pruning) speculation using NEXTN steps=3. This was the baseline—the bar that any new speculative decoding method had to clear.

The team then pivoted to DFlash, a more aggressive speculative decoding approach that uses a small transformer drafter (5 layers, 3.3 GB) to propose tokens, combined with tree attention for exploring multiple candidate paths simultaneously. The drafter weights were obtained from the gated z-lab/Qwen3.6-27B-DFlash repository on HuggingFace. A custom config.json was crafted from scratch, with the target_layer_ids initially guessed (later corrected). After resolving a cascade of issues—missing flash-attn modules, namespace package conflicts between flash-attn v2 and v4, and stale log files—vLLM 0.20.1 was successfully launched with DFlash enabled.

A smoke test confirmed the model produced coherent output with proper reasoning content: a Python function for primality testing, 1402 tokens, finish reason "stop." The system was alive. But was it fast?

The benchmark script bench_qwen36_vllm.py was written specifically for this moment ([msg 6957]), adapted from an earlier SGLang-specific script that used the /generate endpoint. The new version targeted the OpenAI-compatible /v1/completions endpoint, which works with both vLLM and SGLang. The benchmark was designed to sweep concurrency levels from 1 to 100, measuring aggregate tokens per second, per-request throughput, time-to-first-token (TTFT), and success rates.

What the Numbers Reveal

The results, though sparse, are devastating when compared to the MTP baseline.

At concurrency 1: DFlash achieves 17.7 tok/s aggregate throughput with a per-request rate of 18.1 tok/s and a TTFT of 0.64 seconds. The MTP baseline achieved 73.5 tok/s—a 4.2x advantage for the simpler method.

At concurrency 2: DFlash reaches 30.4 tok/s aggregate, with per-request throughput dropping to 16.0 tok/s and TTFT rising to 1.49 seconds. The aggregate throughput scales sub-linearly (only 1.7x improvement for doubling concurrency), and per-request throughput degrades, suggesting the system is already hitting a bottleneck.

The TTFT numbers are also telling. At concurrency 1, 0.64 seconds is reasonable for a 27B model on 2x A6000 GPUs. But at concurrency 2, TTFT jumps to 1.49 seconds—a 2.3x increase for only doubling the request count. This suggests queuing delays or memory bandwidth contention, likely exacerbated by the drafter model's overhead.

Why the User Aborted

The user's decision to abort after only two data points is itself a data point. It signals that the results were so far below expectations that completing the sweep was not worth the time. The benchmark was configured to test concurrency levels up to 100—a 10-minute-plus run at the observed rates. Seeing 17.7 tok/s at concurrency 1 versus the 73.5 tok/s MTP baseline, the user likely concluded that the DFlash configuration was fundamentally broken and that higher concurrency would only confirm the failure.

This interpretation is supported by the broader narrative. Earlier in the session (chunk 0 of segment 43), the assistant had investigated DFlash's acceptance rate and found it catastrophically low—around 1.1%. Three root causes were identified:

  1. A layer-ID +1 offset missing in vLLM's hidden state extraction (fixed by PR #40727)
  2. Sliding window attention (SWA) layers in the drafter being ignored (fixed by PR #40898)
  3. Possible eagle cache drop issues The assistant had installed vLLM from the unmerged PR #40898 branch, which included all three fixes. But the benchmark suggests the fixes were either incomplete, not properly activated, or the fundamental issue was deeper—perhaps the drafter model itself, labeled "still under training" on HuggingFace, simply wasn't good enough to produce acceptable tokens.

Assumptions and Their Consequences

Several assumptions underpin this benchmark, and several proved incorrect.

Assumption 1: The PR #40898 fixes would resolve the acceptance rate problem. The assistant assumed that patching the layer-ID offset and SWA handling would bring DFlash's acceptance rate to a useful level. The benchmark results suggest otherwise—either the patches didn't fully work, or the drafter's quality was the bottleneck.

Assumption 2: DFlash would outperform MTP. The entire pivot from MTP to DFlash was motivated by the belief that tree-based speculative decoding would yield higher throughput. The benchmark shattered this assumption: DFlash was 4x slower than MTP at single-request throughput.

Assumption 3: The benchmark script would run to completion. The user's abort suggests the results were so bad that continuing was pointless. This implies an expectation that DFlash would at least approach MTP's performance.

Assumption 4: vLLM's DFlash implementation was production-ready. The code depended on unmerged PRs and custom patches. The benchmark revealed that the implementation, while functional for correctness, was not optimized for performance.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message produced critical empirical data:

  1. DFlash throughput on 2x A6000 with Qwen3.6-27B: ~17.7 tok/s at concurrency 1, ~30.4 tok/s at concurrency 2.
  2. TTFT characteristics: 0.64s at concurrency 1, rising to 1.49s at concurrency 2.
  3. Scaling behavior: Sub-linear throughput scaling with concurrency, suggesting early saturation.
  4. Comparative performance: DFlash is ~4x slower than MTP speculation for this model/hardware combination. This data effectively ended the DFlash deployment effort and redirected the project toward training a better drafter—a pivot that consumed the remainder of the session.

The Deeper Significance

This message is a case study in the gap between research and production. DFlash and DDTree are published methods with promising theoretical properties—tree-based speculative decoding should, in principle, outperform linear-chain methods like MTP. But the reality of unmerged PRs, custom configs, and "still under training" model weights creates a chasm that engineering effort alone cannot bridge.

The user's abort was not just impatience—it was a strategic decision. The 17.7 tok/s result told them everything they needed to know: the current DFlash setup was not viable, and the path forward required either fixing the drafter (training a better one) or abandoning the approach. They chose the former, pivoting to building a hidden state extraction pipeline for training a custom DFlash drafter—a project that occupied the subsequent chunks of the session.

In this light, message [msg 6958] is the turning point. It is the moment when the team stopped trying to deploy existing speculative decoding methods and started building the infrastructure to create better ones. The benchmark didn't just measure throughput—it measured the gap between what was available and what was needed, and that gap was too large to ignore.