The First DFlash Benchmark: Validating Speculative Decoding for Kimi K2.6 on Blackwell

Introduction

In the high-stakes world of large language model inference, every tok/s matters. When you're running an 8× GPU machine with RTX PRO 6000 Blackwell cards connected only by PCIe, the difference between a well-tuned deployment and a naive one can be a factor of 5× or more in throughput. This message — message index 11554 in the conversation — captures the precise moment when the assistant finally deployed a Kimi K2.6 model augmented with DFlash speculative decoding and ran the first comprehensive benchmark to see whether the speculative drafter was actually worth the complexity.

The message is a single bash command executing an inline Python script that sends a carefully designed sequence of HTTP requests to an SGLang inference server running on a remote machine (CT200, IP 10.1.2.200). It measures throughput at single-request concurrency across five diverse prompts, then sweeps concurrency from 1 to 256 concurrent requests. The output, partially visible in the message, reveals a system achieving 86–90 tok/s on long generations at C=1 and scaling to hundreds of tok/s under load. But the numbers tell only part of the story. This message is the culmination of hours of debugging, infrastructure recovery, parallelism tuning, and a critical CUDA graph incompatibility that forced a last-minute configuration change. It represents the first real validation of the DFlash speculative decoding approach on this particular hardware stack, and its results would shape the direction of the entire project.

The Road to This Benchmark

To understand why this message exists, one must understand the journey that preceded it. The assistant had spent the better part of a day systematically benchmarking Kimi K2.6 — a Mixture-of-Experts (MoE) model — across four parallelism strategies on 8× RTX PRO 6000 GPUs connected only by PCIe (no NVLink). The results were illuminating: pure tensor parallelism (TP8) achieved 98 tok/s at C=1 with CUDA graphs but plateaued at ~1291 tok/s aggregate; expert parallelism with TP2 groups (EP4) reached 1530 tok/s at high concurrency, winning the throughput crown. The key insight was that expert parallelism eliminated the AllReduce bottleneck on MoE layers across PCIe, since each GPU handled a subset of experts independently.

But the user wanted more. They directed the assistant to download a DFlash drafter model from HuggingFace — SubSir/Kimi-K2.6-DFlash-tmp-long, a 6.5 GB speculative decoding drafter with block_size=8, 6 draft layers, and BF16 precision — and deploy it with SGLang's DFlash speculative decoding support. The promise of DFlash is that a small "drafter" model generates multiple candidate tokens cheaply, which the base model then verifies in parallel, achieving higher throughput than autoregressive generation alone.

The deployment hit a snag immediately. The first attempt failed with a NoneType error during CUDA graph replay in the DFlash worker. The assistant diagnosed this as a compatibility issue between the DFlash speculative worker and SGLang's CUDA graph capture mechanism, and made the critical decision to add --disable-cuda-graph to the service configuration. This was a pragmatic tradeoff: sacrificing the ~3× kernel launch speedup from CUDA graphs in exchange for a working DFlash deployment. After restarting, the service took 210 seconds to load the 72 GB model across 8 GPUs and become ready. The subject message is the very next action after that readiness confirmation.

Anatomy of the Benchmark Script

The Python script embedded in the bash command is a carefully constructed benchmark harness. Let us examine its design choices, because they reveal the assistant's assumptions and priorities.

The api() function sends a chat completion request to the OpenAI-compatible endpoint at http://10.1.2.200:30001/v1/chat/completions. It uses urllib.request — a standard library choice that avoids external dependencies — and measures wall-clock time with time.perf_counter(). The function returns three values: throughput in tok/s, wall time in seconds, and completion token count. The use of time.perf_counter() rather than time.time() indicates attention to measurement precision, as perf_counter is monotonic and has higher resolution.

The warmup phase sends two "Say OK" requests with max_tokens=16. This is a standard practice in ML inference benchmarking: the first request after server startup often includes JIT compilation overhead, Triton kernel warmup, and CUDA graph initialization (even though graphs were disabled, other warmup effects remain). By discarding warmup results, the benchmark measures steady-state performance.

The five prompts are worth examining. They are not random; they are designed to probe different generation characteristics:

The Results and Their Meaning

The C=1 results show significant variance across prompts. The quicksort request achieved 86.0 tok/s, the relativity explanation 89.5 tok/s, the "2+2" question only 51.1 tok/s with just 36 tokens generated, the haiku 71.1 tok/s, and the JSON parser 65.5 tok/s. The low throughput on the "2+2" prompt is expected — with only 36 tokens generated, the measurement is dominated by prefill time (processing the input prompt) rather than decode time. The 86–90 tok/s on long generations is the meaningful number: it represents the steady-state decode throughput of the DFlash-augmented system.

How does this compare to the autoregressive baselines? The EP8 autoregressive configuration achieved 65.2 tok/s at C=1, and the TP8 tuned configuration with CUDA graphs reached 97.9 tok/s. The DFlash result of ~86 tok/s sits between these two. This is slightly disappointing — the speculative drafter was expected to provide a clear win over autoregressive decoding. The assistant's decision to disable CUDA graphs likely explains the gap: without graphs, each decode step incurs full kernel launch overhead, eroding the advantage that DFlash's batched verification should provide.

The concurrency sweep begins promisingly. At C=1 with 2048 max tokens, aggregate throughput is 70.2 tok/s (lower than the C=1 short-prompt results because the requests are generating the full 2048 tokens). At C=4, throughput jumps to 133.4 tok/s — nearly double, showing that batching across multiple requests is effective. At C=8: 251.7 tok/s, C=16: 427.2 tok/s. The scaling is sub-linear but healthy, suggesting the system can effectively batch concurrent requests despite the PCIe interconnect.

The message output is truncated, so we cannot see the full sweep. However, the next message ([msg 11555]) reveals the server-side acceptance metrics: acceptance length of ~3.46–3.49 tokens per step, acceptance rate of ~0.35–0.36, and generation throughput of ~1133–1137 tok/s at high concurrency. An acceptance length of 3.5 tokens means that on average, the drafter proposes several tokens and the base model accepts about 3.5 of them per verification step. This is a reasonable result — not spectacular, but solid validation that the DFlash drafter is working correctly.

Assumptions and Their Consequences

Several assumptions underpin this benchmark, and they deserve scrutiny.

First, the assistant assumed that disabling CUDA graphs was the correct fix for the DFlash crash. This was a pragmatic decision based on the error signature (a NoneType in graph replay), but it meant accepting a significant performance penalty. The subsequent B300 benchmarks (in chunk 2 of this segment) would reveal that CUDA graphs provide roughly a 3.8× speedup in eager mode, meaning the DFlash results here are operating at a fraction of their potential. The assistant implicitly assumed that the graph issue was a fundamental incompatibility rather than a configuration bug, and that assumption shaped all the results in this message.

Second, the benchmark assumes that the EP8 parallelism configuration is the right choice for DFlash. The assistant chose EP8 because it had the best single-request latency among the autoregressive baselines (65 tok/s vs TP8's 26 tok/s without graphs). But DFlash changes the compute profile: the verify step processes all draft tokens in parallel, which may benefit from different parallelism strategies. The assistant did not re-optimize parallelism for the DFlash case, instead reusing the EP8 configuration from the autoregressive benchmarks.

Third, the benchmark design assumes that the OpenAI-compatible API endpoint is a reliable measurement instrument. The usage.completion_tokens field from the SGLang server is used to compute throughput, but this field counts tokens generated by the base model, not the drafter. The overhead of running the drafter (its forward passes, the tree construction, the verification logic) is invisible to this measurement. If the drafter consumes GPU compute that could otherwise serve other requests, the aggregate throughput might be lower than the raw tok/s suggests.

Fourth, the assistant assumed that the server logs would provide acceptance metrics, as indicated by the placeholder print("\nAcceptance check (server logs):", flush=True) followed by no actual log-gathering code in this script. The acceptance check was deferred to a separate command in the next message. This is a minor oversight — the benchmark script was written quickly and the log analysis was split across messages — but it means the acceptance metrics are not synchronized with the throughput measurements.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Speculative decoding: The concept that a small draft model generates candidate tokens which a large base model verifies in parallel. DFlash is a specific variant that uses a "draft" model with a block-wise prediction structure (block_size=8 in this case).

SGLang architecture: Knowledge that SGLang is a serving framework for LLMs with support for tensor parallelism (TP), expert parallelism (EP), CUDA graphs, continuous batching, and speculative decoding via the --speculative-algorithm DFLASH flag.

PCIe vs NVLink bottlenecks: Understanding that PCIe interconnect creates a communication bottleneck for AllReduce operations in tensor parallelism, which expert parallelism avoids by keeping expert computation local to each GPU.

CUDA graphs: Knowledge that CUDA graphs capture GPU kernel launches and replay them with minimal CPU overhead, providing significant speedups for inference workloads with predictable execution patterns.

The hardware context: 8× RTX PRO 6000 Blackwell GPUs, each with substantial memory, connected via PCIe only (no NVLink), running Ubuntu 24.04 with CUDA 13.0 toolkit.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. DFlash on EP8 achieves ~86 tok/s at C=1 on PCIe Blackwell, compared to 65 tok/s for autoregressive EP8 and 98 tok/s for TP8 with CUDA graphs. This establishes the baseline for speculative decoding on this hardware.
  2. The DFlash drafter works correctly with SGLang despite the CUDA graph incompatibility. The acceptance length of ~3.5 tokens confirms the drafter is generating useful proposals.
  3. Concurrency scaling is healthy but sub-linear: 70 tok/s at C=1 grows to 427 tok/s at C=16, showing that the system benefits from request batching but with diminishing returns.
  4. The CUDA graph issue is a critical blocker: The decision to disable graphs was necessary for stability but came at a performance cost. This knowledge directly motivated the subsequent investigation into CUDA graph compatibility on the B300 platform, where the team would eventually discover a sm_103-specific graph bug.
  5. The benchmark methodology is validated: The multi-prompt, multi-concurrency approach provides a comprehensive picture of system performance, revealing both strengths (good single-request throughput on code generation) and weaknesses (variance across prompt types).

The Thinking Process Visible in the Message

The message reveals a methodical, measurement-driven approach. The assistant does not simply run a single benchmark and declare success. Instead, it:

  1. Warms up the server to ensure steady-state measurements
  2. Tests diverse prompts to probe for prompt-dependent behavior
  3. Sweeps concurrency systematically from 1 to 256
  4. Handles errors gracefully with try/except and timeout logic
  5. Prints intermediate results with flush=True for real-time visibility The choice of concurrent.futures.wait() with a timeout rather than the simpler as_completed() iterator shows an awareness of real-world failure modes: if a request hangs indefinitely, the benchmark should still complete and report partial results. The use of urllib.request rather than the requests library (which is not installed by default in many environments) shows an assumption about the target environment's package availability. The assistant is writing code that will work without additional pip install steps. The placeholder print("\nAcceptance check (server logs):", flush=True) followed by no log-gathering code is a telling detail. It suggests the assistant intended to include server log parsing in this script but deferred it, perhaps because the logs were not immediately available or because the script was already long enough. The next message ([msg 11555]) completes this task by running a separate journalctl command.

Conclusion

Message 11554 is a pivotal moment in the conversation. It is the first time the DFlash speculative decoding pipeline is tested end-to-end on the Kimi K2.6 model with 8× Blackwell GPUs. The results are mixed but informative: the drafter works correctly and provides reasonable acceptance rates, but the CUDA graph incompatibility limits its throughput advantage over optimized autoregressive configurations. The benchmark methodology is sound, the measurements are credible, and the knowledge created directly informs the next phase of the project — including the pivot to the B300 NVLink platform where CUDA graphs would eventually be made to work with DFlash, achieving 303 tok/s at C=1 and a 2.15× speedup over autoregressive.

This message also exemplifies the assistant's working style: pragmatic, measurement-driven, and willing to make tradeoffs (disabling CUDA graphs) to get a working system rather than chasing theoretical optimality. The benchmark is not the final word on DFlash performance — it is the first data point in an iterative optimization process that would span multiple platforms, multiple parallelism strategies, and multiple debugging sessions. But without this message, none of that subsequent work would have a baseline to improve upon.