Benchmarking the Baseline: Measuring Native DFlash Throughput on Blackwell GPUs

Introduction

In the middle of a complex deployment saga spanning multiple machines, CUDA toolkit versions, and speculative decoding algorithms, there comes a moment when the engineer must stop debugging and start measuring. Message [msg 11217] captures precisely such a moment: after successfully launching a native SGLang DFlash (Draft-Flash) service on a CT200 machine equipped with eight RTX PRO 6000 Blackwell GPUs, the assistant runs a structured throughput benchmark to establish a performance baseline for the linear (non-tree) speculative decoding mode. This single message, consisting of a Python benchmarking script executed via a bash tool call, represents a critical transition from environment bootstrapping to quantitative evaluation — the moment where all the preceding infrastructure work is put to the test.

The Broader Context: A Deployment Odyssey

To understand why this message was written, one must appreciate the journey that led to it. The assistant had been working on deploying a speculative decoding system called DFlash, which uses a small drafter model to propose draft tokens that a larger target model verifies in parallel. A newer variant, DDTree (Draft Tree), extends this by maintaining a tree of possible draft sequences rather than a single linear chain, potentially achieving higher acceptance rates.

The deployment had been anything but smooth. The original target machine, CT129, suffered a GPU failure after a Triton crash, forcing a pivot to CT200 ([msg 11202]). CT200 had no SGLang installed at all — only a temporary standalone DDTree wrapper running on GPU0. The assistant built a new virtual environment from scratch, only to encounter a CUDA ABI mismatch: CT129's DFlash-capable SGLang was compiled against torch 2.11.0+cu130, but CT200 had +cu128. This was resolved by overlaying packages from CT129 ([chunk 62.0]). Then came an xgrammar import error, fixed by switching to --grammar-backend none ([msg 11198]). Then FlashInfer's JIT compiler rejected the SM120 architecture of the Blackwell GPUs, requiring a switch to --attention-backend triton ([msg 11203]). Each fix was a small battle, and by [msg 11206] the native DFlash service was finally healthy and generating coherent text at 123.5 tok/s.

But a single data point is not a benchmark. The assistant needed systematic measurements across diverse prompts to understand the true performance characteristics before comparing DFlash linear with DDTree.

Why This Message Was Written: The Motivation

The immediate trigger for this message was the assistant's stated intention in [msg 11215]: "Now a quick 3-request comparative benchmark: native DFlash vs DDTree shadow-linear, both on CT200." Message [msg 11216] then stopped the DDTree service and started the DFlash service. Message [msg 11217] executes the DFlash half of this comparison.

The deeper motivation, however, is more fundamental. In any optimization workflow, establishing a reliable baseline is the first and most critical step. Without knowing how fast the system runs in its current configuration, there is no way to measure improvement. The assistant had already seen a preliminary throughput number (123.5 tok/s for a single 128-token generation in [msg 11207]), but that measurement had several limitations: it used only one prompt, a single request, and a relatively short generation length. A proper baseline requires multiple prompts, multiple runs for statistical stability, and a consistent methodology.

The assistant's reasoning, visible in the code structure, reveals a methodical approach. The benchmark function bench() is designed as a reusable component — it takes a URL, model name, prompt, and parameters, runs n iterations, and returns both individual results and an average. This modularity suggests the assistant anticipated running this same function against multiple endpoints (DFlash and DDTree) and possibly across different configurations.

How Decisions Were Made: Benchmark Design Choices

Every line of the benchmarking script embodies a design decision, and examining these choices reveals the assistant's engineering judgment.

Choice of prompts: The three prompts span distinct categories — code generation ("Write a Python function fibonacci(n) using iteration"), algorithmic explanation ("Explain the quicksort algorithm in 3 sentences"), and a trivial factoid ("What is 2+2? Answer with just the number."). This diversity is intentional: different prompt types produce different output distributions, which in turn affect the drafter's acceptance rate and overall throughput. Code generation tends to produce structured, predictable output that a drafter can predict accurately, while open-ended explanation may be more variable. The "2+2" prompt with its instruction to answer with just a number tests the shortest-path generation where the model stops early (as seen in the results, where it generated only 159 tokens rather than the requested 256).

Choice of n=2: The function signature defaults to n=3, but the actual call uses n=2. This is a pragmatic tradeoff between statistical reliability and time. Two runs per prompt provide some noise reduction over a single measurement while keeping the total benchmark time under a minute. Given that the assistant was in an interactive debugging session where responsiveness matters, this is a reasonable compromise. However, two samples are far from sufficient for rigorous statistical analysis — a point we will return to.

Choice of max_tokens=256: This is longer than the earlier 128-token test, providing a more representative measurement of sustained generation throughput. Short generations are dominated by prompt processing and time-to-first-token, while longer generations better reflect the steady-state performance of the speculative decoding engine.

Choice of temperature=0: Using greedy decoding (temperature=0) ensures deterministic outputs, making the benchmark reproducible. This is standard practice for performance measurement, though it does not reflect the full range of production usage where sampling temperature varies.

Choice of timeout=300: The generous 300-second timeout per request indicates the assistant was accounting for potential slow initialization or cold-start effects. In practice, all requests completed in under 3 seconds, so this was a safety margin.

Choice of metrics: The benchmark reports three values per run — wall-clock seconds, completion tokens, and tokens per second — plus an average across runs. This is a sensible minimal set. Notably absent are time-to-first-token (TTFT), inter-token latency variance, and any measure of output quality or correctness. The focus is purely on throughput.

Assumptions and Potential Pitfalls

The benchmark rests on several assumptions that are worth examining critically.

Assumption of service stability: The benchmark assumes the service remains healthy throughout. Given the history of crashes documented in earlier messages (FlashInfer JIT failures, xgrammar import errors, signal 9 kills), this is not a trivial assumption. The fact that the service survived all six requests without crashing is itself a meaningful validation.

Assumption of representative workload: Three prompts, even if diverse, cannot capture the full range of production traffic. Real workloads include multi-turn conversations, system prompts, tool-use formatting, and varying generation lengths. The benchmark is a useful proxy but not a substitute for production load testing.

Assumption of negligible network overhead: The benchmark measures end-to-end latency including network round-trips between the client and the server. On a local network (the IP 10.1.2.200 suggests an internal cluster), this overhead is likely small (single-digit milliseconds), but it is not zero. For a more precise measurement, the assistant could have run the client on the same machine or used server-side timing.

Small sample size: With only two runs per prompt, a single outlier could significantly skew the average. The results show good consistency between runs (e.g., 139.7 vs 141.1 tok/s for the first prompt), suggesting low variance, but this is not guaranteed across all conditions.

No warmup: The benchmark does not include warmup requests. LLM serving systems often exhibit higher latency on the first few requests due to CUDA kernel compilation, cache population, and memory allocation. The results may therefore slightly underestimate steady-state performance.

Input Knowledge Required

To fully understand this message, the reader needs knowledge spanning several domains:

Speculative decoding: Understanding that DFlash uses a small drafter model to propose tokens that a larger target model verifies in parallel, achieving speedups by processing multiple draft tokens in a single forward pass. The "linear" variant uses a single chain of drafts, while DDTree maintains a tree structure.

SGLang architecture: Knowledge that SGLang is an LLM serving system with support for speculative decoding, tensor parallelism, and various attention backends (FlashInfer, Triton). The model is loaded from /dev/shm/Qwen3.6-27B, indicating a shared memory filesystem for fast model loading.

Hardware context: The RTX PRO 6000 Blackwell GPUs (compute capability SM120) are high-end workstation GPUs with specific CUDA requirements. The earlier FlashInfer issues arose because its JIT compiler did not recognize SM120 as a valid target.

Benchmarking methodology: Understanding why multiple runs, diverse prompts, and consistent parameters are necessary for reliable measurement.

Output Knowledge Created

The benchmark produces concrete, actionable data:

1. Code generation throughput: 140.4 tok/s average for the fibonacci prompt. This is the highest throughput of the three, likely because code generation produces structured, predictable output that the drafter can predict accurately, leading to high acceptance rates.
2. Explanation throughput: 94.6 tok/s for the quicksort explanation. This is significantly lower than code generation, suggesting that more varied, natural-language output reduces the drafter's effectiveness.
3. Short-answer throughput: 106.5 tok/s for the "2+2" prompt, but with only 159 tokens generated (the model stopped naturally rather than reaching 256). The effective throughput is higher than the explanation prompt but lower than code generation.
4. Consistency: The two runs for each prompt show remarkable consistency (differences of 1-4 tok/s), suggesting the system has stable performance characteristics.
5. Comparison to earlier measurement: The 140.4 tok/s for the fibonacci prompt at 256 tokens is higher than the 123.5 tok/s measured earlier at 128 tokens ([msg 11207]). This may reflect warmup effects or simply the difference in generation length. These numbers serve as the baseline against which DDTree will be compared. The implicit question hanging over this message is: can DDTree beat these numbers?

The Thinking Process Visible in the Code

The structure of the benchmarking script reveals the assistant's thought process. The bench() function is written generically — it takes a URL and model as parameters, suggesting the assistant planned to call it with different endpoints. The comment "=== Native DFlash (linear) ===" in the output implies a corresponding "=== DDTree ===" section was planned for a subsequent call. The script prints "done_dflash" as a clear delimiter, making it easy to parse the output.

The choice to define prompts as a list and iterate over them, rather than hardcoding individual calls, shows an awareness of the need for systematic, repeatable measurement. The assistant is thinking like an engineer designing an experiment, not just a user running ad-hoc tests.

The error handling is minimal — there is no retry logic, no timeout handling beyond the 300-second request timeout, and no validation that the model name in the response matches the requested model. This is appropriate for a quick benchmark in a controlled environment, but it also reflects the assistant's confidence that the service is now stable after the earlier debugging.

Conclusion

Message [msg 11217] is a deceptively simple artifact — a Python script that sends six HTTP requests and prints throughput numbers. But in the context of the broader conversation, it represents a crucial inflection point. After hours of debugging CUDA toolchains, patching SGLang source files, and wrestling with GPU architecture incompatibilities, the assistant finally has a working system and is measuring its performance. The numbers — 94 to 140 tok/s depending on prompt type — establish the baseline that will be used to evaluate the DDTree optimization. The message embodies the engineering discipline of "measure before you optimize," and the results it produces will directly inform the next phase of the work: tuning DDTree budgets, sweeping tensor-parallel configurations, and ultimately producing the comprehensive benchmark report that the user requested. In a single message, the assistant transitions from firefighting to science.