The Baseline That Changed Everything: Measuring SGLang DDTree Throughput on Blackwell
In the middle of an intensive coding session to build a native C/C++/CUDA speculative decoding engine for the Kimi K2.6 model, the assistant executed a single bash command that would fundamentally reshape the trajectory of the entire project. Message [msg 11992] appears, at first glance, to be a routine benchmark run: copy a Python script to a remote server, execute it against a live SGLang DDTree service, and record the throughput numbers. But this seemingly straightforward measurement turned out to be the moment of truth — the data point that forced a critical re-evaluation of what bottlenecks actually mattered.
The Road to the Benchmark
The context leading to this message is essential for understanding its significance. Over the preceding hours, the assistant had built an entire native DDTree inference engine from scratch — a new kdtree-engine/ repository containing custom CUDA kernels for tree building, tree verification with MLA-absorb attention, and greedy tree acceptance. All 27 kernel tests passed bit-exact against numpy references. The GPU tree builder was clocking speedups of 6–13× over SGLang's CPU implementation at batch size 1, and an astonishing ~470× at batch size 64 ([msg 11989]). These were dramatic numbers that seemed to promise transformative end-to-end improvements.
But there was a gap in the evidence. The kernel microbenchmarks measured isolated operations in controlled conditions. The GPU-vs-CPU tree build comparison measured a single pipeline stage in artificial isolation. What was missing was the real-world baseline: how fast was the existing SGLang DDTree service actually running on the production hardware? Without that number, there was no way to know whether the GPU kernel's speedup would translate to meaningful end-to-end gains, or whether it was optimizing a part of the pipeline that wasn't the real bottleneck.
The assistant recognized this gap explicitly in its reasoning at [msg 11990]: "Now I need to get a fresh baseline measurement of SGLang's DDTree throughput at various concurrency levels on the live service to validate these improvements in a real-world scenario." This was the motivation for the message.
What the Message Does
The message is a single bash command block that performs two actions:
First, it copies a newly written benchmark script to the CT200 server (the 8× RTX PRO 6000 Blackwell machine running the live SGLang service):
scp -q -o StrictHostKeyChecking=no kdtree-engine/python/bench_service.py root@10.1.2.200:/root/kdtree-engine/python/This bench_service.py script had been written in the immediately preceding message ([msg 11991]) because no benchmark harness existed on CT200. The assistant created it from scratch — a lightweight concurrent request generator that hits the OpenAI-compatible completions endpoint at various concurrency levels and measures throughput in decoded tokens per second.
Second, it executes that script against the live service:
timeout 300 ssh -o StrictHostKeyChecking=no root@10.1.2.200 '/root/venv_sglang211/bin/python /root/kdtree-engine/python/bench_service.py --max-tokens 256 --concurrency 1 4 8 10 2>&1'The parameters are carefully chosen: --max-tokens 256 ensures each request generates a fixed-length output, making throughput measurements comparable. The concurrency sweep of 1, 4, 8, and 10 captures both single-user latency and multi-user throughput scaling. The b8 t4 in the echo label refers to the SGLang service configuration — batch size 8 and tree budget 4, the current production settings for the DDTree speculative decoding pipeline.
The Results
The output is clean and unambiguous:
conc agg_tok/s per_req_tok/s tokens wall_s
1 135.7 135.7 256 1.89
4 318.0 81.9 1024 3.22
8 479.6 61.8 2048 4.27
10 516.2 53.3 2560 4.96At single-user concurrency, the service delivers 135.7 tokens per second — a strong baseline for a 1-trillion-parameter MoE model running on 8 Blackwell GPUs. Throughput scales nearly linearly up to 8 concurrent requests (479.6 tok/s, 3.5× the single-user rate), then begins to show diminishing returns at 10 concurrent requests (516.2 tok/s, only 8% more than at C=8). Per-request throughput drops predictably from 135.7 tok/s at C=1 to 53.3 tok/s at C=10, reflecting the expected contention for GPU compute resources.
These numbers are the "before" measurement — the reference point against which every optimization would be judged.
The Deeper Significance
What makes this message pivotal is not the numbers themselves, but what the assistant did with them next. In the following message ([msg 11993]), the assistant performed a critical analysis that fundamentally changed the project's direction.
Working through the math: at C=10 with 516 tok/s aggregate throughput and an average acceptance rate around 4.5 tokens per verify step, each step processes roughly 45 tokens total, giving about 87ms per step. The CPU tree build at batch size 10 takes only ~0.8ms — less than 1% of the step time. Even at C=64, where the step time stretches to ~288ms, the CPU tree build is still just 2% of the total.
The conclusion was sobering: the GPU tree builder's 470× speedup, impressive as it was in isolation, would only shave 1–2% off the end-to-end latency at current batch sizes. The real bottleneck was the 1-trillion-parameter MoE forward pass, which is HBM-bound and dominates the step time. The tree build simply wasn't the constraint.
This insight — that the kernel the assistant had spent hours perfecting was optimizing a part of the pipeline that wasn't the bottleneck — is a classic lesson in systems optimization. It's the reason real-world baselines matter more than microbenchmarks. The assistant could have rushed to integrate the GPU tree builder into SGLang, risking service disruption for a 1% gain. Instead, the baseline measurement prevented that mistake.
Knowledge Required and Knowledge Created
To fully understand this message, the reader needs to know: what SGLang is (an inference engine for large language models), what DDTree is (a speculative decoding technique using dynamic draft trees), what the concurrency parameter means in a serving context, and the hardware configuration (8× RTX PRO 6000 Blackwell GPUs with NVLink). The reader also needs the context of the preceding messages — that the assistant had built custom CUDA kernels and was evaluating their impact.
The message creates new knowledge of the highest order: a verified, reproducible baseline measurement of the production system's throughput. This baseline becomes the foundation for all subsequent optimization decisions. It also implicitly validates that the SGLang DDTree deployment is functioning correctly — the service is stable, responsive, and delivering competitive throughput.
Assumptions and Reasoning
The assistant made several assumptions in this message. It assumed the SGLang service on CT200 was running and healthy (it was — the benchmark completed successfully). It assumed the bench_service.py script would work correctly on the remote host (it did — the Python environment was properly configured). It assumed that 256-token generations with greedy decoding would provide a representative throughput measurement (a reasonable choice for a baseline).
The reasoning visible in the assistant's thinking shows a methodical approach: identify the need for a baseline, create the tool to measure it, execute the measurement, and then — crucially — analyze the results honestly even when they challenge the expected narrative. The assistant could have ignored the baseline and proceeded with integration. Instead, it let the data guide the decision.
Conclusion
Message [msg 11992] is a masterclass in empirical systems engineering. A single benchmark run, executed with careful methodology, produced the data point that prevented a costly misallocation of engineering effort. The GPU tree builder was not abandoned — it remains essential for the native engine and for future scaling to higher batch sizes — but the immediate priority shifted. The assistant pivoted from "integrate the GPU kernel into SGLang" to "document the findings honestly and focus on the real bottleneck." This is the kind of decision that separates effective engineering from mere activity, and it all started with a simple bash command that asked: how fast are we actually going right now?