The Validation Benchmark: Confirming a CUDA Graph Fix for DDTree Speculative Decoding

Introduction

In the high-stakes world of large language model serving, a single tensor dimension mismatch can transform a promising speculative decoding pipeline into a silent crash. Message 11618 captures the exact moment when an AI assistant, having just diagnosed and repaired a subtle CUDA graph sizing bug in SGLang's DDTree (Draft-Tree) implementation, runs a comprehensive benchmark to validate the fix and measure the resulting throughput. This message is not merely a test run—it is the culmination of a multi-step debugging journey, a scientific measurement of a complex system, and a data-gathering exercise that will inform the next phase of architectural decisions.

The message contains a Python benchmark script executed against a deployed Kimi K2.6 model with DDTree speculative decoding on an 8× RTX PRO 6000 Blackwell PCIe machine (hostname CT200). The results—approximately 50 tokens/second at concurrency 1, scaling to roughly 88 tok/s at C=64—tell a nuanced story about the interaction between tree-structured speculation, tensor parallelism, and PCIe interconnect bottlenecks.

The Road to This Message

To understand why this benchmark exists, one must trace the debugging trail that precedes it. In the messages leading up to message 11618, the assistant had deployed the K2.6 model with DDTree speculative decoding on CT200, only to discover that the service crashed on the first real request with a cryptic error:

RuntimeError: The size of tensor a (8) must match the size of tensor b (33) at non-singleton dimension 0

This error ([msg 11604]) revealed a fundamental mismatch: the CUDA graph for the target model's verification forward pass had been pre-captured assuming 8 tokens per batch (the DFlash block size), but DDTree's verify forward actually passes 33 tokens per batch (budget + 1, where budget=32). The CUDA graph runner was using speculative_num_draft_tokens (set to block_size=8) to size its buffers, regardless of whether the graph was for the draft worker or the target verify worker.

The assistant traced through the SGLang source code ([msg 11606] through [msg 11611]), examining cuda_graph_runner.py, server_args.py, and spec_info.py to understand how num_tokens_per_bs was set. The root cause was clear: DDTree has two distinct token counts—the draft forward uses block_size=8, but the target verify forward uses budget+1=33—and the code treated them as identical.

The fix ([msg 11613]) added conditional logic: for DDTree target verify workers, set num_tokens_per_bs = speculative_ddtree_budget + 1; for DFlash draft workers, keep using speculative_num_draft_tokens. After applying the patch, restarting the service, and waiting 645 seconds for the 590 GB model to load ([msg 11615]), the assistant confirmed the service was alive and the CUDA graph was active during decode. Message 11616 captured the relief: "The fix works — cuda graph: True during DDTree decode, no crash."

But a single successful request is not proof of robustness. The assistant needed systematic validation: steady-state throughput measurements across diverse prompts and concurrency levels. That is precisely what message 11618 delivers.

Anatomy of the Benchmark

The Python script embedded in message 11618 is a carefully constructed measurement instrument. Let us examine its design choices.

Endpoint and model: The script targets http://10.1.2.200:30001/v1/chat/completions with model /root/models/Kimi-K2.6. This is the OpenAI-compatible API served by SGLang, running on CT200—the PCIe-connected PRO6000 machine with 8 GPUs.

The api() function: A closure that serializes a request, sends it via urllib.request, parses the JSON response, and returns three values: tokens/second (throughput), wall-clock time, and completion token count. The use of time.perf_counter() for timing indicates a concern for precision. The 600-second timeout per request is generous, reflecting the assistant's expectation that the service might be slow under load.

Warmup: Two short requests ("Say OK", max_tokens=16) precede the real measurements. This is not mere ceremony—CUDA graphs must be captured during the first few requests, and the warmup ensures that subsequent measurements reflect steady-state performance rather than one-time compilation overhead.

Single-request benchmarks: Five diverse prompts—quicksort implementation, theory of relativity explanation, a simple arithmetic question, a programming haiku, and a JSON parser—are each sent with max_tokens=512. The diversity is deliberate: different prompt types may trigger different model behaviors (code generation vs. factual recall vs. creative writing), and the assistant wants to measure whether throughput varies significantly across use cases.

Concurrency sweep: Starting from C=1 and doubling through C=8, 16, 32, 64, and 128, the script uses concurrent.futures.ThreadPoolExecutor to send multiple requests in parallel. Each request asks for 2048 tokens, and the overall timeout is 900 seconds. The aggregate throughput (total tokens divided by wall time) is the key metric. This sweep reveals how the system scales under load—whether it maintains per-request throughput or degrades due to contention.

The Results: What the Numbers Say

The single-request results show throughput ranging from 42.3 to 49.6 tok/s across the five prompts, with completion times between 3.4 and 11.0 seconds for 512 tokens. The variation is modest—about 15% between the best and worst cases—suggesting that prompt type has limited impact on generation speed for this configuration.

The concurrency sweep tells a more interesting story:

| Concurrency | Aggregate Throughput | Wall Time | |------------|-------------------|-----------| | C=1 | 50.4 tok/s | 40.6s | | C=8 | 79.8 tok/s | 159.6s | | C=16 | 60.0 tok/s | 86.2s | | C=32 | 76.4 tok/s | 108.5s | | C=64 | 87.6 tok/s | (truncated) |

The scaling is poor. Moving from C=1 to C=64 increases aggregate throughput by only 1.74×, and the scaling is non-monotonic—C=16 is actually worse than C=8. This is characteristic of a system where per-request throughput degrades significantly under concurrency, likely due to GPU memory bandwidth contention or PCIe AllReduce overhead.

Compare these numbers to the earlier EP8 DFlash results from the same session: 86 tok/s at C=1 and ~1146 tok/s aggregate. The TP8+DDTree configuration achieves barely 10% of the aggregate throughput of EP8+DFlash. This is not a failure of the CUDA graph fix—the service runs without crashing—but rather a confirmation that TP8 on PCIe is fundamentally ill-suited for this workload.

What This Benchmark Assumes

Every measurement rests on assumptions, and this benchmark is no exception.

The service is stable: The assistant assumes that the 645-second startup was sufficient and the service will remain healthy throughout the benchmark. The results bear this out—no crashes occurred—but the assumption is tested anew with each concurrent request.

Warmup is sufficient: Two short requests may not fully capture all CUDA graph variants. Different batch sizes, sequence lengths, and tree configurations might trigger different graph captures during the benchmark itself, introducing measurement noise.

The prompts are representative: Five prompts, however diverse, cannot capture the full distribution of real-world usage. The assistant implicitly assumes that throughput is relatively invariant to prompt content—an assumption the results partially validate.

Network overhead is negligible: The benchmark runs from a client that is presumably on the same network as CT200. Any network latency or bandwidth limitation would be conflated with model inference time. The use of a local HTTP API mitigates this, but does not eliminate it.

The concurrency model is realistic: ThreadPoolExecutor with blocking HTTP requests creates a specific pattern of concurrent load. Real serving systems might see different behavior with continuous streaming or varying request sizes.

The Unspoken Narrative

What makes message 11618 compelling is not just the data it produces, but what it reveals about the assistant's decision-making process.

The assistant chose to benchmark TP8+DDTree despite earlier evidence that EP configurations were superior on PCIe. Why? Because the CUDA graph fix was specifically for the TP8 deployment—the EP configuration had different graph capture logic. The benchmark validates the fix on the exact configuration that was crashing, before any optimization or configuration tuning.

The choice of budget=32 is also telling. Earlier in the session, the assistant had experimented with smaller budgets (8, 12, 16) on the PCIe machine. Budget=32 is the most aggressive setting, processing 33 tokens per verify step. This choice maximizes the stress on the CUDA graph fix—if there were any remaining edge cases in the tensor sizing logic, budget=32 would be most likely to expose them.

The absence of acceptance length measurements is a notable omission. The assistant stated the goal as "benchmark TP8+DDTree+cuda graphs and capture steady-state acceptance," but the script only measures throughput. This suggests the assistant was primarily concerned with confirming that the fix works (no crashes under load) and secondarily with gathering throughput data. Acceptance length—the number of draft tokens accepted per step—would require instrumenting the server logs, which is a separate measurement modality.

Connecting to the Broader Arc

This benchmark does not exist in isolation. Within the same chunk, the assistant will deploy the same DDTree stack on an 8× B300 SXM6 NVLink machine and achieve dramatically different results: 303 tok/s at C=1 (2.15× over autoregressive) and 4723 tok/s at C=128. The contrast between the PCIe results (~50 tok/s) and the NVLink results (~303 tok/s) is a 6× improvement, underscoring the critical role of interconnect bandwidth in speculative decoding performance.

The PCIe benchmark in message 11618 thus serves multiple purposes. It validates the CUDA graph fix under the most constrained hardware configuration. It provides a baseline for cross-platform comparison. And it generates evidence that supports the assistant's eventual pivot: for PCIe deployments, EP parallelism with DFlash remains the superior choice, while DDTree's benefits are fully realized only on NVLink-connected GPUs.

Conclusion

Message 11618 is a benchmark, but it is also a proof point. It proves that the CUDA graph sizing fix is correct—the service handles 33-token verify batches without crashing, across multiple prompts and concurrency levels. It proves that the assistant can methodically diagnose a subtle runtime error, trace it through thousands of lines of source code, implement a surgical fix, and validate it with systematic measurement.

The numbers themselves—50 tok/s at C=1, 88 tok/s at C=64—are not impressive in absolute terms. But they are honest measurements of a complex system operating under real constraints. They reveal the limits of TP8 on PCIe for tree-structured speculation, and they set the stage for the NVLink results that will follow. In the narrative of this coding session, message 11618 is the moment when the assistant confirms that the engine is running smoothly, even if the road ahead demands a different vehicle.