The Green Light: How a Single User Command Launched a Comprehensive Benchmarking Campaign for Speculative Decoding on Blackwell GPUs

Introduction

In the complex world of machine learning systems engineering, some of the most consequential moments arrive not as lengthy technical documents or elaborate architectural diagrams, but as brief, almost telegraphic commands. This article examines one such moment: a user message in an opencode coding session that simply reads, "@bench-plan.md execute benchmarks on CT200." At first glance, this appears to be a straightforward instruction—a user telling an AI assistant to run a pre-written benchmark plan. But beneath this surface simplicity lies a rich tapestry of technical context, deferred decision-making, implicit assumptions, and the culmination of hundreds of prior messages spanning days of intensive engineering work.

This message, indexed as <msg id=11252> in the conversation, represents a critical inflection point. It is the moment where planning transitions to execution, where theory meets practice, and where weeks of speculative decoding research—spanning custom SGLang patches, Mamba state leakage debugging, and multi-GPU infrastructure wrestling—crystallizes into a systematic evaluation campaign. To understand why this message matters, we must understand the journey that led to it, the technical landscape it operates within, and the cascade of decisions it set in motion.

The Message in Full

The subject message, as recorded in the conversation, is deceptively brief:

[user] @bench-plan.md execute benchmarks on CT200

This is followed by the full content of the referenced file, bench-plan.md, a 266-line document that constitutes the benchmark plan. The file itself is a meticulously structured document covering hardware configuration, model specifications, software stack details, eight different speculative decoding methods, three tensor parallelism configurations, five workload types, a detailed execution protocol, and a LaTeX report structure with pgfplots charts.

The message is a user command, but it is also an act of delegation. The user is not specifying how to run the benchmarks—they are trusting the assistant to interpret the plan, handle the infrastructure, manage the service lifecycle, collect results, and produce the promised LaTeX report. This trust is earned through the preceding conversation, where the assistant has demonstrated deep competence in SGLang internals, DDTree implementation, and system administration on the exotic Blackwell GPU hardware.

The Journey to This Point

To appreciate the weight of this message, we must trace the arc of the conversation that preceded it. The session began in a very different place: setting up an ML environment on Ubuntu 24.04 with NVIDIA drivers and CUDA Toolkit, resolving flash-attn build issues by reducing parallel compilation jobs from 128 to 20, and creating Python virtual environments. The hardware was exotic—NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, representing the cutting edge of NVIDIA's professional GPU lineup with 96 GB of HBM memory each.

The conversation then shifted to deploying the GLM-5-NVFP4 model using SGLang, a high-performance inference engine. But the real focus emerged when the user pivoted to speculative decoding research, specifically the DFlash (Draft-then-Verify) architecture and its tree-based variant, DDTree (Draft-then-Verify with Dynamic Tree). The assistant implemented native DDTree support directly in SGLang's source code, adding 13 new server command-line flags, creating tree-building utilities, implementing top-k logprob computation for tree construction, and handling the complex non-contiguous KV cache commit logic required for tree verification.

The initial results were promising. On a single GPU (TP1), DDTree with budget=15 achieved 124.2 tok/s average versus DFlash linear's 100.1 tok/s—a 24% throughput improvement. However, higher budgets (32, 64) showed degraded performance due to a fundamental issue: Mamba state leakage. The Qwen3.6-27B model is a hybrid architecture combining GDN (recurrent) layers with attention layers. When DDTree constructs a tree of draft tokens, sibling branches at the same depth corrupt each other's recurrent state during sequential verification, reducing acceptance rates and ultimately throughput.

The assistant had documented all of this in a comprehensive status update at <msg id=11251>, which served as the immediate precursor to the user's command. That update covered the implementation status, benchmark results, blocked items, key decisions, and next steps. It was, in effect, a project status report that positioned the team for the next phase: systematic benchmarking.

Why This Message Was Written

The user's motivation for writing this message can be understood at multiple levels. At the most superficial level, the user is responding to the assistant's previous work: the assistant had written bench-plan.md (at <msg id=11248-11249>) in response to the user's earlier instruction at <msg id=11243>: "Do more max tokens and simulate more agentic thing; Bench TP4 and TP8, sweep draft budgets and create a nice latex report with charts. Right now only write bench-plan.md."

The user had explicitly asked for the plan first, and now, after receiving it, they are authorizing its execution. This is a classic "review-and-approve" workflow, compressed into a single message because the user trusts the assistant's planning.

But there are deeper motivations. The user is clearly driving toward a specific goal: understanding the performance characteristics of DDTree speculative decoding on high-end hardware, across multiple dimensions (tensor parallelism, generation length, concurrency, agentic workloads). This is not idle curiosity—it is research-grade benchmarking intended to produce a LaTeX report with publication-quality charts. The user wants answers to specific questions:

  1. Does DDTree's advantage hold at TP4 and TP8? The initial TP1 results showed DDTree winning on 4 of 5 prompts, but tensor parallelism introduces communication overhead that could change the calculus. The AllReduce operation required for the top-k logprob computation across GPUs might eat into DDTree's gains.
  2. How does generation length affect the comparison? Short generations (256 tokens) favor speculative decoding because the prefill cost is amortized over fewer decode steps. Longer generations (1024, 2048 tokens) might show different dynamics as the KV cache grows and memory bandwidth becomes the bottleneck.
  3. Does DDTree help in agentic scenarios? Multi-turn conversations with growing context are increasingly important for AI coding agents. The user specifically wants to simulate "Build a CLI calculator" and "Debug a data pipeline" scenarios, each with five turns of growing context.
  4. What is the optimal DDTree budget? The initial results suggested budget=15 was the sweet spot, but the user wants a systematic sweep across budgets 8, 12, 15, 16, 32, and 64 to confirm this across different TP configurations and workloads.
  5. How does concurrency affect throughput? Real-world deployments serve multiple users simultaneously. The concurrency sweep (1, 2, 4, 8 concurrent requests) tests how well each method scales under load. The user's message is, in essence, saying: "I have reviewed your plan. It covers the dimensions I care about. Execute it."

Decision-Making in This Message

While the message itself contains no explicit decision-making—it is a command, not a deliberation—it represents several implicit decisions:

Decision 1: Trust the plan. The user could have requested modifications to bench-plan.md, asked clarifying questions, or demanded a different methodology. Instead, they accepted the plan as-is. This decision reflects confidence in the assistant's understanding of the problem space and the assistant's ability to execute.

Decision 2: Delegate execution. The user could have run the benchmarks themselves, writing scripts and managing services manually. Instead, they delegated the entire execution to the assistant. This is a decision about division of labor: the user provides strategic direction, the assistant handles tactical execution.

Decision 3: Prioritize breadth over depth. The benchmark plan covers 8 methods × 3 TP configs × multiple workloads, totaling an estimated 2-2.5 hours of execution time. The user could have asked for a narrower focus (e.g., only TP1, only short generations) to get faster results. Instead, they approved the full sweep, indicating a preference for comprehensive data over speed.

Decision 4: Accept the estimated timeline. The plan estimates ~2-2.5 hours total. The user's command implicitly accepts this timeline, signaling patience for thorough results.

Decision 5: Commit to the LaTeX report. By referencing the plan that includes a detailed LaTeX report structure, the user is implicitly committing to producing a formal, publication-quality document. This is a higher bar than just getting numbers—it requires chart generation, formatting, and narrative writing.

Assumptions Embedded in This Message

Every command carries assumptions, and this one is no exception. Some of these assumptions proved correct; others would be challenged by reality in the subsequent execution.

Assumption 1: The infrastructure is ready.

The user assumes that CT200 (the benchmark host) has all necessary components in place: the models loaded, the virtual environment configured, the service files ready, and the GPUs available. This assumption was largely correct—the assistant had spent significant effort building the environment—but it overlooked some edge cases. For instance, the autoregressive baseline (no speculative decoding) had never been tested with this SGLang build. The assistant would later need to verify that the model could load without speculative flags.

Assumption 2: The benchmark plan is complete and correct.

The user assumes that bench-plan.md covers all necessary configurations, handles edge cases, and has realistic time estimates. In practice, the plan had some optimistic assumptions. It assumed service start/stop overhead of ~20 minutes total, but each service restart involves waiting for GPU release, model loading, and health checks, which could take 60-90 seconds each. With ~16 service transitions across the full sweep, that's 16-24 minutes just for overhead—consistent with the estimate, but tight.

Assumption 3: The autoregressive baseline will work.

The plan includes an autoregressive baseline (no speculative decoding) for comparison. However, the SGLang build had been heavily customized for DFlash and DDTree. The user assumes that omitting the --speculative-* flags will produce a working autoregressive service. This was a reasonable assumption but needed verification—SGLang should handle the Qwen3.6-27B hybrid model natively, but the specific build might have dependencies or configuration requirements.

Assumption 4: The results will be meaningful.

The user assumes that the benchmark results will be clean enough to generate a useful LaTeX report. This assumes no service crashes, no OOM errors, no CUDA initialization failures, and no other infrastructure problems. In practice, the subsequent execution would encounter exactly such issues—a host reboot that required re-downloading the model to /dev/shm, LXC cgroup permission problems blocking CUDA initialization, and PCIe/NUMA tuning requirements.

Assumption 5: The assistant can handle all edge cases.

The user's brief command implicitly trusts the assistant to handle any problems that arise during the 2.5-hour execution. This is a significant assumption—the assistant would need to diagnose service failures, adjust configurations, recover from crashes, and make judgment calls about whether to retry or skip problematic configurations.

Assumption 6: The agentic workloads are well-defined.

The plan describes two agentic scenarios (CLI calculator and data pipeline debugging) with five turns each. The user assumes these scenarios are realistic and will produce meaningful measurements. In practice, the assistant would need to generate a 200-line Python ETL script for the pipeline scenario, which hadn't been prepared in advance.

Potential Mistakes and Incorrect Assumptions

While the user's command was ultimately successful—the benchmarks were executed, results were collected, and a LaTeX report was generated—several assumptions in the plan proved incomplete or incorrect.

The Mamba State Leakage Problem Was Underestimated

The benchmark plan includes DDTree budgets up to 64, but the initial TP1 results had already shown that budgets ≥ 32 suffer from Mamba state leakage. The plan treats this as a known limitation but still includes these configurations in the sweep. In retrospect, including b32 and b64 in the TP4 and TP8 benchmarks was likely wasteful—the results would predictably show poor throughput, and the execution time could have been better spent on more repetitions of the promising configurations (b8, b12, b15).

The Autoregressive Baseline Was Not Pre-verified

The plan assumes the autoregressive service will work, but this was never tested before the benchmark execution began. The assistant's subsequent reasoning at <msg id=11253> reveals uncertainty: "I need to check whether autoregressive mode will actually work with this SGLang setup—the environment was configured for DFlash, so I'm not sure if omitting the speculative flags will let it run properly." This uncertainty should have been resolved before committing to the full benchmark plan.

Infrastructure Recovery Was Not Accounted For

The plan assumes CT200 is in a stable state. In reality, the machine had been rebooted, requiring re-downloading the 52 GB model to /dev/shm (a tmpfs that is lost on restart). The LXC container had cgroup v2 issues blocking the nvidia-uvm device, preventing CUDA initialization. These infrastructure problems would consume significant time during execution, extending the benchmark timeline beyond the estimated 2.5 hours.

The TP8 GPU0 Conflict Was Underappreciated

The plan mentions stopping the GPU0 wrapper before TP8 runs, but this introduces complexity. The wrapper was a temporary DDTree service running on GPU0, and stopping it required systemd service management. More importantly, the plan didn't account for what happens if the wrapper can't be restarted after TP8 completes—a real risk given the infrastructure fragility.

The LaTeX Report Generation Was Underestimated

The plan includes a detailed LaTeX report structure with pgfplots charts, grouped bar charts, line plots, and tables. Generating such a report programmatically from JSON results is a non-trivial task. The plan allocated no explicit time for report generation, treating it as a post-processing step. In practice, creating a robust report generator script would require significant additional work.

PCIe and NUMA Effects Were Not Considered

The benchmark plan treats TP4 and TP8 as simple extensions of TP1, assuming linear scaling. In reality, the 8-GPU configuration spans two NUMA domains on the dual-socket host, and PCIe topology creates communication bottlenecks. The assistant would later discover that TP4 outperforms TP8 for single requests due to cross-NUMA overhead, a finding that the original plan didn't anticipate.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs substantial background knowledge spanning multiple domains:

Speculative Decoding

Understanding the message requires knowledge of how speculative decoding works: a small draft model proposes tokens, a large target model verifies them in parallel, and accepted tokens are committed. DFlash is a specific architecture where the draft model is a lightweight transformer that shares the target model's embedding and language model head. DDTree extends this by constructing a tree of draft tokens at each step, allowing the target model to verify multiple paths simultaneously.

SGLang Architecture

The message references SGLang, a specialized inference engine for large language models. Understanding the benchmark plan requires familiarity with SGLang's service model (systemd units), configuration flags (--speculative-dflash-block-size, --speculative-ddtree-budget, --attention-backend, --grammar-backend), and tensor parallelism (--tp-size).

Tensor Parallelism and Multi-GPU Communication

The benchmark compares TP1, TP4, and TP8 configurations. Understanding the tradeoffs requires knowledge of how tensor parallelism splits model layers across GPUs, how AllReduce operations synchronize gradients during inference, and how PCIe bandwidth and NUMA topology affect communication latency.

Hybrid Model Architectures

The Qwen3.6-27B model combines GDN (recurrent) layers with standard attention layers. This hybrid architecture creates unique challenges for speculative decoding because the recurrent state doesn't compose cleanly with tree-structured verification. The "Mamba state leakage" problem is central to understanding why DDTree budgets beyond 16 show degraded performance.

Blackwell GPU Architecture

The RTX PRO 6000 Blackwell GPUs represent NVIDIA's latest architecture. Understanding the benchmark requires knowledge of SM120 compute capability, the incompatibility with FlashInfer's JIT compiler (requiring the Triton attention backend), and the memory characteristics (96 GB HBM per GPU).

System Administration for ML Infrastructure

The benchmark execution requires managing systemd services, handling CUDA device visibility, configuring NCCL environment variables, and troubleshooting LXC container issues. The message assumes the assistant has root-level access and can manage these systems.

Output Knowledge Created by This Message

The user's command set in motion a chain of events that produced substantial new knowledge:

Empirical Performance Data

The benchmarks produced quantitative results across 8 methods × 3 TP configs × multiple workloads. Key findings included:

Infrastructure Insights

The execution revealed several infrastructure dependencies:

Methodology Validation

The benchmark plan itself was validated as a methodology for comparing speculative decoding methods. The structured approach—systematic sweep across budgets, TP configurations, and workload types—proved effective at identifying the optimal configuration and understanding performance tradeoffs.

LaTeX Report Template

The report generator script created during this phase serves as a reusable template for future benchmarking campaigns. The pgfplots-based approach with grouped bar charts, line plots, and error bars provides a professional presentation format.

Documentation of Mamba State Leakage

The benchmarks provided quantitative evidence of the Mamba state leakage problem at high DDTree budgets. This finding has implications for any speculative decoding system targeting hybrid architectures—it suggests that tree-aware recurrent state management is necessary for correct high-budget operation.

The Thinking Process: From Command to Execution

While the user's message itself is brief, the assistant's response reveals a rich thinking process that illuminates the complexity behind the simple command. The assistant's reasoning at <msg id=11253> shows a careful analysis of the task ahead:

Step 1: Task Decomposition. The assistant immediately breaks the benchmark plan into manageable phases: "Set up bench infrastructure," "TP1 benchmarks," "TP1 agentic workloads," "TP4 benchmarks," "TP8 benchmarks," "Generate LaTeX report." This decomposition is essential for managing the complexity of the 2.5-hour execution.

Step 2: Time Budgeting. The assistant calculates the time required for each phase: "For TP1 I need 8 service starts... That's about 16 total service starts at roughly 90 seconds each, which gives me about 24 minutes of overhead." This shows a realistic assessment of the operational costs.

Step 3: Throughput Estimation. The assistant estimates per-run times based on expected throughput: "At around 100 tokens per second, 256 tokens takes about 2.5 seconds per run... For 1024 tokens at 10 seconds per run, that's 2.5 minutes total." This demonstrates understanding of the relationship between model throughput, generation length, and wall-clock time.

Step 4: Risk Assessment. The assistant identifies potential failure points: "I need to verify whether the Qwen3.6-27B model will load properly without speculative decoding—SGLang should support it natively, but I'll make sure the script handles any failures gracefully."

Step 5: Strategy Selection. The assistant chooses an approach: "Rather than restarting the service for every token length, I'll start each method once and run all the token length variations (256, 1024, 2048) while it's running." This optimization reduces service lifecycle overhead.

Step 6: Tool Selection. The assistant debates whether to run the script locally with SSH calls or directly on CT200: "I'm debating whether to run the script locally and SSH into CT200 for HTTP calls and log collection, or execute it directly on CT200 to keep everything local."

Step 7: Progress Management. The assistant plans for incremental result saving: "One JSON file per configuration and workload combination—so if the script gets interrupted, I don't lose all the results."

Step 8: Execution Strategy. The assistant decides to break the benchmark into phases with progress updates: "I'll structure this as a Python script that can be invoked separately for each test phase, with command-line arguments to specify which one to run."

This thinking process reveals that the assistant is not blindly executing a script but actively planning, estimating, risk-assessing, and optimizing. The user's brief command unleashes this analytical engine.

The Broader Context: Why This Benchmark Matters

To fully appreciate this message, we must understand the broader research context. Speculative decoding is a critical technique for reducing LLM inference latency without sacrificing quality. By using a small draft model to propose tokens that a large target model verifies in parallel, speculative decoding can achieve 2-5× speedups on suitable workloads.

DDTree represents an evolution of this approach. Instead of proposing a single linear sequence of draft tokens, DDTree constructs a tree of possible continuations, allowing the target model to verify multiple paths simultaneously. This is particularly valuable for tasks with high uncertainty, where the draft model's top predictions have similar probabilities.

The Qwen3.6-27B model is an interesting test case because of its hybrid architecture. Most speculative decoding research has focused on pure transformer models. The hybrid GDN+attention architecture introduces the Mamba state leakage problem, which is a novel challenge that the research community is still working to understand.

The RTX PRO 6000 Blackwell GPUs represent the cutting edge of inference hardware. With 96 GB of HBM memory per GPU, they can accommodate large models without quantization. Understanding how speculative decoding performs on this hardware is valuable for anyone deploying LLMs in production.

The user's command, then, is not just about running benchmarks—it is about advancing the state of knowledge in speculative decoding, hybrid model inference, and Blackwell GPU performance characterization. The LaTeX report that results from this work could inform deployment decisions, guide further research, or be submitted to a conference or workshop.

Conclusion

The message "@bench-plan.md execute benchmarks on CT200" is a masterclass in concise, high-leverage communication. In six words, the user delegates a complex, multi-hour benchmarking campaign, implicitly trusts the assistant's planning and execution capabilities, and commits to producing a publication-quality report.

But the message's brevity belies its significance. It represents the culmination of days of intensive engineering work—implementing DDTree in SGLang, debugging Mamba state leakage, wrestling with CUDA compatibility issues, and building a custom inference environment on exotic hardware. It is the moment when preparation meets execution, when planning becomes action.

The message also reveals the power of the assistant-user partnership in this conversation. The user provides strategic direction and domain expertise; the assistant provides tactical execution, technical depth, and tireless attention to detail. Together, they accomplish what would be difficult for either alone: a systematic, multi-dimensional evaluation of a cutting-edge inference technique on state-of-the-art hardware.

In the end, the benchmarks were executed, the results were collected, and the LaTeX report was generated. The user's command was fulfilled. But the true value lies not in the execution but in the knowledge created: the optimal DDTree budget for hybrid models, the scaling characteristics of tensor-parallel speculative decoding, the infrastructure requirements for Blackwell GPU clusters, and the methodology for evaluating speculative decoding systems.

All of this flowed from six words: "@bench-plan.md execute benchmarks on CT200."