The Documentation Threshold: Capturing Definitive Benchmark Results in eagle-fast-verify.md

Introduction

In the course of a long and technically demanding optimization session spanning dozens of messages, one seemingly minor action stands out as a quiet watershed moment. At message index 5585, the assistant executes a single read command:

[assistant] Now let me update the documentation with all results: [read] /home/theuser/glm-kimi-sm120-rtx6000bw/eagle-fast-verify.md

This is not a dramatic intervention — no bash command, no code edit, no server restart. It is a simple file read, a prelude to writing. Yet this message marks the precise moment when a complex investigation transitions from active discovery into permanent record. The assistant has just completed a comprehensive parallel throughput benchmark comparing EAGLE-3 speculative decoding against a baseline server, and the results are definitive enough — and surprising enough — that they demand documentation. This article examines why this message was written, what knowledge it presupposes, what it creates, and the reasoning that led to this point.

The Long Road to This Moment

To understand message 5585, one must appreciate the journey that preceded it. The assistant had been working for many hours on optimizing speculative decoding for the Kimi K2.5 INT4 model running on a machine with 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5. The hardware configuration was challenging: while Blackwell GPUs are powerful, their PCIe interconnect introduces communication bottlenecks that are especially punishing for speculative decoding, which requires frequent all-reduce operations to synchronize draft and target model states across GPUs.

Earlier in the session (segments 32–36), the assistant had systematically worked through a series of optimizations: fixing the EAGLE-3 hidden state wiring, profiling and tuning NCCL settings, sweeping step counts, enabling FlashInfer allreduce fusion for the SM120 architecture, upgrading the CUDA stack to version 13, and patching SGLang for Blackwell support. These efforts transformed EAGLE-3 speculative decoding from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s — a respectable single-stream result.

But single-stream latency is only one dimension of performance. The critical question for any production deployment is: what happens under load? When multiple requests arrive concurrently, does speculative decoding help or hurt total throughput? This question drove the work in segment 37, where the assistant designed and executed a parallel throughput benchmark comparing EAGLE-3 against a baseline server with no speculation, using identical prompts and the same software stack.

The Benchmark Results That Changed Everything

The assistant ran both servers through a battery of concurrency levels: C=1, 2, 5, 10, 30, 70, 100, and 250, each with 30 requests at 512 max tokens. The prompts were updated to coding/agentic tasks to better match the EAGLE-3 drafter's training distribution. The results, visible in the preceding messages ([msg 5577] and [msg 5583]), were stark:

EAGLE-3:

• C=1: 80.9 tok/s (per-request: 81.3 tok/s)
• C=2: 128.3 tok/s (per-request: 65.0 tok/s)
• Saturated at approximately 354 tok/s Baseline (no speculation):
• C=1: 92.7 tok/s (per-request: 92.7 tok/s)
• C=2: 157.7 tok/s (per-request: 78.8 tok/s)
• Saturated at approximately 773 tok/s The finding was unambiguous: the baseline strictly outperformed EAGLE-3 at every concurrency level. At C=1, baseline was already 15% faster (92.7 vs 80.9 tok/s). As concurrency increased, the gap widened dramatically — at saturation, baseline delivered more than double the total throughput (~773 vs ~354 tok/s). EAGLE-3's value proposition collapsed under load: the speculative decoding overhead (draft model forward passes, tree verification, additional all-reduce communication) consumed resources that could otherwise be used for batching more requests through the simpler decode path. This was a sobering result. The assistant had invested enormous effort in making EAGLE-3 work on this hardware, and while single-stream latency was marginally better (the per-request latency at C=1 was 6.3s for EAGLE-3 vs 5.5s for baseline — actually worse for EAGLE-3), the throughput story was clear. Speculative decoding on PCIe-connected Blackwell GPUs was a net negative for any workload with concurrency greater than 1.## The Failed Dynamic Disable Attempt Before arriving at the documentation step, the assistant had attempted a more ambitious intervention: implementing a dynamic speculation disable mechanism that would automatically switch between EAGLE-3 and baseline modes based on server load. The idea was elegant — at low concurrency, use speculative decoding for per-request latency; at high concurrency, fall back to the simpler decode path to maximize throughput. The implementation, however, proved intractable. The assistant attempted to patch the standard EAGLEWorker (v1) path (see [msg 5560]–[msg 5568]), but ran into fundamental issues with deeply coupled batch state management. The out_cache_loc tensor, which tracks where KV cache slots are allocated, was pre-sized for the speculative decode's draft token dimensions (e.g., 160 slots for a batch of 10 requests with 16 draft tokens each). When the fallback code tried to run a normal decode forward pass, the CUDA graph runner would encounter a shape mismatch: tensor size 160 vs expected size 2. The entire decode pipeline — from input_ids to seq_lens to kv_committed_len — was wired for speculative mode, and unwiring it mid-flight required replicating the entire prepare_for_decode logic that the scheduler normally handles for the non-speculative path. The assistant's investigation (see [msg 5569]–[msg 5573]) revealed the depth of the coupling. In the EAGLE path, prepare_for_decode returns early without allocating out_cache_loc — that allocation happens later inside _draft_preprocess_decode in the EAGLE worker. The normal decode path, by contrast, allocates 1 token per request, increments decode_batch_idx, kv_committed_len, and kv_allocated_len, and sets input_ids from previous output tokens. These are not simple fields to patch; they are deeply embedded in the scheduler's state machine. The assistant concluded: "The dynamic switching is a nice-to-have but the fundamental insight is already clear."

Why Document? The Reasoning Behind Message 5585

Given the failed dynamic disable attempt and the definitive benchmark results, the assistant faced a choice. It could continue chasing the dynamic switching feature, or it could pivot to the spec_v2 overlap path (which offers cleaner state separation but requires topk=1, reducing the draft tree from 16 to 3 tokens). Instead, the assistant chose a third option: document the results.

This decision reflects several layers of reasoning:

1. Closure on a hypothesis. The entire optimization effort had been driven by the assumption that EAGLE-3 could provide throughput benefits on this hardware. The benchmarks disproved this assumption for all but the most trivial concurrency levels. Documenting the results provided closure and prevented future wasted effort.
2. Institutional memory. The eagle-fast-verify.md file served as the project's running optimization log. Without updating it, the benchmark results would be scattered across terminal outputs and log files, invisible to anyone who hadn't followed the entire conversation. The assistant recognized that the most valuable output of the benchmarking effort was not a code change but a clear, reproducible record of what worked and what didn't.
3. Shifting priorities. The todo list visible in [msg 5584] shows the assistant systematically checking off completed items. The documentation step was the natural culmination of the "Compare EAGLE-3 vs baseline results" todo item. Once the comparison was complete, the next logical action was to record it.
4. Enabling informed decisions. The benchmark results would inform future strategy — whether to invest in the spec_v2 overlap path, whether to train better draft models, or whether to abandon speculative decoding on this hardware entirely. A documented record was essential for this decision-making.

Assumptions and Input Knowledge

Message 5585 assumes substantial domain knowledge. The reader must understand what speculative decoding is, why EAGLE-3 differs from the baseline, what the --speculative-num-steps 2 and --speculative-eagle-topk 4 flags mean, and why PCIe interconnect matters for multi-GPU communication. It also assumes familiarity with the project's file structure — specifically that /home/theuser/glm-kimi-sm120-rtx6000bw/eagle-fast-verify.md is the canonical optimization document.

The message also makes an implicit assumption about the value of documentation. In a fast-moving optimization session, there is always pressure to move to the next experiment, the next code change, the next server launch. The assistant judged that stopping to update documentation was worth the time — a judgment that reflects an understanding that unrecorded results are ephemeral.

Output Knowledge Created

This message creates knowledge by initiating the documentation of the benchmark results. The file content shown (lines 436–442) reveals the document's structure: it already contained the single-stream result (96.1 tok/s) and was about to receive the parallel throughput comparison. The completed documentation would serve as a reference point for future decisions, a record of methodology (identical prompts, same software stack, controlled concurrency levels), and a clear statement of the finding that baseline outperforms EAGLE-3 at all concurrency levels.

The Thinking Process

The assistant's thinking is visible in the todo list from [msg 5584]: "Excellent! Now I have both datasets with the same coding/agentic prompts. Let me compile the final comparison." The word "Excellent!" conveys satisfaction — the assistant had successfully executed a controlled experiment with matched conditions. The todo list shows a systematic mind: kill servers, start baseline, run benchmark, compare results. Each step was planned and executed in order.

The decision to read the documentation file rather than immediately write to it is also telling. The assistant is checking the current state of the document before making changes — a disciplined approach that avoids duplication or contradiction. The read reveals that the document already contains a "Parallel Throughput" section stub (lines 438–442), suggesting the assistant had previously created the section header and was now ready to fill in the data.

Conclusion

Message 5585 is a documentation checkpoint — a moment of reflection after a grueling optimization journey. It captures the transition from active investigation to knowledge preservation. The benchmark results it records are definitive: EAGLE-3 speculative decoding, despite heroic optimization efforts, cannot match the baseline on PCIe-connected Blackwell GPUs under any realistic concurrency. This finding, once documented, becomes a permanent part of the project's knowledge base, informing future decisions and preventing regression. In a session filled with complex technical work — CUDA upgrades, NCCL tuning, kernel debugging — this simple file read may be the most enduring contribution.