The Game Plan: A Turning Point in the EAGLE-3 Speculation Saga

Subject Message: [user] Write down eagle-k2finetune-game-plan.md (Message 4932 in the conversation, a single-line instruction from the user to the AI assistant)

Introduction

In the sprawling, multi-day effort to deploy speculative decoding for the Kimi-K2.5 language model on an 8-GPU machine, there comes a moment of crystallization. After dozens of tool calls, performance benchmarks, architectural comparisons, and dead ends, the user issues a deceptively simple instruction: "Write down eagle-k2finetune-game-plan.md." This single message, at index 4932 in the conversation, is not a casual note-taking request. It is the pivot point where raw investigation transforms into documented strategy — where the assistant's hard-won understanding of why EAGLE-3 speculation is failing gets formalized into a concrete, actionable plan for the future.

To understand the weight of this message, one must appreciate the journey that led to it. The assistant had just spent an extended session diagnosing a painful reality: the EAGLE-3 speculative decoding setup that once appeared to deliver 94 tok/s was actually not reproducible. The stable baseline was 82-83 tok/s, and EAGLE-3 with 2-step speculation was delivering only 59-61 tok/s — a 27% regression. The root cause had been identified: the verify step in EAGLE-3 runs in extend mode without CUDA graphs, costing approximately 30 milliseconds per cycle regardless of attention mode, compared to roughly 12 milliseconds for a single-token decode with CUDA graphs. On an 8-PCIe-GPU system running a 1-trillion-parameter MoE model, this was simply the real cost of verification — not a bug, not a regression, but a fundamental throughput ceiling.

The Reasoning and Motivation Behind the Message

The user's request to "write down" the game plan is motivated by several converging pressures. First, the investigation had reached a natural conclusion: the 30ms verify cost was not going away, and the assistant had exhausted the plausible software fixes (NCCL tuning, attention mode toggling, engine.py patches, scheduler.py patches, sitecustomize.py persistence). Second, the assistant had just completed a detailed architectural comparison between the team's own K2.5 drafter and the AQ-MedAI K2 drafter downloaded from HuggingFace, confirming they were drop-in compatible — identical hidden_size (7168), intermediate_size (18432), attention heads, fc projection dimensions, and draft_vocab_size (32000). Third, the math had been done: break-even required accept_len 2.46 (the team had 2.0), and 150 tok/s would require 78% conditional accuracy.

The user recognized that this analysis needed to be preserved. The assistant had generated a wealth of interconnected knowledge — performance baselines, architectural comparisons, training data requirements, accuracy targets — that would be lost if not captured in a durable document. The game plan file would serve as a handoff point, allowing future work sessions (or future agents) to pick up where this investigation left off without repeating the same diagnostic cycle.

How Decisions Were Made

Within the subject message itself, no decisions are made — it is a directive, not a deliberation. But the message is the result of decisions made in the preceding context. The key decisions that led to this moment include:

The decision to accept the 30ms verify cost as fundamental. After attempting multiple fixes — NCCL tuning environment variables, --speculative-attention-mode decode, engine.py and scheduler.py patches — none reduced the verify time below ~29ms. The assistant concluded this was the real cost of running a 3-token extend through the 1T MoE model on 8 PCIe GPUs, not a regression from code changes.

The decision to pivot from debugging to planning. Rather than continuing to chase the 10ms cycle-time regression (old measurements showed 20ms, new measurements showed 30ms), the assistant recognized that the previous 20ms measurements were from a different system state and were not reproducible. The correct response was not more debugging but a strategic reassessment.

The decision to investigate the AQ-MedAI drafter. The user explicitly asked the assistant to check the AQ-MedAI K2 model's architecture to determine if it could be fine-tuned for K2.5. The assistant downloaded the model, inspected its config.json and safetensors, and confirmed full architectural compatibility — a critical finding that directly enabled the game plan.

The decision to formalize the plan as a markdown file. The user's instruction to "write down" the plan reflects a choice to create persistent documentation rather than relying on the conversation history, which would scroll away and be lost.

Assumptions Made by the User and Agent

Several assumptions underpin this message, some explicit and some implicit:

That the architectural compatibility finding is correct. The assistant verified that every weight tensor in the AQ-MedAI K2 drafter matches the corresponding tensor in the team's K2.5 drafter — same shapes, same dtypes, same layer structure. The assumption is that this dimensional compatibility translates to functional compatibility: that fine-tuning from K2 weights will converge faster than training from scratch.

That K2 and K2.5 share enough hidden-state representation similarity. This is the critical open question. Both models share the DeepSeek V3 / MLA architecture, but K2.5 is a fine-tuned version of K2 with potentially different hidden-state distributions at layers 2, 30, and 58 (the layers used for EAGLE-3 auxiliary hidden states). The game plan explicitly acknowledges this uncertainty and proposes a "Phase 0" probe to measure it directly.

That more training data will improve accept_len. The assistant's analysis assumes a monotonic relationship between training data volume and draft model accuracy, extrapolating from AQ-MedAI's results (1.4M samples → accept_len 3.2-3.5) to the team's 37K samples (accept_len 2.0). This is a reasonable assumption grounded in the scaling literature, but it has not been validated for this specific model and task.

That the current benchmark methodology is reliable. The baseline of 82-83 tok/s was established through repeated measurements and is described as "consistent and repeatable." The assumption is that this baseline will remain stable across reboots and configuration changes, which the earlier 89→82 tok/s discrepancy had called into question.

Mistakes or Incorrect Assumptions

The preceding context reveals several mistakes and incorrect assumptions that the assistant had to work through:

The mistaken belief that the 94 tok/s measurement was the true baseline. The assistant initially believed that EAGLE-3 speculation was delivering 94 tok/s, only to discover that this measurement was not reproducible. The stable baseline was actually 82-83 tok/s. This discrepancy was eventually attributed to different system states across container boot sessions, but it highlights the fragility of performance measurements on complex multi-GPU systems.

The incorrect assumption that decode attention mode would reduce verify time. The assistant tried --speculative-attention-mode decode based on the theory that decode-mode attention would be faster than prefill-mode attention for the verify step. The experiment showed no improvement — verify remained at ~29ms regardless of attention mode. This was because the verify step's cost is dominated by the full forward pass through the 1T MoE model, not by the attention mechanism specifically.

The mistaken attempt to fix a non-existent NCCL propagation bug. The assistant patched engine.py and scheduler.py to propagate NCCL tuning environment variables to spawned worker processes, and persisted the variables in /usr/lib/python3.12/sitecustomize.py. These changes were well-intentioned but ultimately irrelevant to the verify-time problem. The NCCL tuning may have helped marginally, but it could not eliminate the fundamental 30ms verify cost.

The flawed assumption that the 20ms cycle time from the "old run" was the correct baseline. The assistant spent considerable effort comparing old log files (showing 20ms total cycle time) against new measurements (showing 30ms total cycle time), attempting to identify what had regressed. The eventual conclusion — that the old measurements were from a different system state — meant this entire line of investigation was chasing a phantom regression.

Input Knowledge Required

To understand this message, one needs knowledge spanning several domains:

Speculative decoding fundamentals. Understanding that EAGLE-3 works by having a small "draft" model predict multiple tokens, which the large "target" model then verifies in parallel. The key metric is accept_len — the average number of draft tokens accepted per verification cycle. The relationship between accept_len, cycle time, and throughput is essential: throughput = accept_len / cycle_time.

CUDA graphs and their role in inference optimization. CUDA graphs capture a sequence of GPU kernel launches and eliminate CPU-side launch overhead. The baseline (non-speculative) inference uses CUDA graphs for single-token decode, achieving ~12ms per step. The verify step in EAGLE-3 cannot use CUDA graphs because it needs to capture intermediate hidden states, adding ~18ms of overhead per cycle.

The DeepSeek V3 / MLA architecture. Understanding that both K2 and K2.5 are based on DeepSeek V3 with Multi-head Latent Attention (MLA), with 61 layers, hidden_size 7168, and MoE feed-forward layers. The EAGLE-3 draft model uses hidden states from layers 2, 30, and 58 as auxiliary inputs.

The AQ-MedAI K2 drafter's training methodology. The assistant references that AQ-MedAI trained on 1.4M samples to achieve accept_len 3.2-3.5. Understanding the relationship between training data scale and draft model quality is crucial for evaluating the game plan's feasibility.

Performance benchmarking on multi-GPU PCIe systems. The assistant's analysis accounts for PCIe bandwidth limitations, NCCL communication overhead, and the difference between GPU-compute time and CPU-launch overhead — all of which are magnified on an 8-GPU system where inter-GPU communication traverses PCIe rather than NVLink.

Output Knowledge Created

The message triggers the creation of eagle-k2finetune-game-plan.md, which becomes a lasting artifact of the investigation. This document captures:

Three concrete approaches for improving EAGLE-3 speculation performance, ordered by cost and risk:

The Thinking Process Visible in Reasoning

The subject message itself contains no reasoning — it is a bare instruction. But the reasoning is visible in the context that produced it. The assistant's preceding messages reveal a systematic, hypothesis-driven investigation:

The assistant first verified the performance regression by running fresh benchmarks, establishing that the current baseline was 82-83 tok/s and EAGLE-3 was 59-61 tok/s. It then examined profiling logs to isolate the verify step as the bottleneck (29ms per cycle, consuming 97% of cycle time). It tested the decode attention mode hypothesis and found no improvement. It checked whether sgl-kernel or flashinfer versions had changed (they hadn't). It compared old log files against new ones, discovering the 20ms→30ms cycle time discrepancy. It considered and rejected the hypothesis that SGLang code changes caused the regression. It ultimately concluded that the 30ms verify cost was the real, irreducible cost of running verification through the 1T model on 8 PCIe GPUs.

The assistant then pivoted from debugging to strategic analysis. It computed the break-even accept_len (2.46) and the accuracy requirements for various throughput targets. It downloaded and inspected the AQ-MedAI K2 drafter, comparing every weight tensor against the team's K2.5 drafter. It confirmed full architectural compatibility and identified the missing embed_tokens.weight as a non-issue (it comes from the base model at runtime).

This progression — from measurement to hypothesis testing to root cause identification to strategic planning — is the hallmark of a mature engineering investigation. The user's instruction to "write down" the game plan is the acknowledgment that the investigation phase is complete and the execution phase must begin. The game plan document becomes the bridge between these two phases, preserving the hard-won knowledge of the investigation while charting a clear path forward.

Conclusion

Message 4932 — "Write down eagle-k2finetune-game-plan.md" — is a deceptively simple instruction that represents a major transition in the EAGLE-3 speculation project. It marks the moment when the team stopped debugging and started planning, when raw investigation crystallized into documented strategy. The game plan document that resulted from this message would go on to guide the next phase of the project, capturing performance baselines, architectural findings, accuracy targets, and three concrete approaches for improving speculative decoding throughput. In the broader narrative of the coding session, this message is the fulcrum on which the project turns from diagnosis to execution.