The Verdict Arrives: Benchmarking the AQ-MedAI K2 Drafter on Kimi-K2.5

In the high-stakes world of speculative decoding for large language models, few moments are as decisive as the first benchmark run. After hours of preparation—killing stale processes, inspecting safetensors files, comparing vocab mappings, fixing configuration parameters, launching servers, and waiting through CUDA graph capture—message [msg 4964] delivers the cold, hard numbers that determine whether an entire line of inquiry is viable or doomed. This single message, consisting of a single bash command and its output, is the moment of truth for Phase 0 of the K2→K2.5 fine-tuning game plan: the direct probe of the AQ-MedAI K2 EAGLE-3 drafter against the Kimi-K2.5 target model.

The Context: Why This Benchmark Exists

To understand the significance of this message, one must understand the problem it was designed to solve. The team had been training an EAGLE-3 speculative decoding drafter from scratch for Kimi-K2.5, a massive Mixture-of-Experts language model. The from-scratch drafter had achieved 74.7% validation accuracy and was producing ~60 tok/s, but this was still below the baseline throughput of ~82 tok/s without speculation. The fundamental bottleneck was the verify step—a ~30ms NCCL all-reduce operation that consumed 97% of the speculative decoding cycle time.

Enter the AQ-MedAI K2 drafter. This was a pre-trained EAGLE-3 drafter for the Kimi-K2 model (the predecessor to K2.5), trained on 1.4 million samples with an impressive accept_len of 3.2–3.5. Since both K2 and K2.5 share the same DeepSeek V3 / MLA architecture, hidden_size of 7168, and layer structure, the hypothesis was that the K2 drafter's weights would provide a strong initialization for fine-tuning on K2.5 data. But first, the team needed to establish a baseline: how well does the K2 drafter work with K2.5 without any fine-tuning at all?

Phase 0 of the game plan was precisely this "quick probe"—launch the K2.5 base model with the AQ-MedAI K2 drafter attached, run a standard throughput benchmark, and measure the raw performance. Message [msg 4964] is the execution of that probe.

What the Message Actually Does

The message is deceptively simple. It contains a single tool call: a bash command that SSHes into the remote server (10.1.230.174) and executes a Python benchmark script located at /tmp/benchmark_eagle3.py. The script targets the SGLang server running on http://localhost:8000, which had been launched in the previous round with the K2.5 base model and the AQ-MedAI K2 drafter configured for 2-step EAGLE-3 speculation.

The benchmark configuration is standard for this project: max_tokens=2048 (generating 2048 tokens per run), num_runs=5 (five benchmark runs for statistical significance), and warmup=2 (two warmup runs to stabilize GPU state before measurement). The results come back clean:

| Run | Tokens | Time (s) | Throughput (tok/s) | Prompt Length | |-----|--------|----------|---------------------|---------------| | 1 | 2048 | 40.50 | 50.6 | 32 | | 2 | 2048 | 39.46 | 51.9 | 31 | | 3 | 2048 | 41.69 | 49.1 | 26 | | 4 | 1963 | 35.95 | 54.6 | 28 | | 5 | 2048 | 38.51 | 53.2 | 34 |

The overall throughput averages to approximately 51.8 tok/s—a number that tells an immediate story.

The Reasoning: Interpreting the Numbers

The assistant does not explicitly interpret the results within message [msg 4964] itself—that happens in the very next message ([msg 4965]), where the assistant writes "~52 tok/s — significantly worse than both our drafter (60 tok/s) and baseline (82 tok/s)." But the reasoning is implicit in the very act of running this benchmark. The assistant knew exactly what numbers to compare against:

Baseline (no speculation): ~82 tok/s. This is the throughput of the raw K2.5 model generating tokens autoregressively, one at a time, without any drafter assistance.
Existing from-scratch EAGLE-3 drafter: ~60 tok/s. This was the team's own drafter, trained on 100K K2.5 samples, which had achieved 74.7% validation accuracy but still underperformed the baseline due to the verify-step bottleneck.
AQ-MedAI K2 drafter (this probe): ~52 tok/s. This is the number that just landed. The gap is stark. The K2 drafter is not only worse than the baseline (82 → 52 tok/s, a 37% regression), but it is also worse than the from-scratch K2.5 drafter (60 → 52 tok/s, a 13% regression). This immediately signals a fundamental problem: the K2 drafter's representations do not transfer well to K2.5 without adaptation.

The Hidden Story: What the Benchmark Doesn't Show

The benchmark output is clean and professional, but it hides a deeper story that the assistant and user already knew. Earlier in the session ([msg 4948]), the assistant had discovered that the AQ-MedAI K2 drafter's vocab mapping (d2t tensor, which maps draft token IDs to target token IDs) differed from the K2.5 mapping in 31,748 out of 32,000 positions. Only 252 token positions were shared between the two mappings.

This is a catastrophic mismatch. The EAGLE-3 architecture works by having the drafter's lm_head predict logits over a "draft vocabulary" of 32,000 tokens, and then a d2t mapping converts those draft token IDs to the target model's vocabulary (which for Kimi-K2.5 is 163,840 tokens). The K2 drafter's lm_head was trained to predict specific draft-token positions that correspond to specific target tokens through the K2 mapping. When used with K2.5's different mapping, the same draft-token positions map to completely different target tokens, rendering the drafter's predictions essentially random.

The benchmark results confirm this analysis. An accept_len of 0.58 (revealed in [msg 4965]) means that on average, the drafter's predictions are accepted less than one token per speculation cycle—barely better than random guessing. With 3 draft tokens generated per step and 2 steps, the drafter is producing 6 candidates and having almost all of them rejected by the target model's verification process.

Assumptions and Their Consequences

The probe was conducted under several assumptions, some of which proved incorrect:

Assumption 1: Architectural compatibility implies functional compatibility. The K2 and K2.5 models share the same DeepSeek V3 / MLA architecture, hidden size, and layer structure. The config files are nearly identical. This led to the reasonable expectation that the K2 drafter would at least produce non-trivial acceptance rates. The assumption was violated by the vocab mapping mismatch—a detail that is not part of the model architecture per se, but is critical for the EAGLE-3 mechanism.

Assumption 2: The d2t mapping in the safetensors would be used as-is. The assistant correctly verified that SGLang loads d2t directly from the draft model's safetensors, so the AQ-MedAI mapping would be used. This assumption was correct—the issue is not how the mapping is loaded, but that the mapping itself encodes K2-specific token frequency statistics that don't match K2.5.

Assumption 3: The benchmark script would produce reliable, comparable numbers. This assumption held. The benchmark script (benchmark_eagle3.py) had been used throughout the project and produced consistent results with low variance (49.1–54.6 tok/s across 5 runs). The warmup runs ensured stable GPU state.

Input Knowledge Required

To fully understand this message, one needs to know:

The EAGLE-3 speculative decoding architecture: How draft models work, the role of d2t/t2d vocab mappings, and the verify step.
The SGLang serving framework: How it launches models with speculative decoding, the --speculative-draft-model-path flag, and the CUDA graph capture process.
The project's benchmark methodology: The benchmark_eagle3.py script, its parameters (max_tokens=2048, num_runs=5, warmup=2), and the interpretation of tok/s numbers.
The performance baselines: ~82 tok/s for no speculation, ~60 tok/s for the from-scratch EAGLE-3 drafter.
The hardware context: 8× RTX PRO 6000 Blackwell GPUs with PCIe interconnect, where NCCL all-reduce is the dominant bottleneck.
The vocab mapping discovery: That 31,748/32,000 d2t entries differ between K2 and K2.5, which was established earlier in the session.

Output Knowledge Created

This message creates several critical pieces of knowledge:

Quantitative rejection of the direct-probe hypothesis: The K2 drafter achieves ~52 tok/s, conclusively demonstrating that it does not work out of the box with K2.5.
A baseline for comparison: Any future improvements (fine-tuning, vocab mapping adaptation, etc.) can be measured against this 52 tok/s number.
Confirmation of the vocab mapping analysis: The poor performance validates the earlier discovery that the d2t mappings differ significantly. The 0.58 accept_len is consistent with a drafter whose predictions are mostly wrong for the target vocabulary.
A decision point: The results force a strategic choice—either fine-tune the K2 drafter on K2.5 data (Phase 1 of the game plan) or abandon the K2 approach entirely and pursue other optimizations.

The Thinking Process Visible in the Message

While the message itself is just a tool call and its output, the thinking process is visible in the structure of what was done:

Systematic methodology. The assistant did not simply run the benchmark once and move on. It ran 5 benchmark runs with 2 warmup runs, producing a statistically meaningful sample. The prompt lengths vary slightly (26–34 tokens), which is normal for the benchmark script's random prompt selection, but the throughput is consistent within ~10% variation.

Clean execution. The assistant waited for the server to be fully ready (verified via the /v1/models health endpoint in [msg 4962]) before running the benchmark. This avoids the common pitfall of benchmarking a server that is still loading weights or capturing CUDA graphs.

Implicit comparison framework. The assistant did not need to state the comparison numbers explicitly in this message because the entire project had established them. The benchmark output is presented raw, trusting that the reader (the user and the assistant itself) knows what 52 tok/s means in context.

The truncated "Overall..." line. The conversation_data shows the output ending with "Overall ..." — this is likely a truncation artifact in the data capture, not an actual truncation in the terminal. The full benchmark script would have printed summary statistics (mean, median, std dev) after the individual runs. Even without the full summary, the individual runs tell the complete story.

Conclusion: A Message That Closes a Door

Message [msg 4964] is a classic example of the scientific method in ML engineering: form a hypothesis, design an experiment, collect data, and let the data speak. The hypothesis was that the AQ-MedAI K2 drafter might work reasonably well with K2.5 due to architectural compatibility. The experiment was a clean, repeatable throughput benchmark. The data spoke clearly: 52 tok/s, worse than both alternatives.

This message closes the door on the "just use it directly" approach. The next message ([msg 4965]) opens the server logs to reveal the accept_len of 0.58, confirming the diagnosis. From there, the session would pivot to either fine-tuning the K2 drafter (Phase 1) or, when that also fails, to system-level optimization of the verify step itself. But message [msg 4964] is the turning point—the moment when a promising shortcut is ruled out and the harder path becomes unavoidable.

In the broader narrative of the project, this message represents the transition from "can we reuse existing work?" to "we must build our own or fix the system." The benchmark results are disappointing but invaluable: they save weeks of misguided effort and redirect attention to the real bottlenecks. Sometimes the most important result is a negative one, and message [msg 4964] delivers that negative result with clinical precision.