The $86 Pivot: How OpenRouter Rescued EAGLE-3 Training Data Generation

Introduction

In the sprawling, multi-week effort to train a custom EAGLE-3 speculative decoding drafter for the Kimi-K2.5 language model, few moments carry the weight of a strategic pivot executed under pressure. This chunk of the opencode session captures exactly such a moment: the transition from local GPU inference — slow, resource-constrained, and bottlenecked by hardware — to the OpenRouter API, a cloud-based inference service that promised speed at a predictable cost. What followed was a masterclass in rapid infrastructure development, rigorous validation, and pragmatic decision-making under real-world constraints.

The session had been building toward EAGLE-3 training for days. The assistant had successfully deployed the Kimi K2.5 model (quantized to NVFP4) using SGLang, tuned it to achieve 90 tok/s single-stream throughput, extracted 10,000 hidden states from the local server, and trained an initial EAGLE-3 draft model. But that draft model achieved zero acceptance rate — a complete failure traced to a subtle bug in how hidden states were concatenated across layers. After fixing the concatenation bug, the assistant scaled up the training data by 10×, launching an inference pipeline to generate responses for 83,000 prompts. But even with optimized SGLang throughput of ~930–1350 tok/s, the local GPUs were too slow for the volume needed. The assistant calculated that generating all remaining data locally would take days.

The pivot to OpenRouter was a strategic decision: trade the certainty of local token-level control for the speed of API-based generation. In just 33 minutes, at a cost of ~$86, the assistant completed all six B-datasets (B3 through B8) — a task that would have consumed days of local GPU time. But this speed came at a cost: OpenRouter returns text, not token IDs, and reconstructing the exact token sequences needed for EAGLE-3 training required solving a series of subtle and interconnected technical challenges.

The Token Reconstruction Problem: From Text to Token IDs

The core challenge of using OpenRouter for EAGLE-3 training data generation was deceptively simple: the API returns decoded text, but the training pipeline requires exact token IDs. Every special token, every BPE boundary, every merge decision must be faithfully reconstructed from the textual response. This is not a trivial engineering detail; it is the entire foundation of the training pipeline.

The assistant's investigation into token reconstruction consumed multiple messages and revealed several critical discoveries [1][6][7][8]. The most important was the behavior of the <|im_end|> token — the end-of-sequence marker that terminates each model response. Through careful testing, the assistant discovered that <|im_end|> decodes to the string 'chas' and encodes as token 163586 when passed as text, but the correct end-of-sequence token ID was 163533 — a critical mismatch that would silently corrupt every sample if not handled. This asymmetry between decode and encode behavior is a classic BPE tokenizer pitfall: the tokenizer's merge rules can produce different token sequences depending on context, and special tokens are particularly vulnerable to this effect.

The assistant also verified that the response special token (ID 163607) acts as a clean BPE boundary, preventing token merges across the reasoning/content split. This was confirmed by encoding reasoning and content separately, then injecting the response token ID between them — producing the exact same token sequence as encoding the full concatenated string. This validation was essential because the reconstruction logic concatenates reasoning + " response" + content + "<|im_end|>", and any BPE boundary leakage would produce incorrect token sequences.

These investigations were thorough and methodical. The assistant ran test scripts that compared reconstructed token IDs against original output_ids from local SGLang inference, finding only 0.5% mismatch on B1_glaive and 6.5% on B3_magicoder — and all mismatches were confirmed to be benign BPE tokenization differences where the same text was split into different token sequences. As the assistant concluded: "For EAGLE-3 training this is totally fine — the training data just needs correct text content, and these BPE differences are semantically identical."

The Tool Call Question: A Critical Edge Case

One question remained unanswered during the initial token reconstruction validation: what about tool calls? The B-datasets included prompts with embedded function definitions that could trigger tool-calling behavior from the model. If OpenRouter parsed these tool calls into structured API responses (separating them from the content text), the reconstruction logic would need to handle a completely different data format.

The assistant's investigation in [msg 4018] [1] revealed a crucial insight: since the pipeline was not sending the tools parameter in API requests, OpenRouter's providers would not parse tool calls into structured format. The model might still generate tool call special tokens (<|tool_calls_section_begin|>, <|tool_call_begin|>, etc.), but they would appear as raw text embedded in the content field rather than as structured tool_calls in the API response. This meant the reconstruction approach was simpler than feared: just encode the content text as-is, and it would contain the tool call special tokens as literal strings.

A diagnostic check confirmed the scope: B5_openthoughts had 100% tool-calling prompts, while other datasets had low single-digit percentages. The assistant could proceed with confidence, knowing that the simple reconstruction approach would work for the vast majority of samples.

Building the OpenRouter Inference Pipeline

With the token reconstruction approach validated, the assistant built a new script, run_inference_openrouter.py, capable of handling 2000 concurrent requests with provider routing, rate limiting, and robust resume support. The script excluded specific providers (Fireworks NVFP4 and BaseTen FP4) that used quantizations known to produce different token outputs, and included sophisticated error handling for API failures.

The launch was preceded by a meticulous sanity check ([msg 4037]) [20]: a single test request to verify API connectivity, response format, credit availability, and model selection. The test succeeded on every dimension — $100.00 remaining, correct model selection, proper response format — and the pipeline was greenlit for full-scale execution.

What followed was a remarkable demonstration of API-based data generation efficiency. The pipeline processed all six B-datasets (B3 through B8) in approximately 33 minutes, consuming roughly $86 in OpenRouter credits. The final tally showed 35,314 samples totaling 90.9 million tokens across the B datasets, plus the pre-tokenized A datasets (A1_deepswekimi with 2,800 samples at 44.9M tokens, and A2_kimik25 with 2,000 samples at 2.6M tokens), bringing the grand total to approximately 40,114 samples and 138.4 million tokens.

The Moment of Intervention: "No stop in NOW!"

Just as the pipeline was hitting its stride — B3 completed, B4 beginning — the user intervened with a terse command: "No stop in NOW!" ([msg 4051]) [34]. The assistant immediately killed the remote process. The user's concern, expressed in the following message, was pointed: "Weren't we burning tokens with somewhat wrong semantics for tools at least?" ([msg 4054]) [37].

This moment of intervention was a critical quality gate. The user recognized that even if the token reconstruction was mechanically correct — special tokens surviving, BPE boundaries respected — the semantics of the tool calls might be wrong. The B1_glaive dataset contained prompts with function definitions embedded in system messages. When these prompts were sent to OpenRouter without the tools parameter, the model saw the function definitions only as text. It might still generate tool call tokens, but would it generate the same tool calls, with the same frequency and accuracy, as when tools were formally declared?

The assistant responded with a systematic audit that became a model of ML pipeline validation. The audit script checked all 1637 OpenRouter responses in B3 for structural correctness: proper placement of response (token 163607), correct <|im_end|> termination, absence of spurious tokens, and decode roundtrip fidelity. The results were clean — 0 issues across all 1637 responses. Token counts matched OpenRouter's billing within 0.04% average difference. Every response followed the structure reasoning_tokens + response + content_tokens + <|im_end|>.

The user's concern about tool call semantics was ultimately resolved by the audit's findings: the B3-B8 datasets don't have tool-calling prompts in the way B1 does. The tool call special tokens that appear in responses are generated by the model as raw text, and they encode correctly through the tokenizer. The reconstruction is faithful.

The Transition: From Data Generation to Hidden State Extraction

With data generation complete, the pipeline pivoted to its next phase: merging, shuffling, and hidden state extraction. The user's directive in [msg 4082] set the agenda: "Continue next stages, merge (prompt/think/resp/tool), shuffle, tokenize and perform next steps; Assess how much space is needed for hidden state extraction - might need VM data disk to be expanded. Possibly remove old 10k hidden state extraction to free 1TB."

The assistant's response was a comprehensive analysis of the resource constraints facing the extraction phase. Hidden state extraction — the process of running each sample through the full 61-layer Kimi-K2.5 model to capture intermediate representations — would be the most compute- and storage-intensive phase of the entire pipeline. The naive approach would require 5.5 terabytes of disk space and an estimated 91 hours of compute time.

The assistant's analysis in [msg 4088] [71] explored multiple tradeoff scenarios:

1. Full dataset with 8192 cap: 110.7M tokens, 4,435 GB, ~91 hours — rejected as "too long."
2. Drop A1 entirely, cap at 8192: 87.8M tokens, 3,517 GB, ~72 hours — the chosen plan.
3. Drop A1, cap at 4096: Would lose too much B data — rejected.
4. Subsample A1 (500 of 2800): 91.9M tokens, 3,681 GB, ~75 hours — marginal improvement.
5. Keep A1 with 4096 cap: Still too much — rejected. The decision to drop A1_deepswekimi was driven by a stark asymmetry: 2,800 samples (just 7% of the total sample count) accounted for 44.9 million tokens (32% of the total token budget). The assistant reasoned that the EAGLE-3 drafter would "mostly serve shorter conversations," making the ultra-long agent trajectories less valuable relative to their disproportionate cost. The final plan was pragmatic: drop A1, keep A2+B1-B8, cap sequences at 8192 tokens, yielding 37,314 samples, ~87.8M tokens, ~3,517 GB of hidden states, and an estimated ~72 hours of extraction time. The assistant noted that 72 hours (3 days) was "acceptable" given the available disk space (11 TB on /data, with the old 924 GB of 10K hidden states ready for deletion).

The Broader Significance

This chunk of the session demonstrates several principles that are essential for large-scale ML engineering:

Measure before you build. The assistant consistently resisted the temptation to build complex solutions to hypothetical problems. Instead, it ran quick diagnostic checks — scanning prompts for tool mentions, testing token reconstruction on sample data, auditing response structure — to understand the actual scope of each problem before committing to a solution.

Validate at every transition. Every change in data source (local GPUs to OpenRouter), every new dataset, every reconstruction step was validated empirically. The 1637-response audit, the token count matching within 0.04%, the structural checks for special token placement — these validations provided the confidence needed to proceed at scale.

Know when to pivot. The decision to abandon local GPU inference for OpenRouter was not a failure of the local setup — it was a recognition that different tools are suited to different phases of the pipeline. Local GPUs excel at compute-intensive hidden state extraction; API services excel at high-throughput, diverse data generation. The assistant's ability to recognize this distinction and act on it was the key to the pipeline's success.

Resource constraints drive architecture. The 72-hour extraction bottleneck, the 924 GB of orphaned hidden states, the 11 TB disk limit — these hard constraints shaped every decision about dataset inclusion, sequence length capping, and processing order. The assistant's analysis of these constraints was not an afterthought but a central driver of the pipeline's design.

Conclusion

The $86, 33-minute OpenRouter pivot was not just a cost-saving measure — it was a fundamental rethinking of how EAGLE-3 training data should be generated. The assistant built a new infrastructure from scratch, solved a series of subtle tokenization problems, validated the output across thousands of samples, and navigated a critical user intervention that could have derailed the entire pipeline. The result was a dataset of 40,114 samples and 138.4 million tokens, ready for the compute-intensive hidden state extraction phase.

In the broader narrative of training an EAGLE-3 drafter, this chunk represents the moment when the pipeline shifted from struggling against hardware constraints to operating at cloud-native scale. The lessons learned — about token reconstruction, API validation, and resource-driven decision-making — are applicable far beyond this specific project. They are the building blocks of robust, scalable ML data pipelines.## References

[1] "The Tool Call Investigation: A Pivotal Moment in EAGLE-3 Training Data Generation" — [msg 4018] [6] "The One-Token Mismatch: Debugging BPE Token IDs in an EAGLE-3 Training Pipeline" — [msg 4023] [7] "The Case of the Two <|im_end|> Tokens: A Debugging Deep Dive into Token ID Reconstruction for EAGLE-3 Training Data" — [msg 4024] [8] "The Token That Wasn't: Debugging a Critical Token ID Mismatch in EAGLE-3 Training Data Reconstruction" — [msg 4025] [20] "The Sanity Check: Validating an OpenRouter Pipeline Before Scaling to 2000 Concurrent Requests" — [msg 4037] [34] "\"No stop in NOW!\" — The Moment of Intervention" — [msg 4051] [37] "The Moment of Reckoning: Questioning Tool Call Semantics in OpenRouter-Based Training Data Generation" — [msg 4054] [71] "The 72-Hour Bottleneck: Strategic Dataset Truncation for EAGLE-3 Training at Scale" — [msg 4088]