The $86, 33-Minute Pivot: How OpenRouter Rescued EAGLE-3 Training Data Generation

Introduction

In the sprawling, multi-week effort to train a custom EAGLE-3 speculative decoding drafter for the Kimi K2.5 language model, few moments carry the weight of a strategic pivot executed under pressure. Segment 29 of the opencode session captures exactly such a moment: the transition from local GPU inference — slow, resource-constrained, and bottlenecked by hardware — to the OpenRouter API, a cloud-based inference service that promised speed at a predictable cost. What followed was a masterclass in rapid infrastructure development, rigorous validation, and pragmatic decision-making under real-world constraints.

The session had been building toward EAGLE-3 training for days. The assistant had successfully deployed the Kimi K2.5 model using SGLang, tuned it to achieve 90 tok/s single-stream throughput, extracted 10,000 hidden states from the local server, and trained an initial EAGLE-3 draft model. But that draft model achieved zero acceptance rate — a complete failure traced to a subtle bug in how hidden states were concatenated across layers. After fixing the concatenation bug, the assistant scaled up the training data by 10×, launching an inference pipeline to generate responses for 83,000 prompts. But even with optimized SGLang throughput of ~930–1350 tok/s, the local GPUs were too slow for the volume needed. The assistant calculated that generating all remaining data locally would take days.

The pivot to OpenRouter was a strategic decision: trade the certainty of local token-level control for the speed of API-based generation. In just 33 minutes, at a cost of ~$86, the assistant completed all six B-datasets (B3 through B8) — a task that would have consumed days of local GPU time. But this speed came at a cost: OpenRouter returns text, not token IDs, and reconstructing the exact token sequences needed for EAGLE-3 training required solving a series of subtle and interconnected technical challenges.

The Context: Why Local Inference Wasn't Enough

The project had been running local inference on a machine with 8 RTX PRO 6000 Blackwell GPUs, using SGLang to serve Kimi K2.5 and generate responses for training data. This approach worked for the first two datasets (B1 and B2), but the pipeline was consuming enormous time and compute resources. The assistant had been tuning performance, patching SGLang's hidden state extraction, and debugging acceptance rates — all while the clock ticked on generating the remaining six datasets (B3 through B8).

The local approach had fundamental limitations. Each dataset required approximately 10 million output tokens, and with the model running at roughly 90 tokens per second in single-stream mode, generating a single dataset could take over 30 hours. With six datasets remaining, the timeline stretched to over a week of continuous inference — assuming no crashes, no GPU memory issues, and no other interruptions.

The alternative was OpenRouter: a commercial API that hosts dozens of model providers, including multiple instances of Kimi K2.5. By paying per-token, the assistant could potentially generate all six datasets in parallel, completing in hours instead of days. The budget was $100 — a constraint that would require careful provider selection and cost optimization.

The Research Phase: Mapping the Provider Landscape

Messages 3992 through 4007 represent an intensive research sprint. The assistant queried OpenRouter's model catalog, discovering that Kimi K2.5 was available from eleven providers at varying price points. A critical finding emerged: the model ships natively as INT4 (quantization-aware training), and most providers run these exact weights. However, two providers needed exclusion: Fireworks (running NVFP4, a lower-precision format) and BaseTen (running FP4, also below INT4, and currently down with status -5).

The assistant built a detailed provider comparison table ([msg 3998]), calculating budget scenarios and discovering that 60 million output tokens at the cheapest rate of $2.20 per million tokens would cost $132 — exceeding the $100 budget. This forced a recalibration: either reduce the per-dataset token budget from 10M to 7M, or let the script run until funds were exhausted with resume support.

The budget analysis was further complicated by reasoning tokens. Kimi K2.5 is a "thinking" model that generates extensive internal reasoning before producing its final answer. These reasoning tokens are billed as output tokens. Analysis of the B2 dataset showed an average of 3,500 tokens per response, while B3 (Magicoder) averaged only 1,232 tokens. The variance in reasoning length across different prompt types made precise budget planning difficult.

The solution was to implement budget-aware execution: the script would track spending via OpenRouter's credits API and stop gracefully when funds ran low, with robust resume support so it could be restarted later. The assistant also discovered that OpenRouter's provider.sort: "price" field could automatically route requests to the cheapest available provider, and the provider.ignore field could exclude problematic providers like Fireworks and BaseTen ([msg 4007]).## The Critical Challenge: Reconstructing Token IDs from Text

The most technically challenging aspect of this pivot was the problem of reconstructing exact Kimi K2.5 token IDs from OpenRouter's text responses. This was not a trivial tokenization exercise.

The Kimi K2.5 model uses a special token structure for its reasoning process. When the model "thinks," it generates tokens wrapped in thinking and response markers. These are not just formatting conventions; they are actual token IDs in the model's vocabulary (163606 and 163607 respectively). The OpenRouter API returns the reasoning content in a separate reasoning field, but the exact token boundaries matter for training the EAGLE-3 drafter.

The user's instruction in message 4010 — "Make super sure you capture thinking and tool calls correctly and that output is the exact correct tokens" — set the stage for the investigation that followed. This was not a casual suggestion; it was a hard requirement for the pipeline to succeed.

The Tokenizer Investigation: Four Critical Discoveries

The assistant launched a systematic investigation spanning messages 4011 through 4016, running Python scripts on the remote server to probe the Kimi K2.5 tokenizer's behavior. The investigation yielded four critical findings:

Finding 1: The <|im_end|> Anomaly. The most surprising discovery was that the <|im_end|> token, which the assistant initially believed to be token ID 163533, actually decoded to the string 'chas' — not to <|im_end|>. When the literal string <|im_end|> was encoded, it produced token ID 163586 — a completely different token. This meant the <|im_end|> token could not be reconstructed from text; it had to be injected as a raw token ID.

The implications were profound. If you naively take the text <|im_end|> (as a string) and encode it, you get token ID 163586 — a completely different token. The tokenizer sees the literal characters <, |, i, m, _, e, n, d, |, > and encodes them through its normal BPE process. But the actual special token ID 163533 is what the model generates, and it decodes to 'chas'. This asymmetry between decode and encode behavior is a classic BPE tokenizer pitfall: the tokenizer's merge rules can produce different token sequences depending on context, and special tokens are particularly vulnerable to this effect.

Finding 2: </think> Acts as a Clean BPE Boundary. In contrast to <|im_end|>, the </think> token (ID 163607) behaved as expected. decode([163607]) returned the string '</think>', and encode('</think>') returned [163607]. This token could safely be included as text in a string to be encoded.

More importantly, the assistant verified that </think> acts as a clean BPE boundary. The test script checked five different reasoning/content boundary cases with varying starting characters (newlines, backticks, spaces, punctuation). In every case, encoding the reasoning text separately, injecting the </think> token ID (163607), and encoding the content text separately produced the exact same token sequence as encoding the full concatenated string ([msg 4015]). The reason is that </think> is a special token in the vocabulary — it is assigned its own ID and the BPE algorithm treats it as an atomic unit that cannot be split or merged with adjacent text.

Finding 3: BPE Boundary Merges Exist. The assistant confirmed that BPE tokenizers can merge adjacent characters into single tokens. The test case 'x' + 'y' illustrated this perfectly: encoding 'x' separately gave one token, encoding 'y' separately gave one token, but encoding the combined string 'xy' gave a single merged token. This meant that encoding reasoning text and content text separately, then concatenating the token sequences, could theoretically produce different tokens than encoding the full concatenated string — if BPE could merge tokens across the boundary.

However, the empirical tests confirmed that </think> acts as an impenetrable boundary, making separate encoding safe for the specific case of reasoning/content reconstruction.

Finding 4: Tool Call Special Tokens Roundtrip Correctly. The fourth finding was that tool call special tokens — <|tool_calls_section_begin|> (163595), <|tool_call_begin|> (163597), <|tool_call_argument_begin|> (163598), <|tool_call_end|> (163599), and <|tool_calls_section_end|> (163596) — all roundtripped correctly when encoded from their text form. This was good news because it meant tool call sequences could be safely included as text and would produce the correct token IDs.

The assistant also confirmed that real SGLang output roundtripped perfectly: encode(decode(output_ids)) == output_ids for actual model generations from the B1_glaive dataset. This validated the fundamental approach of reconstructing token IDs by encoding the full decoded text.

The Correct Formula

With these findings, the assistant established the correct token reconstruction formula:

encode(reasoning_text) + [163607] + encode(content_text) + [163533]

This formula:

  1. Encodes the reasoning text separately (from OpenRouter's reasoning field)
  2. Injects the </think> token ID (163607) as a raw integer
  3. Encodes the content text separately (from OpenRouter's content field)
  4. Injects the <|im_end|> token ID (163533) as a raw integer The critical insight is that </think> and <|im_end|> must be injected as raw token IDs, not included as text strings, because <|im_end|> in particular does not survive a decode-encode roundtrip through text.

The Tool Call Question: A Critical Edge Case

One question remained unanswered during the initial token reconstruction validation: what about tool calls? The B-datasets included prompts with embedded function definitions that could trigger tool-calling behavior from the model. If OpenRouter parsed these tool calls into structured API responses (separating them from the content text), the reconstruction logic would need to handle a completely different data format.

The assistant's investigation in message 4018 revealed a crucial insight: since the pipeline was not sending the tools parameter in API requests, OpenRouter's providers would not parse tool calls into structured format. The model might still generate tool call special tokens (<|tool_calls_section_begin|>, <|tool_call_begin|>, etc.), but they would appear as raw text embedded in the content field rather than as structured tool_calls in the API response. This meant the reconstruction approach was simpler than feared: just encode the content text as-is, and it would contain the tool call special tokens as literal strings.

A diagnostic check confirmed the scope: B5_openthoughts had 100% tool-calling prompts, while other datasets had low single-digit percentages. The assistant could proceed with confidence, knowing that the simple reconstruction approach would work for the vast majority of samples.## Building the OpenRouter Inference Pipeline

With the token reconstruction approach validated, the assistant built a new script, run_inference_openrouter.py, capable of handling 2000 concurrent requests with provider routing, rate limiting, and robust resume support. The script excluded specific providers (Fireworks NVFP4 and BaseTen FP4) that used quantizations known to produce different token outputs, and included sophisticated error handling for API failures.

The script architecture was designed around several key principles:

Provider routing. Rather than hardcoding a single provider, the script used OpenRouter's provider.ignore field to exclude Fireworks and BaseTen, and provider.sort: "price" to automatically route to the cheapest available provider. This ensured cost efficiency without manual intervention. The quantizations filter was set to ["int4"] to match the native model format.

Massive concurrency. 2,000 parallel requests was the key advantage over local inference. The local SGLang server could handle perhaps 8-16 concurrent requests before degrading, while OpenRouter's distributed infrastructure could absorb thousands of simultaneous calls. The assistant used asyncio with aiohttp to manage this concurrency efficiently.

Exponential backoff. On 429 (rate limit) and 5xx (server error) responses, the script would back off exponentially, preventing cascading failures while maximizing throughput. This was essential for maintaining high throughput without overwhelming any single provider.

Resume support. The script would append to a JSONL file and skip completed sample IDs on restart, allowing the generation to be interrupted and resumed without losing progress. This was critical given the $100 budget constraint — if the script ran out of funds mid-dataset, it could be restarted later with additional credits.

Spend tracking. Periodic checks against OpenRouter's credits API to stop before exceeding budget. The script would log cumulative spending and estimated remaining budget after each batch of requests.

The Sanity Check

Before launching the full pipeline, the assistant performed a meticulous sanity check in message 4037: a single test request to verify API connectivity, response format, credit availability, and model selection. The test succeeded on every dimension — $100.00 remaining, correct model selection (moonshotai/kimi-k2.5), proper response format with both reasoning and content fields, and accurate token counting. The pipeline was greenlit for full-scale execution.

The Results: 33 Minutes, $86, Zero Issues

The OpenRouter script completed all six B-datasets (B3 through B8) in approximately 33 minutes at a cost of ~$86. The final tally, reported in message 4080, showed:

| Dataset | Samples | Tokens | Source | |---------|--------:|-------:|--------| | B1_glaive | 9,998 | 17.0M | Local SGLang | | B2_opencodeinstruct | 2,932 | 11.4M | Local SGLang | | B3_magicoder | 3,383 | 10.5M | Mixed (local + OR) | | B4_mixturethoughts | 1,891 | 10.2M | OpenRouter | | B5_openthoughts | 2,112 | 11.0M | OpenRouter | | B6_ultrachat | 5,957 | 11.1M | OpenRouter | | B7_sharegpt | 5,476 | 10.9M | OpenRouter | | B8_sweagent | 3,565 | 8.8M | OpenRouter | | Total B | 35,314 | 90.9M | |

Plus the pre-tokenized A datasets:

The Moment of Intervention: "No stop in NOW!"

Just as the pipeline was hitting its stride — B3 completed, B4 beginning — the user intervened with a terse command in message 4051: "No stop in NOW!" The assistant immediately killed the remote process. The user's concern, expressed in message 4054, was pointed: "Weren't we burning tokens with somewhat wrong semantics for tools at least?"

This moment of intervention was a critical quality gate. The user recognized that even if the token reconstruction was mechanically correct — special tokens surviving, BPE boundaries respected — the semantics of the tool calls might be wrong. The B1_glaive dataset contained prompts with function definitions embedded in system messages. When these prompts were sent to OpenRouter without the tools parameter, the model saw the function definitions only as text. It might still generate tool call tokens, but would it generate the same tool calls, with the same frequency and accuracy, as when tools were formally declared?

The assistant responded with a systematic audit that became a model of ML pipeline validation. The audit script checked all 1,637 OpenRouter responses in B3 for structural correctness: proper placement of response (token 163607), correct <|im_end|> termination, absence of spurious tokens, and decode roundtrip fidelity. The results were clean — 0 issues across all 1,637 responses. Token counts matched OpenRouter's billing within 0.04% average difference. Every response followed the expected structure.

The user's concern about tool call semantics was ultimately resolved by the audit's findings: the B3-B8 datasets don't have tool-calling prompts in the way B1 does. The tool call special tokens that appear in responses are generated by the model as raw text, and they encode correctly through the tokenizer. The reconstruction is faithful.

Scoping the Remaining Phases

With the B-datasets complete, the assistant turned to scoping the remaining phases of the pipeline. The next steps were:

  1. Merge and shuffle: Combining all ~40K samples (138.4 million tokens) into a single shuffled dataset. A merge-and-shuffle script was written to handle this.
  2. Hidden state extraction: The compute-intensive phase where the base model's hidden states are extracted for each token in the training data. This requires running the model on local GPUs, since OpenRouter doesn't provide access to internal model states. The assistant's analysis in message 4088 explored multiple tradeoff scenarios for the extraction phase. Hidden state extraction — the process of running each sample through the full 61-layer Kimi K2.5 model to capture intermediate representations — would be the most compute- and storage-intensive phase of the entire pipeline. The naive approach would require 5.5 terabytes of disk space and an estimated 91 hours of compute time. The analysis considered several options:
  3. Full dataset with 8192 cap: 110.7M tokens, 4,435 GB, ~91 hours — rejected as "too long."
  4. Drop A1 entirely, cap at 8192: 87.8M tokens, 3,517 GB, ~72 hours — the chosen plan.
  5. Drop A1, cap at 4096: Would lose too much B data — rejected.
  6. Subsample A1 (500 of 2800): 91.9M tokens, 3,681 GB, ~75 hours — marginal improvement.
  7. Keep A1 with 4096 cap: Still too much — rejected. The decision to drop A1_deepswekimi was driven by a stark asymmetry: 2,800 samples (just 7% of the total sample count) accounted for 44.9 million tokens (32% of the total token budget). The assistant reasoned that the EAGLE-3 drafter would "mostly serve shorter conversations," making the ultra-long agent trajectories less valuable relative to their disproportionate cost. The final plan was pragmatic: drop A1, keep A2+B1-B8, cap sequences at 8192 tokens, yielding 37,314 samples, ~87.8M tokens, ~3,517 GB of hidden states, and an estimated ~72 hours of extraction time. The assistant noted that 72 hours (3 days) was "acceptable" given the available disk space (11 TB on /data, with the old 924 GB of 10K hidden states ready for deletion). A merge-and-shuffle script was written to handle the dataset consolidation, and the old 924 GB of 10K hidden states from earlier training attempts were identified for deletion to free up disk space. The pipeline was now at the transition point between data generation and the compute-intensive hidden state extraction phase.

The Broader Significance

This segment of the conversation exemplifies several principles that are essential for large-scale ML engineering:

Test your assumptions. The assistant's systematic investigation of tokenizer behavior — testing roundtrip fidelity, comparing encoding strategies, probing BPE boundaries — demonstrates the level of rigor required when building ML pipelines that depend on exact token-level alignment. The discovery that <|im_end|> decodes to 'chas' rather than to <|im_end|> is exactly the kind of subtle bug that can silently corrupt an entire training dataset.

Verify at system boundaries. When moving from local GPU inference (where token IDs are directly available) to a cloud API (which returns text), the entire pipeline's correctness hinges on the fidelity of the tokenization roundtrip. The assistant invested significant effort in verifying this boundary before committing to production.

Measure before you build. The assistant consistently resisted the temptation to build complex solutions to hypothetical problems. Instead, it ran quick diagnostic checks — scanning prompts for tool mentions, testing token reconstruction on sample data, auditing response structure — to understand the actual scope of each problem before committing to a solution.

Validate at every transition. Every change in data source (local GPUs to OpenRouter), every new dataset, every reconstruction step was validated empirically. The 1,637-response audit, the token count matching within 0.04%, the structural checks for special token placement — these validations provided the confidence needed to proceed at scale.

Know when to pivot. The decision to abandon local GPU inference for OpenRouter was not a failure of the local setup — it was a recognition that different tools are suited to different phases of the pipeline. Local GPUs excel at compute-intensive hidden state extraction; API services excel at high-throughput, diverse data generation. The assistant's ability to recognize this distinction and act on it was the key to the pipeline's success.

Resource constraints drive architecture. The 72-hour extraction bottleneck, the 924 GB of orphaned hidden states, the 11 TB disk limit — these hard constraints shaped every decision about dataset inclusion, sequence length capping, and processing order. The assistant's analysis of these constraints was not an afterthought but a central driver of the pipeline's design.

Human-AI collaboration. The user's intervention at message 4010 — "Make super sure you capture thinking and tool calls correctly" — was the critical prompt that triggered the tokenizer investigation. The assistant had the technical capability to investigate these issues, but the user provided the strategic intuition about where things might go wrong. Similarly, the "No stop in NOW!" intervention in message 4051 forced a quality gate that validated the entire approach before proceeding further.

Conclusion

The $86, 33-minute OpenRouter pivot was not just a cost-saving measure — it was a fundamental rethinking of how EAGLE-3 training data should be generated. The assistant built a new infrastructure from scratch, solved a series of subtle tokenization problems, validated the output across thousands of samples, and navigated a critical user intervention that could have derailed the entire pipeline. The result was a dataset of 40,114 samples and 138.4 million tokens, ready for the compute-intensive hidden state extraction phase.

In the broader narrative of training an EAGLE-3 drafter, this segment represents the moment when the pipeline shifted from struggling against hardware constraints to operating at cloud-native scale. The lessons learned — about token reconstruction, API validation, and resource-driven decision-making — are applicable far beyond this specific project. They are the building blocks of robust, scalable ML data pipelines.

The token reconstruction problem, in particular, reveals the hidden complexity of working with large language model tokenizers. The discovery that <|im_end|> decodes to 'chas', the confirmation that response acts as a clean BPE boundary, and the validation that tool call special tokens survive encode-decode roundtrips — each of these findings was essential to producing correct training data. As the pipeline moves into the merge-and-shuffle and hidden state extraction phases, the token reconstruction logic validated in this segment will underpin every subsequent step. The EAGLE-3 drafter trained on this data will be built on the foundation of careful, assumption-testing engineering that this segment exemplifies.## References

[1] "The $86 Pivot: Reconstructing Exact Token IDs at 2000 Concurrency for EAGLE-3 Training" — Chunk article covering the token reconstruction investigation and OpenRouter pipeline design.

[2] "The $86 Pivot: How OpenRouter Rescued EAGLE-3 Training Data Generation" — Chunk article covering the pipeline execution, validation, and transition to hidden state extraction.

[3] "The Pivot to OpenRouter: A Strategic Decision Point in EAGLE-3 Training Data Generation" — [msg 3992] — Initial research into OpenRouter providers.

[4] "Mapping the Provider Landscape: The OpenRouter Provider Analysis That Pivoted an EAGLE-3 Training Pipeline" — [msg 3998] — Provider comparison table and budget analysis.

[5] "The Pivot Point: From Research to Implementation in the OpenRouter Inference Pipeline" — [msg 4007] — Final research step before script implementation.

[6] "The Pivotal Instruction That Saved the EAGLE-3 Pipeline" — [msg 4010] — User's critical instruction to verify token correctness.

[7] "The Token Reconstruction Problem: How One Assistant Unraveled the Mysteries of BPE Tokenization for EAGLE-3 Training Data" — [msg 4015] — BPE boundary validation and tokenizer investigation.

[8] "The Tool Call Investigation: A Pivotal Moment in EAGLE-3 Training Data Generation" — [msg 4018] — Tool call handling analysis.

[9] "The Sanity Check: Validating an OpenRouter Pipeline Before Scaling to 2000 Concurrent Requests" — [msg 4037] — Single-request test validation.

[10] "\"No stop in NOW!\" — The Moment of Intervention" — [msg 4051] — User intervention halting the pipeline.

[11] "The Moment of Reckoning: Questioning Tool Call Semantics in OpenRouter-Based Training Data Generation" — [msg 4054] — User's concern about tool call semantics.

[12] "The 72-Hour Bottleneck: Strategic Dataset Truncation for EAGLE-3 Training at Scale" — [msg 4088] — Extraction phase resource analysis and planning.