The $86, 33-Minute Data Pipeline: How OpenRouter Replaced Local GPUs for EAGLE-3 Training

In the world of large language model training, data generation is often the bottleneck that determines whether a project succeeds or stalls. On a Wednesday afternoon, an AI assistant typed a single message that marked the culmination of an intense, multi-hour effort to generate training data for an EAGLE-3 speculative decoding system. The message was deceptively simple — a table of numbers, a cost summary, and a few observations about completion patterns. But behind those numbers lay a story of rapid pivoting, careful validation, and a strategic decision that saved days of GPU time.

The message at [msg 4080] reads:

All done. 33 minutes total, $86 spent. Final tally:

>

| Dataset | Samples | Tokens | Source | |---------|--------:|-------:|--------| | B1_glaive | 9,998 | 17.0M | Local SGLang | | B2_opencodeinstruct | 2,932 | 11.4M | Local SGLang | | B3_magicoder | 3,383 | 10.5M | Mixed (local + OR) | | B4_mixturethoughts | 1,891 | 10.2M | OpenRouter | | B5_openthoughts | 2,112 | 11.0M | OpenRouter | | B6_ultrachat | 5,957 | 11.1M | OpenRouter | | B7_sharegpt | 5,476 | 10.9M | OpenRouter | | B8_sweagent | 3,565 | 8.8M | OpenRouter | | Total B | 35,314 | 90.9M | |

>

Plus the pre-tokenized A datasets: - A1_deepswekimi: 2,800 samples, ~44.9M tokens - A2_kimik25: 2,000 samples, ~2.6M tokens

>

Grand total: ~40,114 samples, ~138.4M tokens

This message was the final report on a critical phase of the EAGLE-3 training pipeline — the generation of synthetic prompt-response pairs that would be used to train a speculative decoding draft model. Understanding why this message was written, what assumptions underpinned it, and what it meant for the project requires diving into the context of the broader effort.

The EAGLE-3 Training Data Pipeline: Why This Message Matters

EAGLE-3 is a speculative decoding framework that accelerates autoregressive language model inference by training a lightweight "draft" model to predict the target model's hidden states. Training such a draft model requires massive amounts of high-quality data — specifically, prompt-response pairs where the response includes both the model's internal reasoning chain and its final answer, tokenized exactly as the target model would produce them.

The project in question was building an EAGLE-3 drafter for the Kimi-K2.5 model, a large language model with 7168-dimensional hidden states. The training data pipeline had been running for days, starting with local inference on a machine equipped with 8 RTX PRO 6000 Blackwell GPUs. The assistant had already generated the "A" datasets — A1_deepswekimi (2,800 ultra-long samples, averaging 16,000 tokens each) and A2_kimik25 (2,000 samples) — using local SGLang inference. But the "B" datasets, which comprised eight different prompt sources totaling tens of thousands of samples, were proving far more challenging to generate locally.

The message at [msg 4080] was the victory lap for a strategic pivot that had occurred just hours earlier: abandoning local GPU inference for the OpenRouter API, a decision that compressed what would have been days of computation into 33 minutes at a cost of $86.

The OpenRouter Pivot: A Calculated Tradeoff

The decision to move from local GPUs to OpenRouter was not made lightly. The assistant had spent considerable effort optimizing the local SGLang server, tuning NCCL settings to push single-stream throughput from 63 tok/s to 90 tok/s, and developing a custom hidden state extraction patch. But generating tens of thousands of diverse responses required not just throughput but also diversity — the local server, running a single quantized model (GLM-5-NVFP4), could only produce one type of output. The OpenRouter API, by contrast, offered access to multiple providers serving the full-precision Kimi-K2.5 model, potentially producing higher-quality and more diverse responses.

The pivot required building an entirely new inference script, run_inference_openrouter.py, capable of handling 2,000 concurrent requests with provider routing, rate limiting, and robust resume support. The assistant had to solve a critical technical challenge: reconstructing exact Kimi-K2.5 token IDs from OpenRouter's text responses. This was non-trivial because the tokenizer uses special tokens like <|im_end|> (token 163586, not 163533 as initially assumed) and response (token 163607) that must be precisely placed to produce valid training data. The assistant had verified through deep audits that the reconstruction was accurate — 1,637 OpenRouter responses passed structural validation with zero issues, and token counts matched OpenRouter's billing within 0.04%.

The Numbers Behind the Summary

The final table in [msg 4080] tells a story of rapid, efficient data generation. The eight B-datasets ranged from 1,891 samples (B4_mixturethoughts) to 9,998 samples (B1_glaive), with token counts clustering around 10-11 million each. The total of 35,314 samples and 90.9 million tokens was generated in just 33 minutes — a throughput of over 17,000 tokens per second.

But the numbers also reveal the assistant's careful reasoning about the data. The assistant noted that B8_sweagent came in under the 10 million token budget "since the prompts are long agent trajectories, so the completion budget didn't hit 10M before all 3572 samples completed." This observation showed an understanding of the data's structure: SWE-agent trajectories are prompt-heavy (long multi-turn conversations with tool outputs), but the model's individual responses are short — brief reasoning chains followed by tool calls or code patches. The assistant had initially observed that the first responses to complete were the short ones, and the average completion length climbed from ~288 tokens to 1,567 tokens as the longer-running requests finished. This pattern was correctly interpreted as a natural ordering effect, not a data quality issue.

The cost of $86 for 33 minutes of API usage represented a significant savings over local GPU inference, which would have consumed hours of GPU time on the 8-GPU machine. The assistant had calculated that the remaining budget of $86.40 was sufficient to complete all five remaining datasets (B4 through B8) at roughly $15 each, and the actual spend came in almost exactly on target.

Assumptions, Validations, and the Thinking Process

The message at [msg 4080] is notable for what it does not say — the extensive validation work that preceded it. In the messages leading up to this summary ([msg 4059] through [msg 4062]), the assistant had conducted deep audits of the OpenRouter response data, checking for structural correctness, token count accuracy, and special token placement. These validations were driven by a critical assumption: that the text-based reconstruction of token IDs from OpenRouter's API responses would produce semantically identical token sequences to what the local SGLang server would have produced.

This assumption was tested rigorously. The assistant checked that every response ended with <|im_end|>, that exactly one response token was present, that no thinking tokens leaked into the output, and that the decoded text round-tripped correctly through the tokenizer. The validation passed with zero issues across 1,637 OpenRouter responses — a strong signal that the reconstruction was correct.

The assistant also made an implicit assumption about the data quality: that the OpenRouter responses, coming from different providers serving the same model, would be suitable for training the EAGLE-3 drafter. This assumption was reasonable given that the goal was to train a draft model to predict hidden states, not to learn specific response patterns. As long as the token sequences were valid and representative of the target model's behavior, the source of the responses should not matter.

One subtle insight in the message is the observation about B8's completion pattern. The assistant noted that "the first responses to complete were the short/easy ones (tool calls, brief patches), then the longer code generation/reasoning ones took more time." This is a classic phenomenon in parallel API inference — the distribution of completion times is bimodal, with short responses completing quickly and long responses taking proportionally more time. The assistant's correct interpretation of this pattern prevented a potential false alarm about data quality.

What Comes Next: The Transition to Hidden State Extraction

The message at [msg 4080] was not an endpoint but a transition point. The assistant had already scoped the remaining phases of the pipeline: merging approximately 40,000 samples (138.4 million tokens) into a shuffled dataset, then performing hidden state extraction — the compute-intensive process of running each prompt through the target model and capturing the internal hidden states that the EAGLE-3 drafter would learn to predict.

The assistant had identified a critical bottleneck: the A1_deepswekimi dataset, with its 2,800 ultra-long samples averaging 16,000 tokens each, dominated the token budget at 44.9 million tokens — nearly a third of the total. Capping sequence length at 8,192 tokens and potentially dropping A1 entirely would reduce the extraction time from an estimated 91 hours to 72 hours, and the storage requirement from 5.5 TB to 3.5 TB. These calculations, while not included in the summary message itself, represented the assistant's forward-looking reasoning about the pipeline's next phase.

The assistant had also written a merge-and-shuffle script and identified that the old 924 GB of 10K hidden states from a previous training run were ready for deletion to free space. The pipeline was at a critical juncture: data generation was complete, but the compute-intensive hidden state extraction phase was about to begin.

Conclusion

The message at [msg 4080] is a masterclass in efficient data pipeline management. In 33 minutes and $86, the assistant generated 35,314 training samples totaling 90.9 million tokens — a task that would have consumed days of local GPU time. But the message's true value lies not in the numbers themselves but in the reasoning behind them: the careful validation of data quality, the understanding of completion time distributions, the forward planning for the extraction phase, and the strategic decision to pivot from local GPUs to API-based inference.

For anyone building training pipelines for large language models, this message demonstrates the power of being willing to change strategy when local infrastructure proves insufficient. The OpenRouter pivot was not a failure of the local setup — it was a recognition that different tools are suited to different phases of the pipeline. Local GPUs excel at the compute-intensive hidden state extraction that comes next; API services excel at the high-throughput, diverse data generation that was just completed. The assistant's ability to recognize this distinction and act on it was the key to the pipeline's success.