The Launch: Scaling EAGLE-3 Training Data Generation via OpenRouter

In the sprawling effort to train an EAGLE-3 draft model for the Kimi-K2.5 architecture, message [msg 4039] represents a decisive inflection point — the moment the pipeline pivoted decisively from local GPU inference to a cloud API, scaling from a handful of concurrent requests to two thousand. The message is deceptively simple: a single nohup command launching a Python script on a remote server, followed by a process listing confirming it started. But behind this brief exchange lies a carefully constructed bridge between two fundamentally different approaches to data generation, one that required solving subtle tokenization puzzles, validating structural integrity across thousands of samples, and making economic tradeoffs between local compute and paid API calls.

The Context: Why Local Inference Failed

The journey to this message began with a straightforward plan: use the eight RTX PRO 6000 Blackwell GPUs on the local machine to run the Kimi-K2.5 model via SGLang, generating training data for an EAGLE-3 draft model. The local SGLang server had been tuned extensively — single-stream throughput reached 90 tok/s, and the team had developed custom patches for hidden state extraction. However, the data requirements for EAGLE-3 training were enormous: tens of thousands of prompt-response pairs, each requiring autoregressive generation with specific sampling parameters.

The local approach hit fundamental bottlenecks. Even with eight GPUs, generating the full training dataset would take days or weeks. The server's throughput, while respectable for a single stream, couldn't scale to the hundreds of concurrent requests needed to generate millions of tokens in a reasonable timeframe. Moreover, the local SGLang server was needed for the subsequent hidden state extraction phase — running both inference and extraction simultaneously would create resource contention.

The pivot to OpenRouter was a strategic decision: trade dollar cost for time. OpenRouter provides API access to dozens of model providers, including several running Kimi-K2.5 in INT4 quantization. By paying per-token, the team could generate training data at massively higher concurrency than local hardware could support, completing in hours what would otherwise take days.

Building the Bridge: The OpenRouter Inference Script

Before this message could be sent, the assistant had to build a complete new inference pipeline. The script run_inference_openrouter.py was written to handle 2000 concurrent requests, with provider routing (excluding Fireworks NVFP4 and BaseTen FP4 providers), robust resume support, and — most critically — a mechanism to reconstruct exact Kimi-K2.5 token IDs from OpenRouter's text responses.

This reconstruction was the central technical challenge. OpenRouter returns text, not token IDs. The EAGLE-3 training pipeline, however, requires exact token sequences for hidden state extraction. The assistant had to reverse-engineer how OpenRouter's response format maps to the model's internal tokenization. This involved:

  1. Discovering the correct IM_END_TOKEN_ID: The <|im_end|> special token maps to token ID 163586, not 163533 as initially assumed. This was discovered through careful comparison of original and reconstructed token sequences.
  2. Analyzing BPE boundary behavior: The response separator between reasoning and content tokens creates specific BPE boundary conditions. The assistant verified that the tokenizer handles these correctly when encoding the reconstructed full text.
  3. Confirming tool call token survival: When the tools parameter isn't sent to OpenRouter, tool calls appear as raw text in the content field. The assistant confirmed these survive the reconstruction process correctly.
  4. Validating structural equivalence: Across 1,637 OpenRouter responses, structural validation showed zero issues, with token counts matching billing within 0.04%. The validation results were encouraging: only 0.5% mismatch on B1 and 6.5% on B3, all attributable to BPE tokenization differences where the model's autoregressive token selection produced different splits than the tokenizer's greedy encoding. Critically, only 1 semantic difference was found out of 500 samples checked — and even that was likely a unicode edge case. For EAGLE-3 training, these BPE differences are harmless because the hidden state extraction uses whatever tokenization the training data specifies, creating a self-consistent learning signal.

The Command: Anatomy of a Launch

The message shows a nohup command launching the OpenRouter inference script with specific parameters:

--partition all
--output-dir /data/eagle3/synth_100k/prepared
--api-key-file /tmp/or-key.txt
--short-concurrency 2000
--short-max-tokens 10240
--long-concurrency 500
--long-max-tokens 16384
--token-budget 10000000

The choice of --partition all is deliberate. The assistant explains: "B1 and B2 are already done. We skip them by using --dataset for each, or better, run --partition all and let the token budget skip B1/B2. Since B1/B2 already have >10M tokens, they'll be skipped immediately." This is an elegant solution — rather than maintaining a separate exclusion list, the script's resume logic naturally skips datasets that have already met their token budget.

The concurrency settings reveal the assistant's understanding of OpenRouter's capabilities and the dataset characteristics. "Short" samples (up to 10,240 tokens) get 2,000 concurrent requests — an aggressive setting that assumes OpenRouter's provider network can handle this load without rate limiting. "Long" samples (up to 16,384 tokens) get only 500 concurrent requests, reflecting the longer generation time and higher cost per request for extended responses. The 10M token budget represents the total target for this phase, balancing data quantity against the ~$86 cost that would ultimately be incurred.

The nohup and background execution indicate this is a long-running process that should survive terminal disconnection. The output is redirected to a log file for later inspection. The PYTHONUNBUFFERED=1 environment variable ensures log output isn't buffered, providing real-time visibility into progress.

Assumptions and Risks

This launch rests on several assumptions, some explicit and some implicit:

OpenRouter reliability: The script assumes OpenRouter's API will maintain 2000 concurrent connections without rate limiting or degradation. This is aggressive — most API providers recommend much lower concurrency. The script likely includes retry logic and exponential backoff, but the assumption that this concurrency level is sustainable is untested at this scale.

Structural consistency: The token reconstruction assumes OpenRouter's response format remains consistent across all providers and all requests. If a provider returns a slightly different format (e.g., different handling of reasoning vs. content), the reconstruction could produce incorrect token sequences.

Cost predictability: The $100 credit balance and ~$86 projected cost assume no unexpected failures that require retrying expensive long samples. A single failed long sample retried multiple times could significantly impact the budget.

Data quality equivalence: The assumption that OpenRouter-generated text is identical to what the local model would generate is reasonable — both run the same model weights. However, different providers may use different quantization methods, sampling parameters, or even different model versions, potentially introducing subtle distribution shifts.

Resume correctness: The script assumes it can correctly identify which samples have been completed by reading the existing raw_responses.jsonl files. If the format of previously written samples differs from what the script expects (e.g., the local SGLang script wrote output_ids while OpenRouter writes reconstructed IDs), resume could produce duplicates or gaps.

The Significance: A Pipeline in Transition

This message marks the transition between two major phases of the EAGLE-3 training pipeline. Phase 1 — data generation — is now fully in motion, with OpenRouter handling the heavy lifting. Phase 2 — hidden state extraction — awaits completion of this run. The assistant has already begun planning for this transition, noting that the old 924GB of 10K hidden states are ready for deletion to free space, and that the A1 dataset's 2,800 ultra-long samples (44.9M tokens) dominate the token budget and may need special handling.

The output of this message is not just a running process — it's the entire B3-B8 dataset being generated in parallel, at a scale that local hardware couldn't match. Within 33 minutes, all datasets would be complete at a cost of ~$86, demonstrating the economic viability of API-based data generation for this use case.

The thinking visible in the message's brief commentary — "B1 and B2 are already done... let the token budget skip B1/B2" — reveals a pragmatic, systems-level mindset. Rather than over-engineering the launch command, the assistant leverages the script's built-in resume logic to handle dataset exclusion naturally. This is the mark of a well-designed pipeline: the launch command is simple because the complexity is already encoded in the tooling.

Conclusion

Message [msg 4039] captures the moment when careful preparation meets execution. The weeks of environment setup, the hours of debugging token reconstruction, the validation of structural consistency — all culminate in a single nohup command. The process listing confirms the script is running, consuming CPU cycles and API credits, generating the training data that will eventually become an EAGLE-3 draft model. It's a quiet moment in a noisy pipeline, but one that represents a critical transition: from preparation to production, from local to cloud, from tens of samples to thousands.