The Pivot: From Local GPUs to OpenRouter API for EAGLE-3 Training Data

"Good — standard messages format. Now let me check which provider slugs to use for the ignore list."

In the sprawling, multi-session effort to train a custom EAGLE-3 speculative decoding drafter for the Kimi K2.5 model, message [msg 4009] marks a quiet but decisive inflection point. It is the moment the assistant abandons local GPU inference and pivots to the OpenRouter API for generating the massive training dataset needed to improve the drafter's accuracy. The message itself is brief—a few lines of reasoning, a file write, and a scattering of LSP diagnostics from an unrelated file—but it encapsulates a complex chain of technical decisions, budget calculations, and architectural tradeoffs that had been building across dozens of preceding messages.

The Context: Why Local Inference Was Failing

To understand why this message matters, one must appreciate the dead end it resolves. The assistant had been running an elaborate local inference pipeline on a machine with 8× RTX PRO 6000 Blackwell GPUs, serving the Kimi K2.5 model via SGLang to generate training data for an EAGLE-3 drafter. The initial 10K-sample dataset had produced a drafter that achieved exactly zero acceptance rate—meaning the speculative decoding was doing no useful work at all ([msg 3990]). The root cause was identified: the auxiliary hidden state capture mechanism wasn't properly activated for the KimiK25 model architecture. After fixing that bug, the assistant benchmarked the improved EAGLE-3 setup at 82.3 tok/s—still below the 90 tok/s baseline without speculative decoding ([msg 3991]).

The conclusion was stark: the drafter needed significantly more training data, perhaps 10× more. But generating that data locally would require running inference on tens of thousands of prompts, each producing thousands of tokens of output. With the local SGLang server already tuned to its limits, this would take hours or days, consuming GPU cycles needed for the actual training runs.

The assistant had been exploring an alternative: using the OpenRouter API to outsource inference to third-party providers running Kimi K2.5. This would free up local GPUs for training while potentially completing data generation faster through massive parallelism.

The Message: A Decision Point

Message [msg 4009] is the moment this alternative becomes real. The assistant writes:

"Now let me write the script. This runs locally (not on the container) since it just needs network access to OpenRouter and disk access to write results. Actually, the data files are on the container at /data/eagle3/synth_100k/prepared/. I'll need to either: 1. Run the script on the container (it has the prompts and output files) 2. Copy prompts locally, run here, copy results back. Running on the container makes more sense since all the data is there and we need the tokenizer."

This internal debate reveals a critical architectural insight: the OpenRouter API returns text, not token IDs. The local inference pipeline had been working with raw token IDs—the integer indices representing each subword unit in the model's vocabulary. The training data for EAGLE-3 required output_ids arrays: the exact token-level representation of each generated response. If the assistant ran the script on its local machine (where the tokenizer was available via Hugging Face libraries), it could download the prompts, send them to OpenRouter, receive text responses, and tokenize them locally. But the prompts lived on the remote container, and copying them would add complexity.

The decision to run on the container was pragmatic: the data was there, the tokenizer was there (installed alongside the model), and the script could write results directly to the output directory. The assistant would write the script locally, then SCP it to the container for execution.

The Hidden Complexity: Reconstructing Token IDs from Text

The most technically challenging aspect of this pivot—barely visible in the message itself but extensively researched in the preceding messages—was the problem of reconstructing exact Kimi K2.5 token IDs from OpenRouter's text responses. This was not a trivial tokenization exercise.

The Kimi K2.5 model uses a special token structure for its reasoning process. When the model "thinks," it generates tokens wrapped in thinking and response markers. These are not just formatting conventions; they are actual token IDs in the model's vocabulary. The OpenRouter API returns the reasoning content in a separate reasoning field (or prepended to the content depending on provider configuration), but the exact token boundaries matter for training the EAGLE-3 drafter.

The assistant had discovered through careful analysis that the tokenizer's behavior at these boundaries was subtle. For example, the sequence response (space + "response") might be tokenized as a single token or split across multiple tokens depending on the BPE (Byte-Pair Encoding) algorithm's merge rules. The token <|im_end|> was found to be token ID 163586, not 163533 as might be naively expected. These details mattered because the EAGLE-3 training process required exact token-level alignment between the draft model's predictions and the target model's outputs.

Furthermore, the assistant had to account for tool call tokens. When the model generates function calls (a key capability of Kimi K2.5), those appear as structured JSON in the response. If the tools parameter isn't sent to the API, these tool calls survive as raw text in the content field—which was actually the desired behavior for training data, since the EAGLE-3 drafter needed to learn to predict tool call tokens as well.

The Budget Calculus

The preceding messages ([msg 3998] through [msg 4002]) reveal an extensive budget analysis that underpins this pivot. The assistant had researched OpenRouter's provider landscape for Kimi K2.5, discovering that every provider runs the native INT4 quantized weights (the only publicly available release). The key exclusions were Fireworks (running NVFP4, a lower-precision format that could degrade quality) and BaseTen (running FP4 and currently down with a status of -5).

The cost analysis was sobering. At the cheapest provider rate of $2.20 per million output tokens (Chutes), generating 60 million output tokens across six datasets would cost approximately $132—exceeding the $100 budget. The assistant calculated that capping at 7 million tokens per dataset would bring the total to about $94.30, fitting within budget with a small buffer.

But there was another complication: reasoning tokens. Kimi K2.5 is a "thinking" model that generates extensive internal reasoning before producing its final answer. These reasoning tokens are billed as output tokens. The assistant's analysis of the B2 dataset showed an average of 3,500 tokens per response, while B3 (Magicoder) averaged only 1,232 tokens. The variance in reasoning length across different prompt types made precise budget planning difficult.

The solution was to implement budget-aware execution: the script would track spending via OpenRouter's credits API and stop gracefully when funds ran low, with robust resume support so it could be restarted later.

The Script Architecture

The assistant wrote run_inference_openrouter.py with several key design decisions:

  1. Provider routing: Use OpenRouter's provider.ignore field to exclude Fireworks and BaseTen, and provider.sort: "price" to automatically route to the cheapest available provider. This ensured cost efficiency without hardcoding provider selection.
  2. Massive concurrency: 2,000 parallel requests. This was the key advantage over local inference—the local SGLang server could handle perhaps 8-16 concurrent requests before degrading, while OpenRouter's distributed infrastructure could absorb thousands of simultaneous calls.
  3. Exponential backoff: On 429 (rate limit) and 5xx (server error) responses, the script would back off exponentially, preventing cascading failures while maximizing throughput.
  4. Resume support: The script would append to a JSONL file and skip completed sample IDs on restart, allowing the generation to be interrupted and resumed without losing progress.
  5. Spend tracking: Periodic checks against OpenRouter's credits API to stop before exceeding budget.

Assumptions and Risks

The pivot to OpenRouter rested on several assumptions that deserve scrutiny:

Assumption 1: Provider equivalence. The assistant assumed that all providers running the "same" INT4 weights would produce equivalent outputs. In practice, different providers may use different inference engines (vLLM, TGI, SGLang), different batch sizes, and different sampling parameters. Even with identical weights, these implementation differences can produce divergent outputs, especially for a stochastic sampling process.

Assumption 2: Tokenizer compatibility. The assistant assumed that tokenizing OpenRouter's text responses with the local Hugging Face tokenizer would produce the exact same token IDs that the provider's inference engine would have produced internally. This is generally true for standard tokenizers, but edge cases—particularly around special tokens, whitespace handling, and Unicode normalization—could introduce subtle mismatches.

Assumption 3: Reasoning token fidelity. The assistant assumed that OpenRouter would faithfully return the model's complete reasoning trace. The research had revealed that some providers silently drop reasoning tokens ([msg 4006]). If a provider stripped the thinking... response structure, the tokenized output would be missing critical training signal.

Assumption 4: Cost predictability. The budget analysis assumed relatively stable pricing and token counts. In practice, OpenRouter pricing can fluctuate based on provider availability, and the model's reasoning depth varies dramatically by prompt, making token counts hard to predict.

The Output Knowledge Created

This message produced the run_inference_openrouter.py script—a concrete artifact that transformed the assistant's research into executable code. The script embodied several novel insights:

The Broader Significance

Message [msg 4009] represents a classic engineering tradeoff: build vs. buy, local vs. cloud, control vs. convenience. The local inference pipeline offered complete control—exact token IDs, full access to hidden states, no per-token costs—but was bottlenecked by GPU memory and compute. The OpenRouter API offered massive parallelism and zero GPU overhead but introduced cost, latency, and fidelity risks.

The assistant's decision to split the pipeline—using the API for text generation and local GPUs for hidden state extraction—was a sophisticated compromise that leveraged each approach's strengths. It's the kind of architectural decision that emerges not from abstract planning but from the concrete experience of hitting walls: the local pipeline was too slow for 60M tokens of generation, but the API couldn't provide the hidden states needed for EAGLE-3 training.

In the end, this pivot proved spectacularly successful. The OpenRouter script completed all six B-datasets in approximately 33 minutes at a cost of ~$86 ([chunk 29.0]), generating the training data that would eventually produce a working EAGLE-3 drafter. The message that launched this pivot—brief, technical, and seemingly mundane—was the turning point that made the rest of the project possible.