The Pivot to OpenRouter: Researching Provider Routing for EAGLE-3 Training Data Generation

In the middle of an intensive machine learning engineering session focused on training a custom EAGLE-3 speculative decoding drafter for the Kimi-K2.5 language model, a critical strategic pivot occurs. The subject message, <msg id=4004>, is deceptively simple on its surface — a single tool call to an external search engine querying for "OpenRouter API provider routing exclude specific providers order preference python example." But this query represents the fulcrum of a major architectural decision: abandoning local GPU inference in favor of a paid API service for generating training data, and all the technical complexity that entails.

The Context: Why This Message Was Written

To understand why this message exists, we must trace the chain of events that led to it. The session had been running a local SGLang inference server on a machine with 8 GPUs, generating responses for EAGLE-3 training data across multiple datasets (B1-B8). The assistant had estimated 14-19 hours remaining for local inference across 6 remaining datasets, totaling ~60M output tokens at ~900-1200 tok/s server throughput (see <msg id=3987>).

Then the user intervened with a dramatic change of plan in <msg id=3988>:

"Chanfe to use openrouter to run the inference; API key in /tmp/or-key.txt; Pick kimi-k2.5 providers who don't quantize (fireworks e.g. runs nvfp4). Run 2000 inferences in parallel, still 10M tok out per batch. Other steps we still run on my GPUs. Handle request errors / ratelimits. Account has $100, if it runs out make sure it's resumable"

The motivation is clear: speed. Local inference at ~900 tok/s would take nearly a full day. OpenRouter, with 2000 concurrent requests across dozens of provider endpoints, could potentially finish the same work in minutes — albeit at a monetary cost. The user was willing to spend up to $100 of their OpenRouter account balance to accelerate the pipeline.

The user's instruction contained several implicit assumptions that the assistant would need to validate:

  1. That OpenRouter has Kimi-K2.5 providers that don't further quantize the model below its native INT4 precision
  2. That 2000 concurrent requests is feasible without hitting rate limits
  3. That the $100 budget would be sufficient for 6 datasets × 10M tokens each
  4. That the API responses from OpenRouter could be used to reconstruct exact token IDs needed for EAGLE-3 training

The Research Phase Preceding the Subject Message

Before the subject message, the assistant engaged in an extensive research campaign spanning messages <msg id=3992> through <msg id=4003>. This involved:

  1. Fetching the full OpenRouter model list to find Kimi-K2.5 endpoints and their pricing
  2. Parsing provider metadata to identify quantization levels, status, and uptime
  3. Analyzing pricing to determine whether $100 would cover 60M output tokens
  4. Researching quantization via web searches to determine what precision each provider actually uses A critical finding emerged: the Kimi-K2.5 model ships natively as INT4 with quantization-aware training (QAT). There is no BF16 or FP16 release publicly available. This means that every provider is fundamentally running the same INT4 weights — the question is whether any provider applies additional quantization below INT4 (like Fireworks' NVFP4 or BaseTen's FP4). The assistant's pricing analysis in <msg id=4002> revealed a sobering reality: at the cheapest output rate of $2.20/M tokens, 60M output tokens would cost $132 — exceeding the $100 budget. Even with careful provider routing, only ~43M tokens (7M per dataset) would fit within budget.

What the Subject Message Actually Contains

The subject message <msg id=4004> is a single tool invocation:

[assistant] [exa_get_code_context_exa] {"query":"OpenRouter API provider routing exclude specific providers order preference python example","tokensNum":5000}

This is an Exa search (a code/documentation search tool) looking for Python examples of OpenRouter's provider routing API — specifically how to exclude certain providers and specify order preferences.

The search returned documentation from OpenRouter's provider selection guide, which describes the provider object that can be included in chat completion request bodies. Key fields include:

The Reasoning and Decision-Making Process

The assistant's thinking at this point is highly structured. Having already identified the provider landscape, it now needs to translate that knowledge into code. The key decisions being researched are:

  1. Which providers to exclude: Fireworks (NVFP4 — further quantized below INT4) and BaseTen (FP4 — also further quantized, and currently down with status -5)
  2. Which providers to prefer: The cheapest available providers running native INT4 — primarily Chutes ($2.20/M output) and DeepInfra ($2.25/M output)
  3. How to implement provider routing: Using OpenRouter's provider object with order to specify preferred providers, and potentially ignore to exclude specific providers
  4. How to handle the budget constraint: Since 60M tokens at $2.20/M would cost $132, exceeding the $100 budget, the assistant needs to either reduce per-dataset token budgets or implement spend tracking with graceful shutdown The assistant's assumption is that OpenRouter's API supports excluding specific providers by slug. The search results confirm this — the provider object can specify ordering and fallback behavior. However, the assistant hasn't yet confirmed whether OpenRouter supports an explicit "exclude" list or if the only mechanism is to specify an order list that simply omits unwanted providers.

Assumptions and Potential Pitfalls

Several assumptions underpin this research:

Assumption 1: Provider slugs are stable and well-documented. The assistant has been using provider names like "fireworks" and "baseten" as slugs, but the actual API slug format might differ (e.g., "fireworks-ai" or "fireworks-ai-inc"). If the slugs don't match, the exclusion won't work.

Assumption 2: The native INT4 weights are sufficient for EAGLE-3 training data. The entire premise of switching to OpenRouter rests on the belief that INT4-precision outputs are adequate for training a speculative decoding drafter. If the drafter requires higher-precision training data (e.g., from a BF16 local run), the OpenRouter data could introduce systematic biases.

Assumption 3: OpenRouter can sustain 2000 concurrent requests. The user specified 2000 parallel inferences, but OpenRouter may have rate limits per API key, especially for a $100 account. The assistant will need to implement retry logic and adaptive rate limiting.

Assumption 4: Token IDs can be reconstructed from text. This is perhaps the most critical technical assumption. The EAGLE-3 training pipeline requires exact token IDs from the Kimi-K2.5 tokenizer, but OpenRouter returns text strings, not token IDs. The assistant will need to re-tokenize the text responses client-side, which requires knowing the exact tokenizer configuration and handling edge cases like special tokens, BPE boundaries, and tool call formatting.

Input Knowledge Required

To understand and act on this message, the assistant needed:

  1. OpenRouter's API structure: Knowledge that chat completion requests can include a provider object for routing control
  2. Provider quantization landscape: Which providers run INT4 vs. NVFP4/FP4, gathered from the preceding research
  3. Kimi-K2.5 model characteristics: That the model ships natively as INT4 with QAT, and no higher-precision release exists
  4. Budget math: The cost calculations showing $100 would cover ~43M tokens, not the full 60M
  5. The EAGLE-3 training pipeline: Understanding that training data must consist of exact token IDs, not just text

Output Knowledge Created

This message, combined with the surrounding research, produces several concrete outputs:

  1. Provider exclusion strategy: A plan to exclude Fireworks and BaseTen using OpenRouter's provider routing API
  2. Cost model: A revised budget of ~7M tokens per dataset (instead of 10M) to stay within $100
  3. Implementation approach: The assistant now knows the API surface needed to write run_inference_openrouter.py
  4. Risk identification: The token reconstruction problem is now visible as a major technical challenge that will need to be solved

The Broader Significance

This message represents a classic engineering trade-off: time vs. money. The local inference pipeline would take ~18 hours but cost nothing beyond electricity. The OpenRouter approach would complete in ~30 minutes but cost up to $100. The assistant is researching how to execute this trade-off correctly, ensuring that the paid service produces data of equivalent quality to what would have been generated locally.

The provider routing research is particularly nuanced because the user's concern about quantization reflects a deep understanding of model behavior. Lower-precision quantization (FP4, NVFP4) can introduce subtle distributional shifts in outputs. For training a speculative decoding drafter, which must closely mimic the base model's distribution, even small differences in the training data could degrade the drafter's acceptance rate. The assistant's careful exclusion of NVFP4 and FP4 providers shows awareness of this risk.

What makes this message noteworthy is that it sits at the intersection of multiple complex domains: API integration, model quantization, budget management, distributed systems (2000 concurrent requests), and ML training data pipelines. The assistant must synthesize knowledge from all these areas to make the right implementation decisions.

The next message in the conversation, <msg id=4005>, shows the assistant continuing its research by querying for OpenRouter's credits/balance API — another essential piece for implementing spend tracking and graceful shutdown when the $100 budget is exhausted. Together, these research steps form the foundation for the run_inference_openrouter.py script that will be built in the subsequent messages.