The Quantization Detective: How One Message Uncovered the True Cost of OpenRouter Inference for EAGLE-3 Training

Introduction

In the sprawling, multi-day effort to train a custom EAGLE-3 speculative decoding drafter for the Kimi-K2.5 model, there comes a moment where the entire trajectory of the project pivots on a single insight. Message [msg 4002] is that moment. It is a message that appears, on its surface, to be a routine budget calculation—a simple spreadsheet exercise of dollars-per-million-tokens. But beneath the arithmetic lies a fascinating piece of detective work: determining what quantization precision every major OpenRouter provider actually runs, discovering that the model ships natively as INT4 with no higher-precision release available, and realizing that the $100 budget cap fundamentally cannot support the original 10-million-token-per-dataset plan. This message is where the assistant transitions from researching to deciding, and where the user's ambitious data generation plan collides with economic reality.

Context: The Road to OpenRouter

To understand why message [msg 4002] matters, we must first understand the predicament that led to it. The project had been generating synthetic training data for EAGLE-3 using local GPU inference on a machine with 8 RTX PRO 6000 Blackwell GPUs. The local inference pipeline, running SGLang with the Kimi-K2.5 model, was producing responses at roughly 850–1000 tokens per second. With six remaining datasets (B3 through B8) each needing 10 million output tokens, the estimated wall-clock time was 14 to 19 hours. This was slow but feasible.

Then the user made a strategic decision in [msg 3988]: abandon local inference entirely and switch to the OpenRouter API. The reasoning was clear—OpenRouter could run 2,000 concurrent requests, massively parallelizing the work and potentially completing all datasets in minutes rather than hours. The user specified a $100 budget cap, instructed the assistant to exclude providers that quantize below the native precision (specifically calling out Fireworks' NVFP4), and demanded robust resume support in case the budget ran out mid-run.

This pivot introduced a new set of challenges. The assistant first had to research which OpenRouter providers offered Kimi-K2.5, what quantization they used, and whether the $100 budget could cover the 60 million output tokens needed (6 datasets × 10 million tokens each). The research phase spanned several messages: fetching the OpenRouter model list ([msg 3992]), parsing provider endpoints ([msg 3994]), scraping provider status pages ([msg 3996]), and analyzing endpoint pricing data ([msg 3997]). By the time we reach message [msg 4002], the assistant has accumulated a wealth of raw data and is ready to synthesize it into actionable conclusions.

The Core Discovery: Native INT4 and the Quantization Landscape

The most important finding in message [msg 4002] is stated plainly in its opening line: "The model ships natively as INT4 (QAT)." This is the key that unlocks everything else. The assistant has discovered that Moonshot AI's official release of Kimi-K2.5 is an INT4-quantized model trained with Quantization-Aware Training (QAT). There is no BF16 release, no FP8 release, no higher-precision variant available anywhere. The Hugging Face discussion boards confirm this absence.

This discovery has profound implications for provider selection. The user's original instruction was to "pick kimi-k2.5 providers who don't quantize"—the assumption being that some providers run the model at full precision while others apply additional quantization. But if the model only exists as INT4, then every provider is running the same INT4 weights. The question shifts from "who doesn't quantize?" to "who further quantizes below INT4?"

The assistant identifies exactly two providers that apply additional quantization below INT4:

  1. Fireworks — Runs NVFP4 (NVIDIA's FP4 format), which is a lower precision than INT4. The user explicitly flagged this one.
  2. BaseTen — Tagged as FP4 in the OpenRouter endpoint data, and its status is -5 (down/unavailable anyway). All other providers—Chutes, DeepInfra, SiliconFlow, AtlasCloud, Together, Novita, Parasail, Moonshot AI (the official provider), and Venice—are running the native INT4 weights. The "unknown" quantization tags in the OpenRouter API data are simply a reflection of the fact that INT4 is the default and doesn't need to be explicitly advertised. This analysis is remarkably thorough. The assistant doesn't just accept the API data at face value; it cross-references the OpenRouter endpoint data with external research (the Hugging Face blog post and discussion boards) to confirm that no BF16 release exists. It also reasons from first principles: if the model is only publicly available as INT4, then any provider not explicitly advertising a different quantization must be running INT4. This is a logical inference, not just an API scrape.

The Budget Reality Check

Having established which providers are acceptable, the assistant turns to the budget calculation. This is where the message's second major insight emerges: the $100 budget is insufficient for the original plan.

The assistant performs a series of calculations that reveal the arithmetic problem:

Assumptions and Their Implications

Message [msg 4002] rests on several assumptions, some explicit and some implicit. Understanding these assumptions is crucial for evaluating the correctness of the assistant's conclusions.

Assumption 1: INT4 is the only available precision. The assistant cites a Hugging Face blog post and discussion boards as evidence that no BF16 release exists. This is a reasonable conclusion, but it depends on the accuracy and timeliness of those sources. If Moonshot AI were to release a BF16 variant in the future, or if one existed but wasn't publicly documented, this assumption would be invalid. However, given the project's timeframe and the thoroughness of the research, this is a safe assumption.

Assumption 2: Providers with "unknown" quantization are running native INT4. This is a logical inference but not a verified fact. The assistant explicitly acknowledges the uncertainty by listing DeepInfra, Together, Novita, and Venice as "unknown" before reasoning that INT4 is the only option. There is a small risk that a provider might apply additional quantization without advertising it in the OpenRouter tag field. However, the assistant's reasoning is sound: if the model only exists as INT4 and a provider doesn't claim to use a different quantization, INT4 is the default.

Assumption 3: Average response length is ~2,000 tokens. This estimate is used to calculate the number of requests and input token costs. The assistant has empirical data from the local inference run showing that B2 (opencodeinstruct) averaged ~3,500 tokens per response while B3 (magicoder) averaged ~1,232 tokens. The 2,000-token average is a middle-ground estimate. If actual responses are longer, the number of requests decreases but the output token cost remains the same (since we're budgeting by output tokens, not by request count). If responses are shorter, more requests are needed, increasing input token costs slightly. The sensitivity is low—input costs are a small fraction of the total.

Assumption 4: The $100 budget is a hard cap. The user stated "Account has $100, if it runs out make sure it's resumable." The assistant interprets this as a strict budget constraint and designs around it. An alternative interpretation would be that the user is willing to spend up to $100 per session but could add more funds. The assistant's conservative interpretation is prudent.

Assumption 5: Provider uptime and reliability are acceptable. The assistant checks uptime statistics from the OpenRouter endpoint data (DeepInfra at 99.0%, SiliconFlow at 95.6%, Novita at 100.0%, etc.) but doesn't explicitly factor this into the provider selection beyond excluding the downed BaseTen provider. The assumption is that the remaining providers have sufficient uptime for the batch inference task.

Mistakes and Incorrect Assumptions

While message [msg 4002] is generally well-reasoned, there are a few points worth examining critically.

The budget calculation slightly underestimates input costs. The assistant calculates input costs based on 200 tokens per prompt and 2,000 tokens per response. However, the prompts for datasets like B2_opencodeinstruct include system prompts, conversation histories, and formatting tokens that may push the average input length higher. Additionally, the tokenizer for Kimi-K2.5 (listed as "Other" in the architecture data) may have different tokenization characteristics than the assistant assumes. The input cost error is small (input is only ~2% of total cost), so this doesn't materially affect the conclusion.

The assistant doesn't consider prompt caching discounts. OpenRouter offers reduced pricing for cached input reads (the input_cache_read field in the pricing data shows $0.00000015 per token for Chutes, compared to $0.00000045 for uncached prompt tokens). If the inference script sends many similar prompts (e.g., the same system prompt with varying user messages), prompt caching could reduce input costs by up to 67%. The assistant doesn't factor this in, which means the actual cost could be slightly lower than estimated.

The "let it run until money runs out" strategy is mentioned but not fully developed. The assistant presents this as an alternative but doesn't explore its implications in depth. If the script runs until the budget is exhausted, it will stop partway through a dataset. The resume-safe design ensures it can restart, but the user would need to add funds to the OpenRouter account. This creates a dependency on the user's willingness to top up the account, which may not be desirable.

The assumption that all "unknown" providers run INT4 could be wrong for one provider. DeepInfra is listed as running K2-Thinking as FP4 (the assistant mentions this in the opening line). FP4 is a lower precision than INT4. If DeepInfra applies FP4 quantization to Kimi-K2.5, it should be excluded alongside Fireworks and BaseTen. However, the assistant's research specifically says "DeepInfra runs K2-Thinking as FP4"—this refers to a different model variant (K2-Thinking, not K2.5). The assistant correctly distinguishes between models, but the distinction is subtle and could be missed by a reader.

Input Knowledge Required

To fully understand message [msg 4002], the reader needs familiarity with several concepts:

Quantization and precision formats: The message discusses INT4, FP4, NVFP4, BF16, FP8, NF4, and QAT. Understanding the hierarchy of precision (BF16 > FP8 > INT4 > FP4/NF4) and the implications for model quality is essential. The assistant's concern is that providers applying quantization below INT4 (like NVFP4 or FP4) would produce lower-quality outputs, which would degrade the EAGLE-3 training data.

OpenRouter's provider ecosystem: OpenRouter is an API gateway that routes requests to multiple LLM providers. Each provider may run the same model at different precisions and prices. The assistant uses OpenRouter's /endpoints API to discover provider-specific details.

Token budgets and billing: The assistant calculates costs in dollars per million tokens, accounting for separate input and output pricing. Understanding that reasoning tokens are billed as output tokens is important for accurate cost estimation.

The EAGLE-3 training pipeline: The broader context is that these inference runs generate synthetic responses that will be used to train an EAGLE-3 speculative decoding drafter. The quality of the training data directly affects the drafter's accuracy. This is why the assistant is so careful about provider quantization—lower-quality data would produce a worse drafter.

The dataset structure: The project has datasets labeled B1 through B8 (and A1/A2 for longer samples). B1 and B2 are already completed via local inference. B3 through B8 are the remaining targets for OpenRouter inference.

Output Knowledge Created

Message [msg 4002] produces several concrete outputs that shape the subsequent work:

  1. Provider exclusion list: Fireworks (NVFP4) and BaseTen (FP4, also down) are excluded. All other providers are acceptable for running native INT4 inference.
  2. Budget constraint: The $100 budget supports approximately 42–45 million output tokens, not the 60 million originally planned. This translates to roughly 7 million tokens per dataset instead of 10 million.
  3. Strategy options: The assistant identifies two viable paths: (a) reduce per-dataset token budgets to 7 million, or (b) keep 10 million but make the script resume-safe so it stops when funds run out.
  4. Provider routing strategy: Rather than pinning a specific provider, the assistant plans to use OpenRouter's automatic routing (excluding Fireworks and BaseTen) to get the cheapest available pricing at each request.
  5. Confidence in provider quality: The research establishes that all non-excluded providers run the same INT4 weights, so there is no quality difference between them. This simplifies the provider selection to a purely cost-based decision.

The Thinking Process

The assistant's reasoning in message [msg 4002] follows a clear arc from discovery to decision. Let me trace the chain of thought visible in the message.

Step 1: Synthesize the research. The message opens with a summary of the key finding: "The model ships natively as INT4 (QAT)." This is the conclusion drawn from the Hugging Face research and the OpenRouter endpoint analysis. The assistant immediately connects this to the provider quantization question: since only INT4 exists, providers with "unknown" quantization must be running INT4.

Step 2: Identify the exceptions. The assistant identifies Fireworks and BaseTen as providers that apply additional quantization below INT4. Fireworks is flagged for NVFP4 (as the user specified), and BaseTen is flagged for FP4 and is also down. These are the only two providers to exclude.

Step 3: Run the budget numbers. The assistant executes a Python script that calculates costs at various per-dataset token budgets. The script accounts for input and output tokens separately, uses the cheapest available pricing ($2.20/M output, $0.45/M input), and estimates request counts based on average response length.

Step 4: Interpret the results. The numbers show that 7 million tokens per dataset fits within $100, 8 million does not, and 10 million is significantly over budget. The assistant presents these findings without editorializing, allowing the data to speak.

Step 5: Propose alternatives. The assistant mentions the "let it run until money runs out" strategy as an alternative, showing awareness that the user might prefer to maximize data quantity rather than strictly adhering to the per-dataset budget.

Step 6: Note the reasoning token factor. The assistant explicitly calls out that reasoning tokens are billed as output tokens, which is a subtle but important point. This shows an understanding of how modern LLM APIs work and prevents the reader from underestimating costs.

The thinking process is methodical and data-driven. The assistant doesn't make gut-feeling decisions; it runs calculations, checks sources, and presents evidence. This is characteristic of the best kind of technical decision-making—transparent, verifiable, and grounded in facts.

Why This Message Matters

Message [msg 4002] is pivotal for several reasons. First, it resolves the provider quantization question definitively. The user's instruction to "pick providers who don't quantize" was based on an assumption that some providers run full precision. The assistant's research reveals that this assumption is incorrect—there is no full-precision version to run. This changes the provider selection from a quality-based decision to a cost-based one.

Second, it surfaces the budget constraint as a hard limit on the project's scope. The original plan of 10 million tokens per dataset was formulated when inference was running locally, where the only cost was time. Switching to OpenRouter introduces a monetary cost that the $100 budget cannot fully cover. This is a classic engineering tradeoff: speed (OpenRouter's 2,000 concurrent requests) versus cost (the $100 cap limits total data volume).

Third, it demonstrates the importance of understanding the full cost structure before committing to a plan. The assistant could have simply written the OpenRouter script with 10-million-token budgets and let it fail when the money ran out. Instead, it proactively calculated the costs and presented the tradeoffs to the user, enabling an informed decision.

Fourth, the message showcases the value of deep research. The assistant didn't just accept the OpenRouter API data at face value; it cross-referenced with Hugging Face discussions, checked provider status pages, and reasoned about the implications of the model's native precision. This thoroughness prevents a potential quality disaster—if the assistant had included Fireworks or BaseTen without understanding their quantization, the EAGLE-3 training data would have been generated at FP4/NVFP4 precision, potentially degrading the drafter's quality.

Conclusion

Message [msg 4002] is a masterclass in technical decision-making under constraints. It begins with a research question—what quantization do OpenRouter providers use?—and ends with a clear, actionable plan for spending $100 across six datasets. Along the way, it discovers that the model only exists as INT4, identifies two providers to exclude, calculates precise cost estimates, and presents the user with a clear tradeoff between data volume and budget.

The message's greatest strength is its transparency. Every assumption is stated, every calculation is shown, and every conclusion is supported by evidence. The assistant doesn't hide uncertainty behind confident assertions; it explicitly notes where information is inferred rather than verified (the "unknown" quantization providers) and where alternative strategies exist (the resume-safe fallback).

For anyone building a similar pipeline—generating training data via API inference for a quantized model—this message offers a template for how to approach provider selection and budget planning. The key lesson is simple but often overlooked: understand what precision your model actually ships in, verify what each provider runs, and calculate costs before you start spending. A few minutes of research and arithmetic can save hours of wasted inference and prevent the accumulation of unusable training data.