The Quantization Hunt: A Single Search Query That Reshaped a $100 Budget

Introduction

In the sprawling transcript of an opencode coding session spanning thousands of messages, most individual messages blend into the flow of tool calls, bash commands, and status updates. But occasionally, a message that appears trivial on the surface — a single web search — reveals itself as a critical pivot point in the entire session's trajectory. Message [msg 4001] is exactly such a moment: a lone Exa web search query asking about DeepInfra's quantization precision for the Kimi K2.5 model. On its face, it is barely more than a line of JSON. Yet this search, and the reasoning that prompted it, would determine whether the entire EAGLE-3 training data pipeline could proceed within a strict $100 budget, and whether the resulting data would be compatible with the local deployment at all.

The Message Itself

Let us first quote the message exactly as it appears in the conversation:

[assistant] [exa_web_search_exa] {"numResults":5,"query":"DeepInfra kimi-k2.5 what quantization precision do they use INT4 FP8 BF16"} Title: Kimi K2.5: Still Worth It After Two Weeks? - Hugging Face Published Date: 2026-02-23T16:43:31.204Z URL: https://huggingface.co/blog/mlabonne/kimik25 Text: Kimi K2.5: Still Worth It After Two Weeks?

[Back to Articles]

# Kimi K2.5: Still Worth It After Two Weeks?

[Community Article] Published February 23, 2026

[Upvote 5]

[Maxime Labonne mlabonne Follow]

Beijing-based Moonshot AI released Kimi K2.5 on January 27, 2026. Beyond traditional claims on benchmarks like HLE (50.2% with too...

The message contains a single tool invocation: an Exa web search configured to return five results, querying specifically about DeepInfra's quantization precision for Kimi K2.5. The search returned a Hugging Face community article by Maxime Labonne — a well-known figure in the open-source LLM community. The result text is truncated, showing only the article's introduction.

Why This Message Was Written: The Reasoning and Motivation

To understand why this seemingly narrow query was necessary, we must reconstruct the chain of reasoning that led to it. The story begins in [msg 3988], where the user issued a pivotal instruction: switch from local GPU inference to OpenRouter API for generating EAGLE-3 training data. The user specified several critical constraints: run 2,000 inferences in parallel, maintain a 10-million-token output budget per dataset, handle errors and rate limits gracefully, ensure resumability if the $100 account balance runs out, and — most crucially — "pick kimi-k2.5 providers who don't quantize (fireworks e.g. runs nvfp4)."

This last constraint is the key to understanding [msg 4001]. The user was concerned about quantization because the entire EAGLE-3 training pipeline depends on generating training data that is distributionally compatible with the locally deployed model. The local deployment runs the Kimi K2.5 model at its native INT4 quantization (a quantization-aware training, or QAT, precision that ships as the official release). If training data were generated by a provider that applies additional quantization — reducing precision below INT4 to FP4, NF4, or NVFP4 — the resulting token distributions would differ from what the local model produces. The EAGLE-3 draft model trained on such data might learn patterns that don't match the local deployment, potentially rendering the entire training exercise worthless.

The assistant had already begun investigating this question in [msg 4000], running an initial search about DeepInfra's quantization and discovering that the estimated cost of generating 60 million output tokens across six datasets would exceed the $100 budget. The cost projection — $136+ at the cheapest rates — forced a fundamental re-evaluation. If the assistant could confirm that all providers were running the same native INT4 weights, then the cheapest providers could be used without quality concerns, and the budget could be stretched further. But if some providers were running lower-precision quantizations, they would need to be excluded, potentially driving costs even higher.

This is the immediate motivation for [msg 4001]: the assistant needed to resolve the quantization question definitively before proceeding to write the OpenRouter inference script. The search query was refined from the previous attempt — now explicitly listing the candidate precisions (INT4, FP8, BF16) and targeting DeepInfra specifically, which appeared to be one of the cheapest "unknown quantization" providers in the pricing table the assistant had compiled in [msg 3999].

The Assumptions Embedded in the Query

The search query itself encodes several assumptions worth examining. First, the assistant assumed that DeepInfra's quantization precision was undocumented or unclear — that it wasn't simply listed in the OpenRouter API response. This assumption was validated by the earlier analysis in [msg 3999], which showed DeepInfra's quantization as "unknown" in the provider table. The assistant had already fetched the OpenRouter model endpoints API ([msg 3997]) and parsed the provider list, but the quantization field was not consistently populated across all providers.

Second, the assistant assumed that the answer could be found through web search — that someone in the community had documented DeepInfra's deployment configuration for Kimi K2.5. This was a reasonable assumption given that Kimi K2.5 was a popular model and DeepInfra is a well-known inference provider, but it was not guaranteed. The search result returning a Hugging Face article by Maxime Labonne — a prominent figure who regularly benchmarks and analyzes LLM deployments — validated this assumption.

Third, the assistant implicitly assumed that the Hugging Face article would contain the relevant technical detail. The article's title, "Kimi K2.5: Still Worth It After Two Weeks?", suggested a retrospective analysis that would likely discuss deployment configurations across different providers. This turned out to be correct, as the subsequent message ([msg 4002]) reveals the key finding: "The model ships natively as INT4 (QAT). DeepInfra runs K2-Thinking as FP4. The 'unknown' quantization providers are almost certainly running the official INT4 weights — that's all that's publicly available (there is no BF16/FP8 release)."

Input Knowledge Required

To fully understand this message, one needs considerable background knowledge spanning several domains. First, familiarity with the concept of quantization in large language models — the practice of reducing numerical precision to decrease memory footprint and increase inference speed — is essential. Understanding the hierarchy of precisions (BF16 > FP8 > INT4 > FP4/NF4) and the implications for model quality is necessary to grasp why the user cared about this distinction at all.

Second, knowledge of OpenRouter's architecture is required. OpenRouter is an API gateway that routes requests to multiple inference providers, each of which may deploy the same model with different hardware, different quantization levels, and different pricing. The provider object in OpenRouter's API allows excluding specific providers or ordering preferences — this is the mechanism the assistant would use to filter out Fireworks (NVFP4) and BaseTen (FP4).

Third, understanding the EAGLE-3 training pipeline is crucial. The assistant and user had been working for hours — across multiple segments of the conversation — to build a speculative decoding system where a lightweight "draft" model predicts the next several tokens, and the full "target" model verifies them in parallel. The training data for the draft model must come from the same distribution as the target model's outputs. If the data is generated by a differently-quantized version of the model, the distribution shift could render the draft model useless.

Fourth, the reader must understand the budget arithmetic. The assistant had calculated that 60 million output tokens at $2.20 per million tokens would cost $132, exceeding the $100 account balance. But this assumed the cheapest providers were acceptable. If the cheapest providers turned out to use lower-precision quantization and had to be excluded, the effective cost per token would rise, making the budget problem even worse.

Output Knowledge Created

The direct output of this message is minimal: a single search result pointing to a Hugging Face article. But the knowledge created by the reasoning chain that this message belongs to is substantial. In the following message ([msg 4002]), the assistant synthesizes several critical conclusions:

All providers run INT4: The native release of Kimi K2.5 is INT4 with quantization-aware training. There is no BF16 or FP8 public release. Therefore, every provider is fundamentally running the same weights, and the "unknown quantization" providers are almost certainly using the official INT4 weights.
Fireworks runs NVFP4 and BaseTen runs FP4: These are the exceptions — providers applying additional quantization below INT4. The user had already identified Fireworks, and the assistant confirmed BaseTen (which was also down with status -5).
The budget can work: With the confirmation that most providers run native INT4, the assistant could use OpenRouter's automatic routing (excluding only Fireworks and BaseTen) to get the cheapest rates. This meant the $100 budget could cover approximately 43-45 million output tokens — about 7.2 million per dataset instead of the originally planned 10 million.
A cost-cap strategy is viable: Rather than setting per-dataset token budgets, the assistant could let inference run until the money runs out, relying on the resume-safe design to continue later if needed. These conclusions fundamentally reshaped the approach to the entire data generation pipeline. Without this research, the assistant might have either (a) excluded too many providers, driving costs up and making the $100 budget insufficient, or (b) included providers running degraded quantizations, producing training data that would not transfer to the local deployment.

The Thinking Process Visible in the Reasoning

While [msg 4001] itself contains no explicit reasoning text — it is purely a tool call — the surrounding messages reveal a sophisticated chain of reasoning. In [msg 3999], the assistant built a detailed provider pricing table, categorizing each provider's quantization as "int4," "fp4," or "unknown." This table was the foundation for the subsequent investigation.

In [msg 4000], the assistant explicitly states its reasoning: "Hmm, $136+ for 60M output tokens exceeds the $100 budget. Let me research what quantization the 'unknown' providers actually use, and figure out the right budget approach." This reveals that the budget constraint was the trigger for the quantization research — the assistant realized it needed to know whether the cheapest providers were acceptable before it could plan the budget.

The first search in [msg 4000] returned the same Hugging Face article, but the assistant apparently didn't get enough detail from it — hence the refined second search in [msg 4001] with a more specific query. This iterative refinement demonstrates a key aspect of the assistant's problem-solving approach: when initial results are insufficient, it narrows the query rather than accepting ambiguity.

The synthesis in [msg 4002] is where the real intellectual work happens. The assistant connects the search result (the Hugging Face article) with its prior knowledge (the model ships natively as INT4, no BF16 release exists) and the provider pricing data to reach a conclusion. It then performs detailed budget arithmetic, calculating exact token counts and costs for different per-dataset budgets. This is not a simple lookup — it's a multi-step reasoning process combining external information, internal knowledge, and mathematical calculation.

Potential Mistakes and Incorrect Assumptions

The most significant potential mistake in this message is the assumption that the Hugging Face article's information about DeepInfra is accurate and current. Inference providers change their deployment configurations frequently — a provider running INT4 today might switch to FP4 tomorrow, or vice versa. The assistant does not verify the article's claims against any official DeepInfra documentation or by testing the API directly.

Additionally, the assistant assumes that "unknown quantization" for providers like Together, Novita, and Venice means they are running the native INT4 weights. This is a reasonable inference — if there were no public release of higher-precision weights, then INT4 is the only option — but it is not definitively proven. A provider could theoretically apply its own quantization on top of the INT4 weights, producing an even lower-precision variant.

There is also a subtle assumption about the relationship between quantization and token distribution. The assistant assumes that INT4 quantization produces tokens that are "close enough" to the local deployment's distribution for EAGLE-3 training to work. But the local deployment also runs INT4 — so the question is whether different INT4 implementations (different hardware, different kernels, different calibration data) produce meaningfully different token distributions. This is an open question that the assistant does not address.

Conclusion

Message [msg 4001] is a study in how a single web search can encapsulate an entire decision-making process. The query about DeepInfra's quantization precision was not idle curiosity — it was the critical piece of information needed to resolve a tension between quality constraints (use only non-quantized providers) and budget constraints (stay within $100). The assistant's methodical approach — building a provider table, identifying unknowns, searching iteratively, and synthesizing results into actionable conclusions — demonstrates the kind of structured reasoning that makes AI-assisted coding sessions productive.

The message also reveals something important about the nature of these sessions: the most consequential work often happens in the research and reasoning steps, not in the code writing. The script that eventually emerged from this investigation — run_inference_openrouter.py — would be shaped entirely by the conclusions reached in [msg 4002], which were made possible by the search in [msg 4001]. Without understanding that all providers run native INT4, the assistant might have written a script that was either too restrictive (driving costs over budget) or too permissive (producing incompatible training data). The search query, for all its brevity, was the fulcrum on which the entire pipeline turned.