The $86 Pivot: Reconstructing Exact Token IDs at 2000 Concurrency for EAGLE-3 Training

Introduction

In the sprawling engineering effort to train an EAGLE-3 speculative decoding drafter for the Kimi K2.5 language model, there came a moment when the entire trajectory of the project shifted. After days of local GPU inference on 8× RTX PRO 6000 Blackwell GPUs — tuning SGLang servers, patching hidden state extraction, debugging zero acceptance rates — the assistant and user faced a stark reality: generating the remaining six datasets locally would take days, if not weeks. The solution was a dramatic pivot to the OpenRouter API, promising to complete all data generation in roughly 33 minutes at a cost of ~$86.

But this speed came with a hidden cost: complexity. OpenRouter is a chat completions API that returns human-readable text, not the raw token IDs that the local SGLang pipeline provided. The EAGLE-3 training pipeline required exact token-level sequences — every <|im_end|>, every </think>, every tool call delimiter had to be precisely the right integer ID in the Kimi K2.5 vocabulary. A single off-by-one error in a special token ID, a BPE boundary mismatch across concatenated text segments, or a silently dropped reasoning field would corrupt the training data and waste the entire budget.

This article tells the story of how the assistant navigated this pivot — building a 2000-concurrent-request inference script, reconstructing exact token IDs from text through meticulous tokenizer investigation, completing all datasets in record time, and scoping the remaining phases of the pipeline. It is a case study in the kind of rigorous, assumption-testing engineering that separates successful ML pipelines from silently broken ones.

The Context: Why Local Inference Wasn't Enough

The project had been running local inference on a machine with 8 RTX PRO 6000 Blackwell GPUs, using SGLang to serve Kimi K2.5 and generate responses for training data. This approach worked for the first two datasets (B1 and B2), but the pipeline was consuming enormous time and compute resources. The assistant had been tuning performance, patching SGLang's hidden state extraction, and debugging acceptance rates — all while the clock ticked on generating the remaining six datasets (B3 through B8).

The local approach had fundamental limitations. Each dataset required approximately 10 million output tokens, and with the model running at roughly 90 tokens per second in single-stream mode, generating a single dataset could take over 30 hours. With six datasets remaining, the timeline stretched to over a week of continuous inference — assuming no crashes, no GPU memory issues, and no other interruptions.

The alternative was OpenRouter: a commercial API that hosts dozens of model providers, including multiple instances of Kimi K2.5. By paying per-token, the assistant could potentially generate all six datasets in parallel, completing in hours instead of days. The budget was $100 — a constraint that would require careful provider selection and cost optimization.

The Research Phase: Mapping the Provider Landscape

Messages 3992 through 4007 represent an intensive research sprint. The assistant queried OpenRouter's model catalog, discovering that Kimi K2.5 was available from eleven providers at varying price points. A critical finding emerged: the model ships natively as INT4 (quantization-aware training), and most providers run these exact weights. However, two providers needed exclusion: Fireworks (running NVFP4, a lower-precision format) and BaseTen (running FP4, also below INT4, and currently down with status -5).

The assistant built a detailed provider comparison table, calculated budget scenarios, and discovered that 60 million output tokens at the cheapest rate of $2.20 per million tokens would cost $132 — exceeding the $100 budget. This forced a recalibration: either reduce the per-dataset token budget from 10M to 7M, or let the script run until funds were exhausted with resume support.

The budget analysis was further complicated by reasoning tokens. Kimi K2.5 is a "thinking" model that generates extensive internal reasoning before producing its final answer. These reasoning tokens are billed as output tokens. Analysis of the B2 dataset showed an average of 3,500 tokens per response, while B3 (Magicoder) averaged only 1,232 tokens. The variance in reasoning length across different prompt types made precise budget planning difficult.

The solution was to implement budget-aware execution: the script would track spending via OpenRouter's credits API and stop gracefully when funds ran low, with robust resume support so it could be restarted later.

The Critical Challenge: Reconstructing Token IDs from Text

The most technically challenging aspect of this pivot was the problem of reconstructing exact Kimi K2.5 token IDs from OpenRouter's text responses. This was not a trivial tokenization exercise.

The Kimi K2.5 model uses a special token structure for its reasoning process. When the model "thinks," it generates tokens wrapped in thinking and response markers. These are not just formatting conventions; they are actual token IDs in the model's vocabulary (163606 and 163607 respectively). The OpenRouter API returns the reasoning content in a separate reasoning field, but the exact token boundaries matter for training the EAGLE-3 drafter.

The user's instruction in message 4010 — "Make super sure you capture thinking and tool calls correctly and that output is the exact correct tokens" — set the stage for the investigation that followed. This was not a casual suggestion; it was a hard requirement for the pipeline to succeed.

The Tokenizer Investigation: Four Critical Discoveries

The assistant launched a systematic investigation spanning messages 4011 through 4016, running Python scripts on the remote server to probe the Kimi K2.5 tokenizer's behavior. The investigation yielded four critical findings:

Finding 1: The <|im_end|> Anomaly

The most surprising discovery was that the <|im_end|> token, which the assistant initially believed to be token ID 163533, actually decoded to the string 'chas' — not to <|im_end|>. When the literal string <|im_end|> was encoded, it produced token ID 163586 — a completely different token. This meant the <|im_end|> token could not be reconstructed from text; it had to be injected as a raw token ID.

The implications were profound. If you naively take the text <|im_end|> (as a string) and encode it, you get token ID 163586 — a completely different token. The tokenizer sees the literal characters <, |, i, m, _, e, n, d, |, > and encodes them through its normal BPE process. But the actual special token ID 163533 is what the model generates, and it decodes to 'chas'.

Finding 2: </think> Acts as a Clean BPE Boundary

In contrast to <|im_end|>, the </think> token (ID 163607) behaved as expected. decode([163607]) returned the string '</think>', and encode('</think>') returned [163607]. This token could safely be included as text in a string to be encoded.

More importantly, the assistant verified that </think> acts as a clean BPE boundary. The test script checked five different reasoning/content boundary cases with varying starting characters (newlines, backticks, spaces, punctuation). In every case, encoding the reasoning text separately, injecting the </think> token ID (163607), and encoding the content text separately produced the exact same token sequence as encoding the full concatenated string. The reason is that </think> is a special token in the vocabulary — it is assigned its own ID and the BPE algorithm treats it as an atomic unit that cannot be split or merged with adjacent text.

Finding 3: BPE Boundary Merges Exist

The assistant confirmed that BPE tokenizers can merge adjacent characters into single tokens. The test case 'x' + 'y' illustrated this perfectly: encoding 'x' separately gave one token, encoding 'y' separately gave one token, but encoding the combined string 'xy' gave a single merged token. This meant that encoding reasoning text and content text separately, then concatenating the token sequences, could theoretically produce different tokens than encoding the full concatenated string — if BPE could merge tokens across the boundary.

However, the empirical tests confirmed that </think> acts as an impenetrable boundary, making separate encoding safe for the specific case of reasoning/content reconstruction.

Finding 4: Tool Call Special Tokens Roundtrip Correctly

The fourth finding was that tool call special tokens — <|tool_calls_section_begin|> (163595), <|tool_call_begin|> (163597), <|tool_call_argument_begin|> (163598), <|tool_call_end|> (163599), and <|tool_calls_section_end|> (163596) — all roundtripped correctly when encoded from their text form. This was good news because it meant tool call sequences could be safely included as text and would produce the correct token IDs.

The assistant also confirmed that real SGLang output roundtripped perfectly: encode(decode(output_ids)) == output_ids for actual model generations from the B1_glaive dataset. This validated the fundamental approach of reconstructing token IDs by encoding the full decoded text.

The Correct Formula

With these findings, the assistant established the correct token reconstruction formula:

encode(reasoning_text) + [163607] + encode(content_text) + [163533]

This formula:

1. Encodes the reasoning text separately (from OpenRouter's reasoning field)
2. Injects the </think> token ID (163607) as a raw integer
3. Encodes the content text separately (from OpenRouter's content field)
4. Injects the <|im_end|> token ID (163533) as a raw integer The critical insight is that </think> and <|im_end|> must be injected as raw token IDs, not included as text strings, because <|im_end|> in particular does not survive a decode-encode roundtrip through text.

The Script Architecture: 2000 Concurrent Requests

The assistant wrote run_inference_openrouter.py with several key design decisions:

Provider routing: Use OpenRouter's provider.ignore field to exclude Fireworks and BaseTen, and provider.sort: "price" to automatically route to the cheapest available provider. This ensured cost efficiency without hardcoding provider selection.

Massive concurrency: 2,000 parallel requests. This was the key advantage over local inference — the local SGLang server could handle perhaps 8-16 concurrent requests before degrading, while OpenRouter's distributed infrastructure could absorb thousands of simultaneous calls.

Exponential backoff: On 429 (rate limit) and 5xx (server error) responses, the script would back off exponentially, preventing cascading failures while maximizing throughput.

Resume support: The script would append to a JSONL file and skip completed sample IDs on restart, allowing the generation to be interrupted and resumed without losing progress.

Spend tracking: Periodic checks against OpenRouter's credits API to stop before exceeding budget.

The Results: 33 Minutes, $86, Zero Issues

The OpenRouter script completed all six B-datasets (B3 through B8) in approximately 33 minutes at a cost of ~$86. Structural validation of 1,637 OpenRouter responses showed zero issues, and token counts matched billing within 0.04% — a remarkable level of accuracy that validated the token reconstruction approach.

The chunk summary captures the achievement: "All 8 B-datasets (B3-B8) were completed in ~33 minutes at ~$86 cost, with structural validation of 1637 responses showing 0 issues and token counts matching billing within 0.04%."

This was a dramatic improvement over the local inference pipeline, which would have taken over a week for the same volume of data. The pivot to OpenRouter had succeeded beyond expectations.

Scoping the Remaining Phases

With the B-datasets complete, the assistant turned to scoping the remaining phases of the pipeline. The next steps were:

1. Merge and shuffle: Combining all ~40K samples (138.4 million tokens) into a single shuffled dataset. A merge-and-shuffle script was written to handle this.
2. Hidden state extraction: The compute-intensive phase where the base model's hidden states are extracted for each token in the training data. This requires running the model on local GPUs, since OpenRouter doesn't provide access to internal model states. The assistant analyzed the impact of A1_deepswekimi's 2,800 ultra-long samples (44.9 million tokens, averaging 16K tokens per sample) on the extraction time. These long samples dominate the token budget — capping sequence length at 8192 and potentially dropping A1 entirely would reduce extraction from ~5.5 TB / 91 hours to ~3.5 TB / 72 hours. The old 924 GB of 10K hidden states from earlier training attempts were identified for deletion to free up disk space. The pipeline was now at the transition point between data generation and the compute-intensive hidden state extraction phase.

The Broader Significance

This chunk of the conversation exemplifies several principles of robust ML engineering:

Test your assumptions: The assistant's systematic investigation of tokenizer behavior — testing roundtrip fidelity, comparing encoding strategies, probing BPE boundaries — demonstrates the level of rigor required when building ML pipelines that depend on exact token-level alignment.

Verify at system boundaries: When moving from local GPU inference (where token IDs are directly available) to a cloud API (which returns text), the entire pipeline's correctness hinges on the fidelity of the tokenization roundtrip. The assistant invested significant effort in verifying this boundary before committing to production.

Budget-aware execution: The $100 budget constraint forced careful cost modeling and the implementation of spend tracking. The assistant calculated that 7M tokens per dataset would fit within budget, and built the script to gracefully stop when funds ran low.

Massive parallelism: The 2000-concurrent-request design was aggressive but effective. It leveraged OpenRouter's distributed infrastructure to complete data generation in minutes rather than days.

Human-AI collaboration: The user's intervention at message 4010 — "Make super sure you capture thinking and tool calls correctly" — was the critical prompt that triggered the tokenizer investigation. The assistant had the technical capability to investigate these issues, but the user provided the strategic intuition about where things might go wrong.

Conclusion

The pivot from local GPU inference to OpenRouter API for EAGLE-3 training data generation was a dramatic success. In 33 minutes and $86, the assistant completed what would have taken over a week on local hardware. But this success was built on a foundation of meticulous investigation — testing tokenizer behavior, verifying BPE boundaries, confirming tool call handling, and validating the reconstruction approach against real model outputs.

The token reconstruction problem, in particular, reveals the hidden complexity of working with large language model tokenizers. The discovery that <|im_end|> decodes to 'chas', the confirmation that </think> acts as a clean BPE boundary, and the validation that tool call special tokens survive encode-decode roundtrips — each of these findings was essential to producing correct training data.

As the pipeline moves into the merge-and-shuffle and hidden state extraction phases, the token reconstruction logic validated in this chunk will underpin every subsequent step. The EAGLE-3 drafter trained on this data will be built on the foundation of careful, assumption-testing engineering that this chunk exemplifies.## References

[1] "The Architecture of Knowledge: How an AI Assistant Built a Comprehensive Engineering Status Document in a Complex ML Deployment" — Article on the engineering status document (msg 3963)

[2] "The Green Light: Autonomy, Trust, and the Art of Knowing When to Proceed" — Article on autonomous decision-making (msg 3964)

[3] "The Pivot Point: State Checking in ML Pipeline" — Article on pipeline state verification (msg 3965)

[4] "The Diagnostic Pivot: How a Simple ps aux Command Revealed the State of an 8-GPU Inference Pipeline" — Article on diagnostic commands (msg 3966)

[5] "The Moment of Truth: When a Log Tail Revealed the Need for a Strategic Pivot" — Article on log analysis (msg 3967)

[6] "The Quiet Check: How a Simple wc -l Revealed the State of a Distributed Inference Pipeline" — Article on file counting (msg 3968)

[7] "The Pivot Point: Reading run_inference.py at a Critical Juncture in EAGLE-3 Data Generation" — Article on code review (msg 3969)

[8] "The Pivot Point: Measuring B2's Progress Before Abandoning Local Inference" — Article on progress measurement (msg 3970)

[9] "The 10.7M Token Threshold: A Pivotal Resource Management Decision in EAGLE-3 Data Generation" — Article on resource management (msg 3971)

[10] "The Calculus of Waste: A Pivot Decision in the EAGLE-3 Training Pipeline" — Article on pivot decision-making (msg 3972)

[11] "The Moment of Discovery: Catching a Subtle AsyncIO Bug Before Deployment" — Article on bug discovery (msg 3973)

[12] "The Bare Coroutine Trap: Averting a Silent Failure in an Async Inference Pipeline" — Article on async bugs (msg 3974)

[13] "The Art of Verification: Tracing Dataset State Before Deployment" — Article on dataset verification (msg 3975)

[14] "The Quiet Audit: How a Single Bash Command Revealed the State of an EAGLE-3 Training Pipeline" — Article on pipeline auditing (msg 3976)

[15] "Tracing the Restart Logic: A Moment of Deliberation in an Automated Pipeline" — Article on restart logic (msg 3977)

[16] "The Moment of Commitment: Killing an Inference Pipeline to Reclaim Efficiency" — Article on pipeline termination (msg 3978)

[17] "The Pivot Point: A Single SCP Command That Redirected an EAGLE-3 Training Pipeline" — Article on file transfer (msg 3979)

[18] "The Token Budget Pivot: Precision Resource Allocation in EAGLE-3 Training Data Generation" — Article on token budgeting (msg 3980)

[19] "The Ambiguity of Logs: A Moment of Uncertainty in a Distributed Inference Pipeline" — Article on log ambiguity (msg 3981)

[20] "The Token Budget Checkpoint: Verifying Resource Optimization in a Large-Scale ML Data Pipeline" — Article on budget verification (msg 3982)

[21] "Verification in Motion: Confirming the Token-Budget Inference Pipeline" — Article on pipeline confirmation (msg 3983)

[22] "The Art of Waiting: Monitoring a Long-Running Inference Pipeline" — Article on pipeline monitoring (msg 3984)

[23] "The 18-Hour Estimate That Lasted 33 Minutes: A Case Study in Assumption-Bound Planning" — Article on estimation accuracy (msg 3985)

[24] "The Status Checkpoint: How a Simple Todo Update Marks a Critical Pipeline Transition" — Article on status tracking (msg 3986)

[25] "The Checkpoint Message: Orchestrating EAGLE-3 Training Data Generation at Scale" — Article on pipeline orchestration (msg 3987)

[26] "The Pivot: From Local GPUs to OpenRouter — A Strategic Decision in EAGLE-3 Training Data Generation" — Article on the OpenRouter pivot (msg 3988)

[27] "The Pivot: Switching from Local GPU Inference to OpenRouter API for EAGLE-3 Training Data Generation" — Article on the API switch (msg 3989)

[28] "The Moment Before the Correction: A Single Bash Command That Exposed an API Key" — Article on security (msg 3990)

[29] "The Half-Second Security Lesson: When an AI Assistant Almost Leaked an API Key" — Article on security awareness (msg 3991)

[30] "The Pivot to OpenRouter: A Strategic Decision Point in EAGLE-3 Training Data Generation" — Article on strategic pivot (msg 3992)

[31] "The Pivot to OpenRouter: A Single Grep That Changed the Pipeline" — Article on grep-based discovery (msg 3993)

[32] "Parsing the OpenRouter Landscape: How One Bash Command Shaped an EAGLE-3 Data Pipeline" — Article on provider analysis (msg 3994)

[33] "The Pivot to OpenRouter: A Single API Call That Changed the Data Pipeline" — Article on API exploration (msg 3995)

[34] "The Pivot to OpenRouter: A Single WebFetch That Redirected an ML Pipeline" — Article on web-based research (msg 3996)

[35] "The Strategic Pivot: From Truncated HTML to Structured API" — Article on API research (msg 3997)

[36] "The Provider Analysis Pivot: How One Message Charted the Course from Local GPUs to OpenRouter for EAGLE-3 Data Generation" — Article on provider analysis (msg 3998)

[37] "Mapping the Provider Landscape: The OpenRouter Provider Analysis That Pivoted an EAGLE-3 Training Pipeline" — Article on provider mapping (msg 3999)

[38] "The $100 Question: How a Budget Constraint Forced a Pivot in EAGLE-3 Training Data Generation" — Article on budget analysis (msg 4000)

[39] "The Quantization Hunt: A Single Search Query That Reshaped a $100 Budget" — Article on quantization research (msg 4001)

[40] "The Quantization Detective: How One Message Uncovered the True Cost of OpenRouter Inference for EAGLE-3 Training" — Article on cost discovery (msg 4002)

[41] "The Pivot to OpenRouter: A Strategic Research Step in EAGLE-3 Data Generation" — Article on research phase (msg 4003)

[42] "The Pivot to OpenRouter: Researching Provider Routing for EAGLE-3 Training Data Generation" — Article on routing research (msg 4004)

[43] "The Last Research Step: How a Single API Query Marked the Pivot from Local GPUs to OpenRouter" — Article on final research (msg 4005)

[44] "The Pivot to OpenRouter: Designing a Cloud Inference Pipeline for EAGLE-3 Training Data" — Article on pipeline design (msg 4006)

[45] "The Pivot Point: From Research to Implementation in the OpenRouter Inference Pipeline" — Article on research-to-implementation transition (msg 4007)

[46] "The Verification Pivot: How a Single SSH Command Shaped the OpenRouter Inference Pipeline" — Article on format verification (msg 4008)

[47] "The Pivot: From Local GPUs to OpenRouter API for EAGLE-3 Training Data" — Article on script creation (msg 4009)

[48] "The Pivotal Instruction That Saved the EAGLE-3 Pipeline" — Article on the user's critical instruction (msg 4010)

[49] "The Precision Imperative: Reconstructing Exact Token IDs from OpenRouter's Text Responses for EAGLE-3 Training" — Article on token reconstruction planning (msg 4011)

[50] "The Tokenizer Roundtrip Problem: Reconstructing Exact Token IDs from OpenRouter API Responses" — Article on tokenizer diagnostics (msg 4012)

[51] "The Tokenization Investigation That Failed: Reconstructing Exact Token IDs from OpenRouter Text Responses" — Article on failed investigation (msg 4013)

[52] "The Tokenization Autopsy: Validating BPE Boundaries for EAGLE-3 Training Data via OpenRouter" — Article on BPE boundary validation (msg 4014)

[53] "The Token Reconstruction Problem: How One Assistant Unraveled the Mysteries of BPE Tokenization for EAGLE-3 Training Data" — Article on token reconstruction reasoning (msg 4015)

[54] "Reconstructing Exact Token IDs from OpenRouter Text: The Tokenization Detective Work Behind EAGLE-3 Training Data" — Article on confirmation of approach (msg 4016)

[55] "Bridging Two Worlds: Reconstructing Native Token Sequences from OpenRouter API Responses" — Article on tool call handling (msg 4017)