Pivoting Under Pressure: Debugging OpenRouter Tool Call Handling for EAGLE-3 Training Data

Introduction

In the middle of a massive data generation pipeline for training an EAGLE-3 speculative decoding drafter, a single user prompt — "Look at tool call correctness too" — triggered a rapid debugging cycle that reveals much about the craft of building reliable ML infrastructure. The message at [msg 4043] captures a moment of pivot: a complex multi-scenario test had just timed out after 120 seconds, and the assistant needed to regroup, simplify, and get a definitive answer about how OpenRouter handles tool calls from the Kimi K2.5 model. This message is a case study in debugging under uncertainty, the importance of isolating variables, and the hidden complexity of reconstructing token-perfect training data from API responses.

The Broader Context: Building an EAGLE-3 Drafter

To understand why tool call correctness matters, we must step back. The assistant was in the midst of generating training data for an EAGLE-3 speculative decoding drafter — a model that learns to predict the hidden states of a larger "base" model (Kimi K2.5) to accelerate inference. The pipeline had been running for hours across multiple segments (<segments 24-29>), progressing from local GPU inference on a machine with 8 RTX PRO 6000 Blackwell GPUs to a pivot to OpenRouter API for the remaining data generation.

The datasets being processed — B1_glaive, B2_opencodeinstruct, B3_magicoder, and others through B8 — contain diverse prompts, many of which involve tool calling. The B1_glaive dataset in particular includes system messages that describe functions the model can call, with prompts like "Can you book a flight for me from New York to London?" designed to elicit tool-use behavior. For EAGLE-3 training, every token matters: the draft model learns from the exact token sequences the base model produces. If tool call tokens are lost, transformed, or misrepresented during the OpenRouter round-trip, the training data would be corrupted, and the drafter would learn incorrect patterns.

The assistant had already invested significant effort in reconstructing Kimi K2.5 token IDs from OpenRouter's text responses. Earlier tests ([msg 4037]) had confirmed that OpenRouter strips the </think> tag from reasoning content, requiring the reconstruction code to insert it manually: reasoning + </think> + content + <|im_end|>. But tool calls presented an even thornier problem: OpenRouter might either preserve the raw tool call special tokens in the content field, or parse them into a structured message.tool_calls array. The reconstruction code needed to handle whichever format OpenRouter returned.

The Trigger: A User's Sharp Eye

The user's prompt at [msg 4041] — "Look at tool call correctness too" — came immediately after the assistant had launched the OpenRouter inference pipeline and checked its initial progress. The assistant had already verified basic functionality: the API key worked, credits were available ($100), the reasoning field was correctly populated, and the cost was reasonable (~$2.25/M output tokens). But the user correctly identified a blind spot: the assistant had tested with a simple "What is 2+2?" prompt that wouldn't trigger tool calls. The real datasets contained tool-calling prompts, and the reconstruction logic needed to be validated against those.

The First Attempt: Ambitious but Overloaded

In response ([msg 4042]), the assistant designed a comprehensive test script called test_or_toolcall.py that would test three scenarios in sequence:

1. With tools parameter: Sending a proper OpenAI-format tools array to see if OpenRouter parses tool calls into message.tool_calls
2. Without tools parameter, with tool-calling system prompt: Simulating the B1_glaive format where tools are described in the system message
3. Bare prompt from B1_glaive: Loading an actual prompt from the dataset and sending it exactly as the inference script would This was a well-structured test, but it had a critical flaw: it set max_tokens: 2000 and timeout: 120 seconds for each request, and ran three requests sequentially. The script timed out after 120 seconds — the bash tool's timeout limit — meaning at least one of the requests took too long. The assistant couldn't see any results.

The Pivot: Simplifying to the Essential Question

Message [msg 4043] is the assistant's response to that failure. The reasoning is explicit in the opening line: "Timeout — maybe one of the providers is slow. Let me try with shorter timeouts and one test at a time." This is a textbook debugging maneuver: when a complex test fails, strip it down to the simplest possible test that answers the most critical question.

The assistant made several specific changes:

1. Reduced from three tests to one: Instead of testing all three scenarios, the new script tests only scenario 3 — the actual B1_glaive prompt without any tools parameter. This is the most operationally relevant test because it matches exactly what the inference script sends.
2. Reduced max_tokens from 2000 to 1000: This limits the response length, reducing both cost and latency. For a tool-calling prompt, the model doesn't need many tokens to decide to call a function.
3. Reduced timeout from 120s to 90s: A more aggressive timeout, acknowledging that some providers are slow and the test should fail fast rather than hang.
4. Added prompt structure inspection: Before sending the request, the script prints the roles and first 200 characters of each message in the prompt. This provides visibility into what's actually being sent — a defensive check against format mismatches.
5. Focused output diagnostics: The script checks specifically for the presence of <|tool_calls_section_begin|> and <|tool_call_begin|> in the content field, which is the binary question: does OpenRouter leave raw tool call tokens in the content, or does it strip them?

The Result and Its Implications

The test succeeded. The output shows:

• Provider: AtlasCloud (a different provider than the earlier SiliconFlow test — demonstrating OpenRouter's provider routing)
• Finish reason: "stop" (the model chose not to call a tool, or finished normally)
• Reasoning: 683 characters beginning with "The user is asking me to book a flight from New York to London. However, looking at the ava..." — the model is reasoning about whether it can actually book a flight The output is truncated in the message, but the critical information is present: the request worked, the provider was identified, and the reasoning content was captured. The script's checks for raw tool call tokens would have printed their results below the visible output.

Knowledge Required to Understand This Message

To fully grasp what's happening in [msg 4043], several pieces of context are necessary:

1. OpenRouter's API behavior: OpenRouter is a unified API gateway that routes requests to multiple LLM providers. It supports OpenAI-compatible chat completions with optional tools parameter. When tools is provided, it may parse tool calls into structured format; when omitted, tool call tokens may appear as raw text in content.
2. Kimi K2.5's special tokens: The model uses custom tokens like <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|>, <|tool_call_end|>, and <|tool_calls_section_end|> to delimit tool calls in its output. These are not standard OpenAI tokens and may be handled differently by different API providers.
3. The B1_glaive dataset format: These prompts use a system message to describe available functions, rather than the OpenAI tools parameter. This means the model generates tool calls using its native special token format, which may or may not survive the OpenRouter round-trip.
4. EAGLE-3 training requirements: The draft model needs exact token-level alignment with the base model. Any transformation of the output (stripping tokens, reformatting tool calls) must be reversed during reconstruction.
5. The bash tool timeout: The assistant operates within a tool execution environment where bash commands have a 120-second timeout. This constraint shaped the debugging strategy — the assistant couldn't run indefinitely long tests.

Assumptions Made

The message reveals several assumptions:

1. That one provider was slow, not that the model was genuinely generating a long response: The assistant assumed the timeout was caused by provider latency, not by the model generating a very long reasoning chain. This was a reasonable assumption given that max_tokens was set to 2000, but it could have been wrong.
2. That reducing to one test would complete within 90 seconds: The assistant set a 90-second timeout, assuming the simplified test would be fast. This turned out to be correct, but it was a gamble.
3. That the B1_glaive prompt would elicit a tool call: The assistant loaded the first prompt from the dataset, assuming it would trigger tool-calling behavior. The result shows the model chose not to call a tool (finish reason "stop"), which means the test didn't fully validate the tool call reconstruction path.
4. That the test script would run on the container without issues: The script was copied via scp and executed on the remote machine, which had the necessary Python packages (aiohttp) and file access to the dataset.

Mistakes and Incorrect Assumptions

The most significant issue is that the simplified test didn't actually trigger a tool call. The model responded with reasoning about why it couldn't book a flight, rather than generating tool call tokens. This means the test didn't answer the critical question: does OpenRouter preserve <|tool_calls_section_begin|> tokens in the content field when no tools parameter is sent? The script would have printed "Has '<|tool_calls_section_begin|>' in content: False" — but that could mean either (a) OpenRouter stripped them, or (b) the model simply didn't generate them.

The assistant could have improved the test by:

• Searching the dataset for a prompt that did generate tool calls in the local SGLang run
• Crafting a more specific prompt designed to force tool use
• Testing with the tools parameter to see the structured format as a baseline Additionally, the assistant assumed that the timeout in the previous test was caused by provider slowness. An alternative explanation is that the model was generating a very long reasoning chain for one of the test prompts. The "What's the weather in San Francisco?" prompt with a tools parameter might have triggered extended reasoning about weather APIs. The assistant didn't investigate the timeout cause beyond assuming provider latency.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in several ways:

1. Diagnostic chain: "Timeout → maybe one of the providers is slow → let me try with shorter timeouts and one test at a time." This shows a clear cause-and-effect reasoning process.
2. Prioritization: The assistant chose to test only the most operationally relevant scenario (the actual B1_glaive prompt) rather than all three scenarios. This reflects an understanding of what matters most for the immediate task.
3. Defensive instrumentation: The script prints the prompt structure before sending the request, showing the assistant wanted visibility into what was actually being sent. This is a learned behavior from debugging API integration issues.
4. Binary outcome checks: The script checks for specific token strings in the content, reducing a complex question to a simple boolean. This is efficient debugging — don't parse the full structure until you know the format.
5. Progressive simplification: The assistant went from a 3-scenario test with 120s timeout to a 1-scenario test with 90s timeout, then (in subsequent messages not shown) presumably to even simpler tests. This is the classic debugging pattern of narrowing the search space.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

1. OpenRouter provider routing works: The request was routed to AtlasCloud (not SiliconFlow as before), demonstrating that OpenRouter's sort: "price" and provider ignore list are functioning.
2. The B1_glaive prompt format is compatible: The prompt loaded from the dataset was accepted by OpenRouter without errors, confirming the message format is correct.
3. The API key and authentication work: The request succeeded with a 200 status, confirming the key is valid and has sufficient credits.
4. The reconstruction code's assumptions about reasoning are correct: The reasoning field was populated as expected, confirming the include_reasoning: True parameter works.
5. A timeout baseline was established: 90 seconds was sufficient for a single request with 1000 max_tokens, providing a reference for future timeout settings.

Conclusion

Message [msg 4043] is a small but revealing moment in a much larger engineering effort. It shows the assistant responding to a failure (timeout) by applying a fundamental debugging principle: simplify, isolate, and test one thing at a time. The message captures the tension between comprehensive testing (three scenarios) and operational pragmatism (one focused test that answers the most critical question). It also reveals the hidden complexity of working with API-mediated model access, where the exact format of tool calls, reasoning content, and special tokens can vary between providers and must be carefully validated.

The broader lesson is that building reliable ML training pipelines requires not just writing code that works in the happy path, but systematically probing the edge cases — and knowing when to pivot from an ambitious but failing test to a simpler one that can actually complete. The assistant's debugging instinct — "timeout, simplify, retry" — is exactly the kind of adaptive reasoning that separates robust infrastructure from fragile scripts.