Probing the Black Box: How One Assistant Validated Tool Call Correctness Across OpenRouter's API Boundary

Introduction

In the complex pipeline of training a speculative decoding model like EAGLE-3 for the Kimi K2.5 architecture, data integrity is paramount. Every stage of the pipeline — prompt generation, response inference, hidden state extraction, and model training — depends on the assumption that the data flowing through it is semantically and structurally correct. When the assistant behind an opencode session pivoted from local GPU inference to the OpenRouter API for generating training data, a new class of questions emerged: does the API faithfully reproduce the model's output, including its special token structure? This article examines a single message (index 4042) in which the assistant, prompted by a user's two-word request — "Look at tool call correctness too" — designed and executed a sophisticated probe of OpenRouter's handling of Kimi K2.5 tool calls, only to have the command time out after 120 seconds.

The Scene: A Pipeline at a Critical Juncture

By message 4042, the session had been running for hours across multiple segments. The team was building an EAGLE-3 speculative decoding drafter for the Kimi K2.5 model, a large language model with a 7168-dimensional hidden state and a specialized architecture. The training pipeline required generating tens of thousands of model responses across multiple datasets (B1 through B8), extracting hidden states from those responses, and training a lightweight draft model to predict tokens in parallel.

The original approach used a local SGLang server running on a machine with 8 RTX PRO 6000 Blackwell GPUs. However, after extensive tuning, the local inference pipeline proved too slow for the scale required. The assistant pivoted to using OpenRouter's API, which provided access to Kimi K2.5 hosted by various providers (SiliconFlow, among others) at a cost of roughly $2.25 per million output tokens. A new script, run_inference_openrouter.py, was written with 2000-concurrent request handling, provider routing, and robust resume support.

The critical challenge in this pivot was reconstructing exact token IDs from OpenRouter's text responses. The local SGLang server returned raw token IDs (output_ids), which could be used directly for hidden state extraction. OpenRouter, being a standard chat API, returned text — the model's reasoning in a reasoning field and its content in a content field. The assistant had to reconstruct the original token sequence by concatenating these fields with special tokens like <|im_end|> and <|tool_calls_section_begin|>, then re-encoding the result through the tokenizer.

Previous validation (see [msg 4028]) had shown that this reconstruction achieved a 0.5% mismatch rate on the B1 dataset and 6.5% on B3, with all mismatches attributable to BPE tokenization differences — the model's autoregressive token selection sometimes produced different BPE splits than the tokenizer's greedy left-to-right encoding of the same text. The assistant had concluded this was acceptable for EAGLE-3 training because "the hidden state extraction will use whatever tokenization we produce, and the draft model learns from those exact tokens."

The User's Intervention

Then came the user's message at index 4041, a simple two-word prompt: "Look at tool call correctness too." This was a critical intervention. The assistant's validation had focused on the general reconstruction pipeline — concatenating reasoning, content, and special tokens — but had not specifically probed how OpenRouter handled tool calls. The B1_glaive dataset, which contained tool-calling examples, had been tested, but the reconstruction function in the validation script passed None for the tool_calls parameter, effectively assuming that tool calls would either not appear or would be handled correctly.

The user recognized this blind spot. Tool calls in Kimi K2.5 are represented using special tokens in a structured format:

<|tool_calls_section_begin|><|tool_call_begin|>functions.function_name:0<|tool_call_argument_begin|>{"args"}<|tool_call_end|><|tool_calls_section_end|>

If OpenRouter parsed these tool calls into a structured tool_calls array in the API response, the assistant's reconstruction code would need to convert them back into this exact token format. Any discrepancy — a missing token, an incorrectly formatted argument, a wrong index — would produce a different token sequence, potentially corrupting the training data.

The Assistant's Response: A Three-Scenario Probe

The assistant's response at index 4042 demonstrates a thorough, methodical approach to validating an assumption that could have silently broken the entire pipeline. The thinking process is visible in the opening paragraph:

"Good call. Let me think about tool call correctness more carefully. The issue is: when OpenRouter receives a response from Kimi K2.5 that contains tool calls, what exactly does it return? There are two scenarios: 1. We don't send tools in the request → model may still generate tool call special tokens in content (they appear as raw text) 2. OpenRouter parses tool calls → they appear in message.tool_calls array, and the text between </think> and the tool call section goes into content"

This framing reveals the assistant's mental model of the problem. The key uncertainty is whether OpenRouter acts as a transparent pipe (scenario 1) or as an interpreter that restructures the model's output (scenario 2). The assistant correctly identifies that the B1_glaive data showed the "native format" — raw tool call tokens embedded in the content text — but this was from local inference. OpenRouter might behave differently.

The assistant then designs three test scenarios, each probing a different aspect of OpenRouter's behavior:

Test 1: With the tools Parameter

The first test sends a request with a proper tools parameter containing a function definition for get_weather. This is the standard OpenAI-compatible tool-calling format. If OpenRouter follows the OpenAI specification, it should parse the model's tool call output into the message.tool_calls array. The test checks what appears in content, reasoning, and tool_calls, and whether the finish reason is tool_calls.

This test answers the question: "When OpenRouter knows about available tools, does it restructure the output?"

Test 2: Without tools, With a Tool-Calling System Prompt

The second test omits the tools parameter but includes a system prompt that describes the available functions in natural language and instructs the model to output tool calls in the native format. This simulates the B1_glaive dataset, where tool capabilities are described in the system message rather than declared via the API's tools parameter.

This test answers: "When OpenRouter doesn't know about tools from the API parameters, does it still parse tool calls from the model's output?"

Test 3: A Real B1_glaive Prompt

The third test loads an actual prompt from the B1_glaive dataset — the same data the pipeline is processing — and sends it through OpenRouter without any special handling. This is the most realistic test, directly probing the exact scenario the pipeline will encounter.

The test checks whether the returned content contains raw tool call tokens (like <|tool_calls_section_begin|>) or whether OpenRouter has stripped them into the tool_calls field.

The Code: A Well-Structured Probe

The Python script the assistant writes is notable for its structure. It uses asyncio and aiohttp for HTTP requests, matching the concurrency model of the production run_inference_openrouter.py script. Each test is clearly separated with print banners. The script prints detailed information about each response: content, reasoning, tool calls, finish reason, provider, and message keys.

The script also handles errors gracefully, printing error information if the API returns an error rather than crashing. The timeout is set to 120 seconds, which turns out to be insufficient — the command ultimately times out.

One subtle detail: the script reads the API key from /tmp/or-key.txt using Path.read_text(), which is a clean, modern Python idiom. The key is redacted in the message output, appearing as sk-or-v1-2...4a82.

Assumptions Embedded in the Probe

The assistant makes several assumptions in designing this test:

1. That OpenRouter's behavior is consistent across providers. The test uses the provider parameter with ignore: ["fireworks", "baseten"] and sort: "price", which means the actual provider handling the request may vary. If different providers handle tool calls differently, the test results might not generalize.
2. That a single test per scenario is sufficient. Each scenario is tested with exactly one request. Tool-calling behavior could be non-deterministic — the model might choose to call a tool in one request and answer directly in another, especially for a simple query like "What's the weather in San Francisco?"
3. That the B1_glaive prompt loaded from the first line of prompts.jsonl is representative. The script reads the first prompt from the file, which may or may not be one that actually triggers tool calls. The test checks for raw tool call tokens in the content, but if the model doesn't call a tool, the test tells us nothing about how OpenRouter handles tool calls.
4. That the API key and endpoint are stable. The test assumes the key file exists and the OpenRouter API is available, which is reasonable given the previous successful test at [msg 4037].

Input Knowledge Required

To understand this message, the reader needs knowledge of:

• The OpenAI chat completions API format, including the tools, tool_calls, and finish_reason fields.
• The Kimi K2.5 model's special token format for tool calls, including <|tool_calls_section_begin|>, <|tool_call_begin|>, <|tool_call_argument_begin|>, and <|tool_call_end|>.
• The EAGLE-3 training pipeline and why exact token ID reconstruction matters — because the draft model learns to predict the next token given hidden states, and any mismatch between training data tokenization and extraction tokenization would produce inconsistent training signals.
• The B1_glaive dataset structure, which contains tool-calling examples where tools are described in system prompts rather than declared via API parameters.
• OpenRouter's provider routing and pricing model, including the ability to ignore specific providers and sort by price.
• The concept of BPE tokenization and how autoregressive decoding can produce different token splits than greedy encoding — the root cause of the previously observed mismatches.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

1. A test methodology for validating API behavior. The three-scenario approach provides a template for testing any API that mediates between a model's raw output and a structured response format.
2. Documentation of OpenRouter's tool-calling behavior for Kimi K2.5. The test results (had they completed) would document whether OpenRouter parses tool calls into structured fields or leaves them as raw text, which is critical information for anyone building pipelines on top of this API.
3. A reusable test script. The Python script written in this message can be adapted for testing other models, other API endpoints, or other tool-calling scenarios.
4. Risk identification. Even without completing, the message identifies a concrete risk: if OpenRouter strips tool call tokens from the content and puts them in tool_calls, the reconstruction code must handle this perfectly, or the training data will be corrupted.

The Timeout: An Unresolved Question

The command times out after 120 seconds. This is notable because the previous single-test request at [msg 4037] completed successfully in what appears to be a reasonable time. The timeout could be due to:

• The tool-calling request taking longer than a simple text generation. Tool-calling scenarios often require the model to reason about whether to call a tool, which function to call, and what arguments to provide, potentially increasing generation time.
• Provider routing delays. The sort: "price" parameter may route to a slower provider.
• Network issues. The SSH connection to the remote machine (10.1.230.174) could have experienced latency or packet loss.
• The cumulative time of three sequential requests. Each with a 120-second timeout, the total could exceed the SSH command's patience. The timeout means the question remains unanswered at the end of this message. The assistant will need to retry with longer timeouts or individual tests in subsequent messages.

The Thinking Process: Visible Reasoning

The assistant's reasoning in this message is particularly visible in the opening paragraph, where it works through the two possible scenarios for how OpenRouter handles tool calls. This internal monologue reveals:

1. Recognition of the blind spot. The assistant immediately acknowledges that the user's suggestion is "good" and that more careful thought is needed.
2. Identification of the key variable. The central question is whether OpenRouter strips tool call tokens from content and puts them in a structured field, or leaves them as raw text.
3. Connection to prior knowledge. The assistant recalls that "B1_glaive data showed the native format" — the raw special token format — and contrasts this with what OpenRouter might do.
4. Implication analysis. The assistant recognizes that if OpenRouter parses tool calls, "my reconstruction needs to be perfect" — a high-stakes requirement.
5. Test design. The assistant designs three scenarios that systematically probe the boundary between OpenRouter's behavior and the model's native output format.

Conclusion

Message 4042 represents a critical moment of quality assurance in a complex ML pipeline. Prompted by a two-word user intervention, the assistant designed a sophisticated probe of OpenRouter's API behavior, testing three scenarios that systematically covered the space of possible tool-calling behaviors. The test script was well-structured, the reasoning was clear, and the risk was properly identified — even though the command ultimately timed out without returning results.

This message exemplifies the kind of meticulous validation that separates robust ML pipelines from fragile ones. The assistant could have assumed that OpenRouter's text responses would faithfully reproduce the model's output, including tool call tokens. Instead, it probed the boundary, tested the assumption, and identified a potential failure mode that could have silently corrupted thousands of training samples. The timeout is a reminder that even well-designed probes can encounter operational friction, but the thinking and methodology remain valuable regardless of the immediate outcome.

The article also highlights the importance of user oversight in AI-assisted development. A two-word prompt — "Look at tool call correctness too" — triggered a chain of reasoning and testing that addressed a genuine blind spot. In the complex dance between human and machine, sometimes the most valuable contributions are the simplest questions.