The Missing Tool Calls: A Methodical Investigation into Data Quality at Scale

In any large-scale machine learning pipeline, the most dangerous failures are the silent ones—the bugs that don't crash, the data that looks right but isn't, the assumptions that quietly erode the quality of the final product. Message [msg 7666] captures a moment of precisely such vigilance. It is a short, focused diagnostic query, but it reveals the careful, methodical thinking required to validate a complex data generation pipeline operating across seven GPUs, hundreds of thousands of prompts, and a remote server infrastructure.

The message itself is a single bash command executed over SSH on a remote machine (root@213.173.111.134, port 36472). It runs an inline Python script that performs a targeted investigation: are the tool-call prompts in the dataset actually producing completions? The answer it returns is concise and troubling: "Completed tool prompts: []". None of the five tool-call prompts checked have been processed yet. This discovery sits at the intersection of data quality assurance, pipeline architecture, and the practical challenges of generating training data at scale.

The Context That Demanded This Investigation

To understand why this message exists, we need to trace back through the conversation. The assistant and user are in the middle of a massive data generation run. They are using Qwen3.6-27B—a large language model with thinking/reasoning capabilities—to generate completions for 904,286 prompts. These completions will be used to train a DFlash speculative decoding drafter, a model that learns to predict the hidden states of the target model to accelerate inference.

The generation pipeline is sophisticated: seven independent SGLang server instances, each pinned to one GPU on a B200 NVL node, running with speculative decoding (EAGLE algorithm, 3 draft steps, 4 draft tokens). The servers handle ~10 requests per second, generating ~25,000 tokens per second aggregate, with an estimated 27 hours remaining. Completions are saved to S3 in batches of 500.

At [msg 7662], the user asks a critical question: "Status so far? Are we doing prompts with tool calls too?" This question reveals the user's awareness that the dataset contains multi-turn conversations and tool-use examples—and a concern about whether those are being handled correctly. The assistant's response at [msg 7664] provides the first layer of analysis: 12.5% of the 913,786 prompts have tool/function definitions in their system messages, and 8.4% are multi-turn conversations. But the assistant also checks a small sample of completed outputs and finds zero tool calls in the generated text—a worrying signal that warrants deeper investigation.

The Investigation Unfolds

Message [msg 7665] continues the probe. The assistant examines a sample tool-call prompt to understand its structure. The sample (index 800000) reveals a ShareGPT-format conversation with roles ['system', 'human', 'gpt', 'tool', 'gpt']—a multi-turn interaction where the model previously made a function call and received a tool response. The system message contains a JSON function definition for search_recipe. The assistant also checks the generation script's sharegpt_to_openai converter and confirms that system messages are passed through to the OpenAI-format messages sent to the model.

This brings us to [msg 7666], the subject message. The assistant now executes a more targeted query. Instead of sampling random completions (which happened to come from early prompt indices and showed no tool calls), the script directly identifies the indices of tool-call prompts and checks whether those specific indices appear in the .done_indices file—the authoritative record of which prompts have been completed.

The Python script is elegantly minimal. It opens the prompts file, iterates until it finds five prompts with system messages containing "function" in their value, records their indices, then cross-references those indices against the set of completed indices. The output is unambiguous: the tool-call prompts are at indices 800000, 800001, 800003, 800005, and 800006, and none of them have been completed.

What This Discovery Means

At first glance, this seems like a non-issue: the generation run started from index 0 and had completed roughly 27,690 out of 904,286 remaining prompts at that point (see [msg 7663]). The tool-call prompts begin at index 800,000, so they simply haven't been reached yet. The earlier finding of zero tool calls in completed outputs was an artifact of sampling only from early batches.

But the deeper significance lies in what this investigation reveals about the pipeline's architecture and the team's approach to quality assurance. The .done_indices file is a simple but effective tracking mechanism—a flat file listing every prompt index that has been successfully completed. This allows the generation script to resume after interruptions (as happened when the max output tokens were bumped from 4096 to 8192 at [msg 7658]) without reprocessing completed prompts. It also enables exactly this kind of targeted audit: "Has prompt X been processed yet?"

The investigation also surfaces an implicit assumption: that the model, when given a tool-call prompt, will actually produce a valid tool call in its output. The earlier check at [msg 7664] found zero tool calls in 3 sampled completions, but those completions came from non-tool prompts. The real question—whether the model correctly generates function calls when given tool definitions—remains unanswered until the generation reaches index 800,000. This is a tension between wanting early quality signals and the practical reality that the most interesting prompts (multi-turn, tool-using, complex reasoning) are often the hardest and may be batched later.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the investigation. There is a clear chain of inference:

  1. The user asks about tool calls (msg 7662).
  2. The assistant first establishes the scale: 12.5% of prompts have tool definitions (msg 7664).
  3. It then samples completed outputs and finds zero tool calls (msg 7664). This is a potential red flag.
  4. It examines the raw prompt format to understand how tool calls are represented (msg 7665).
  5. It verifies the generation script passes system messages through correctly (msg 7665).
  6. Finally, in msg 7666, it checks whether tool-call prompts have been reached yet—ruling out the simplest explanation for the zero tool calls in output. This is classic diagnostic reasoning: eliminate the trivial explanations before investigating deeper bugs. The zero tool calls in output could mean (a) the model isn't generating tool calls, (b) the tool-call prompts haven't been processed yet, or (c) the output format doesn't match the detection patterns. Message [msg 7666] eliminates (b), confirming that the tool-call prompts simply haven't been reached.

Input Knowledge Required

To fully understand this message, one needs to know several things about the pipeline:

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Tool-call prompts exist at specific indices: [800000, 800001, 800003, 800005, 800006]. This confirms the dataset contains tool-use examples and they are concentrated in the later portion of the dataset.
  2. None have been completed yet: The generation hasn't reached index 800,000. This rules out the hypothesis that the model is failing to generate tool calls.
  3. The pipeline is processing sequentially: The .done_indices file can be used to check the status of any specific prompt, confirming that the generation order is roughly sequential from index 0 upward.
  4. No immediate action required: Since the tool-call prompts simply haven't been reached, there's no bug to fix. The investigation can continue once the generation reaches those indices.

Assumptions and Potential Pitfalls

The investigation makes several assumptions that are worth examining:

The Broader Significance

This message, for all its brevity, exemplifies a critical practice in large-scale ML data pipelines: continuous data quality monitoring. When generating 900,000+ completions over 27+ hours, it's tempting to set the pipeline running and check back only when it finishes. But silent data quality issues—wrong prompt formatting, incorrect tokenization, missing system messages, model misbehavior on specific prompt types—can render the entire dataset useless.

The assistant's investigation is structured as a series of increasingly targeted queries, each narrowing the hypothesis space. This is the scientific method applied to data engineering: observe an anomaly (zero tool calls in output), formulate hypotheses (tool prompts not reached vs. model not generating tool calls vs. detection heuristic wrong), and test each hypothesis with the minimum necessary experiment.

The discovery that tool-call prompts simply haven't been reached yet is, in some sense, an anticlimax. No bug was found. No code change is needed. But this is precisely the value of the investigation: it rules out a class of potential failures, allowing the team to focus their attention elsewhere. In a complex pipeline with dozens of components, knowing what is not broken is almost as valuable as knowing what is.

Conclusion

Message [msg 7666] is a small but revealing snapshot of the work required to maintain data quality in a large-scale ML generation pipeline. It demonstrates methodical reasoning, targeted diagnostics, and the importance of tracking infrastructure (like .done_indices) that enables precise auditing. The investigation it belongs to—spanning messages [msg 7664], [msg 7665], and [msg 7666]—shows how a simple user question ("Are we doing prompts with tool calls too?") can trigger a chain of analysis that validates the entire pipeline's handling of a critical data subset. In the end, the answer is reassuring: the tool-call prompts are in the dataset, they're formatted correctly, the generation script passes them through properly, and they simply haven't been reached yet. The generation continues, and the question of whether the model produces valid tool calls will be answered in due time.