From Infrastructure to Intelligence: How an AI Assistant Systematically Discovered Agentic Training Data for Speculative Decoding

Introduction

In the sprawling lifecycle of a production machine learning deployment, the journey from bare-metal infrastructure to intelligent model behavior is rarely linear. It is a winding path through driver installations, build errors, configuration bugs, and data sourcing — each layer presenting its own unique challenges. This article examines a pivotal segment of one such journey: the moment when a team, having just resolved a critical speculative decoding bug and benchmarked their fix, pivoted sharply to address the fundamental bottleneck of training data quality and scale.

The chunk under analysis captures a subagent conversation spawned from a root session that had already accomplished a remarkable amount. The parent session began with a full ML environment setup on Ubuntu 24.04, installing NVIDIA drivers version 590.48.01 and CUDA Toolkit 13.1, configuring two RTX PRO 6000 Blackwell GPUs, and creating a Python virtual environment with PyTorch using uv. A major effort was spent resolving flash-attn installation issues — a saga that required installing a secondary CUDA 12.8 toolkit, reducing parallel compilation jobs from 128 to 20 to avoid memory exhaustion, and later rebuilding flash-attn against the correct PyTorch version (2.9.1) after vLLM downgraded it. The environment was ultimately stabilized with a fully compatible stack: PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1.

The session then shifted to application deployment. The machine was upgraded to 8 GPUs, and the GLM-5-NVFP4 model was deployed using a nightly build of SGLang. But a critical bug emerged: the EAGLE-3 hidden state concatenation was failing silently. The root cause was traced to a simple but devastating flag mismatch — the server was started with --speculative-algorithm EAGLE instead of EAGLE3. The is_eagle3() check is strict; only EAGLE3 triggers the target model to capture and concatenate intermediate layer hidden states from layers [2, 30, 58]. With EAGLE, the draft model received 7168-dim final-layer-only states instead of the expected 21504-dim concatenated states, causing the fc fusion layer to be silently bypassed and all trained weights to be useless. After restarting with the correct flag, hidden states correctly arrived as 21504-dim and the draft model's predictions began being accepted (accept_len ~2.1, up from 1.0).

Benchmarking the fix showed the best EAGLE-3 configuration achieving 82.3 tok/s (with CUDA graphs and 5 draft tokens) against a 90 tok/s non-speculative baseline — still approximately 9% slower. The accept_len of ~2.1 was insufficient to overcome speculation overhead. The EAGLE-3 paper's scaling curve suggested that more training data was the primary lever. This realization set the stage for the subagent's mission: find and curate agentic coding and tool-use datasets to scale up training data by 10×.

The Dataset Discovery Mission: A Multi-Round Research Campaign

The subagent was spawned with a precise mandate from the user ([msg 0]): search for agentic coding and tool-use datasets on HuggingFace suitable for training a speculative decoding draft model. The user specified four categories — agentic coding datasets, tool-use/function-calling datasets, multi-turn agent conversation datasets, and SWE-bench style datasets — provided five specific search queries, and asked for at least 10 candidate datasets with structured metadata including name, size, format, content description, and availability.

What followed was a masterclass in systematic information retrieval, unfolding across five messages in a carefully orchestrated multi-round campaign.

Round 1: Broad Exploration (Message 1)

The assistant's first response ([msg 1]) launched five parallel Exa web searches covering the full breadth of the user's request. The queries targeted general agentic coding datasets, tool-use and function-calling datasets with a 2024-2025 date filter, SWE-bench training data, multi-turn agent conversations, and a speculative decoding-specific catch-all query. This initial salvo returned a mix of results: the SWE-bench/SWE-smith dataset card, the allenai/Dolci-Instruct-SFT-Tool-Use page, a blog post about "The Rise of Agentic Data Generation," and a DeepFabric article. The assistant had begun mapping the territory.

As [3] notes in its analysis of this message, the assistant's decision to launch all five searches in parallel reflects a sophisticated understanding of the tool environment's capabilities and a deliberate strategy to minimize round trips. The queries were not copied verbatim from the user's suggestions — they were adapted. The user suggested "tool use function calling dataset huggingface 2024" and the assistant used "tool use function calling dataset huggingface 2024 2025," extending the date range. The user suggested "SWE-bench training dataset huggingface" and the assistant added "code agent" for specificity. These adaptations reveal an assistant that is not blindly following instructions but actively refining them based on its own judgment.

Round 2: Targeted Follow-Up on Known Datasets (Message 2)

The broad searches had identified several candidates, but the assistant recognized significant gaps. Well-known, high-quality datasets like Glaive Function Calling v2, Salesforce xLAM, NousResearch Hermes Function Calling, and R2E-Gym had not surfaced. The assistant's second round ([msg 2]) therefore pivoted from category-level exploration to named-entity discovery, launching four parallel searches for these specific datasets.

As [6] observes, this transition from broad to targeted search mirrors how a human researcher would approach the same task: start with generic queries to map the landscape, then drill into specific known resources. The assistant's queries were highly precise — "glaive-function-calling-v2 dataset huggingface tool calling training," "xlam function calling dataset salesforce huggingface 2024," "NousResearch hermes function calling dataset huggingface agent trajectories," and "R2E-Gym dataset huggingface coding agent reinforcement learning." Each query targeted a specific HuggingFace dataset ID, reflecting the assistant's pre-existing knowledge of the dataset landscape.

The results confirmed the availability of these datasets and provided initial metadata. The Glaive query returned a Microsoft blog post about fine-tuning small language models for function-calling. The Salesforce xLAM query returned the dataset page (gated, requiring acceptance of terms). The NousResearch Hermes query returned the dataset page with 295 likes and JSON format. The R2E-Gym query returned limited results, foreshadowing that this dataset would require triangulation from secondary sources.

Round 3: Deep Dives on Priority Candidates (Message 3)

With a growing list of candidates, the assistant shifted to gathering specific metadata on the most promising datasets. Message 3 ([msg 3]) launched two targeted searches: one for NVIDIA's Nemotron-Agentic-v1 dataset (querying for "size rows tool use multi-turn") and one for the nebius/SWE-agent-trajectories dataset (querying for "80k rows software engineering").

As [2] highlights, this message represents the moment when the assistant transitions from "what's out there?" to "what's best for this specific task?" The two datasets targeted in this round would later emerge as the top recommendations in the final report. The Nemotron-Agentic-v1 dataset, described as "designed to strengthen models' capabilities as interactive, tool-using agents," focuses on multi-turn conversations where language models decompose user goals, decide when to call tools, and reason over tool outputs. The SWE-agent-trajectories dataset contains approximately 80,000 rows of full SWE-agent trajectories where an autonomous coding agent navigates codebases, reads files, edits code, runs tests, and generates patches to solve GitHub issues.

The assistant's query formulation reveals its prioritization criteria: both queries emphasize multi-turn structure, tool-use patterns, and scale. The assistant is clearly operating with a mental model of what makes good training data for speculative decoding draft models — and it is using that model to guide its search.

Round 4: Precision Detail Gathering (Message 4)

The final round of data collection ([msg 4]) targeted two more datasets that had been identified but lacked sufficient detail: Salesforce's APIGen-MT-5k and AI2's Dolci-Instruct-SFT-Tool-Use. The assistant's query for APIGen-MT specifically asked about the "multi-turn dataset huggingface Salesforce function calling," while the Dolci query asked about "dataset size OLMo" — seeking to determine how large the tool-use subset was within the larger 2.15M-row Dolci-Instruct-SFT dataset.

As [4] notes, this message represents a critical inflection point: the shift from exploration to verification, from breadth to depth. The assistant's decision to invest an additional round in targeted searches reflects a commitment to thoroughness. The phrase "Now let me get details on a few more datasets I've identified" reveals metacognitive awareness — the assistant knows what it knows and what it doesn't know, and it is systematically filling gaps.

Round 5: Synthesis and Recommendations (Message 5)

The culmination of this multi-round effort was a comprehensive report ([msg 5]) cataloging twelve datasets with detailed metadata and strategic recommendations. The datasets were ordered from most to least relevant, with each entry including the HuggingFace identifier, approximate size, format, content description, availability/license, and a relevance assessment.

The top-tier datasets included:

1. nebius/SWE-agent-trajectories (~80K rows, Parquet) — Full coding agent trajectories with 2-817 turns per example, described as "the gold standard for this use case" with "extremely high" relevance.
2. SWE-bench/SWE-smith (~59K rows, Parquet) — Large-scale synthetic SWE-bench training data with code patches and test cases, used to train SWE-agent-LM models.
3. nvidia/Nemotron-Agentic-v1 (multiple subsets, tens of thousands of rows) — Purpose-built multi-turn tool-use data with LLM-as-judge quality filtering, commercial use permitted.
4. Salesforce/xlam-function-calling-60k (60K rows, JSON) — Verified function-calling examples with format checking, execution verification, and semantic verification.
5. Salesforce/APIGen-MT-5k (5K multi-turn dialogues, ShareGPT-like JSON) — High-quality multi-turn agent data with 99% human-evaluated success rate.
6. glaiveai/glaive-function-calling-v2 (~113K rows, JSON) — The largest function-calling dataset, covering hundreds of function schemas, with 243+ models trained on it. The assistant's final recommendation was not to pick a single best dataset but to combine complementary sources: "A mix of high-volume function-calling data (glaive, xlam) with deep agentic trajectory data (nebius, Nemotron) would give the best coverage." This insight — that draft model training requires both breadth (volume and diversity of function-calling patterns) and depth (full trajectory structure including system prompts, tool calls, and environment responses) — transforms the report from a simple list into a strategic plan.

The Broader Significance: What This Chunk Reveals

This chunk is remarkable for what it reveals about the nature of AI-assisted development in complex engineering contexts. Several themes emerge.

The Iterative Nature of ML Infrastructure Work

The parent session's journey — from bare-metal Ubuntu setup through NVIDIA driver installation, CUDA configuration, flash-attn build troubleshooting, model deployment, bug fixing, benchmarking, and finally data sourcing — illustrates the deeply iterative nature of production ML work. Each layer reveals new challenges. The flash-attn build issues required installing a secondary CUDA toolkit and reducing parallel compilation jobs. The EAGLE-3 bug required tracing a silent failure through hidden state dimensions. The benchmarking revealed that even with the bug fixed, the draft model's acceptance rate was insufficient — leading to the data scaling effort.

This is not a linear pipeline. It is a spiral of increasing understanding, where each loop reveals new constraints and opportunities. The chunk captures one complete turn of that spiral: infrastructure → deployment → debugging → benchmarking → data.

The Power of Systematic Information Retrieval

The subagent's dataset discovery campaign demonstrates a sophisticated research methodology that mirrors the best practices of human researchers. The progression from broad exploration to targeted follow-up to deep verification to synthesis is a classic pattern in systematic literature reviews and market research. The assistant's ability to execute this autonomously, across multiple rounds, with parallel tool calls, represents a significant capability.

As [5] observes, the assistant's research methodology reveals several implicit prioritization frameworks: full trajectory data is valued over isolated examples, multi-turn interactions are valued over single-turn, verified data is valued over synthetic, domain-specific data is valued over general, and provenance matters (datasets used in successful models receive higher credibility).

The Self-Referential Nature of the Task

There is a beautiful self-referential quality to this sub-session. The assistant is using web search — a tool-use behavior — to find datasets about tool-use. The datasets it discovers contain examples of AI agents using tools, calling functions, and writing code — the very behaviors the assistant is demonstrating in its search process. The assistant is, in effect, searching for training data that would teach a model to behave more like itself.

This self-referential loop is not merely philosophical. It has practical implications. If the assistant can successfully find and curate datasets that capture agentic behavior, and if those datasets are used to train a draft model that accelerates inference for agentic coding tasks, then the assistant's own search behavior becomes a data point in the training distribution. The assistant is bootstrapping its own future capabilities.

The Transition from Infrastructure to Intelligence

Perhaps the most significant arc in this chunk is the transition from infrastructure concerns to intelligence concerns. The early part of the parent session is dominated by hardware and build issues: which CUDA version, how many parallel compilation jobs, which PyTorch version is compatible with which flash-attn version. These are questions of infrastructure — getting the bits to compile and run.

The later part of the session, including this subagent's work, is dominated by data questions: which datasets contain the right conversational patterns, how large should the training corpus be, what mix of trajectory data and function-calling data is optimal. These are questions of intelligence — teaching the model to behave in desired ways.

The pivot point is the EAGLE-3 bug fix and benchmarking. Once the infrastructure is stable and the model is running, the bottleneck shifts from "can it run?" to "is it fast enough?" And the answer to "is it fast enough?" leads to "do we have enough data?" This is a pattern that recurs across ML deployments: infrastructure enables intelligence, but intelligence is ultimately limited by data.

Conclusion

The chunk examined in this article captures a pivotal moment in a larger ML engineering effort. It begins with a team that has just resolved a critical speculative decoding bug and benchmarked their fix, only to discover that their draft model's acceptance rate is insufficient. It follows a subagent through a multi-round, systematic dataset discovery campaign that identifies twelve candidate datasets spanning agentic trajectories, function-calling examples, multi-turn dialogues, and code patches. And it culminates in a strategic recommendation to combine high-volume function-calling data with deep agentic trajectory data for optimal draft model training.

The work in this chunk is a microcosm of the broader ML lifecycle: infrastructure enables deployment, deployment reveals bugs, bugs lead to fixes, fixes enable benchmarking, benchmarking reveals data limitations, and data limitations drive data sourcing. Each step builds on the previous one, and each step reveals new challenges that were invisible at the previous level of abstraction. The assistant's systematic approach to dataset discovery — broad search, targeted follow-up, deep verification, synthesis — provides a template for how AI systems can autonomously navigate complex research tasks in support of larger engineering goals.

The datasets discovered in this effort would go on to inform a 10× scaling of training data, with an inference pipeline launched on the baseline SGLang server at approximately 830 tok/s throughput, processing 83K prompts through Kimi-K2.5 to regenerate responses matching the target model's token distribution. That pipeline, expected to run for 24-55 hours, represents the next turn of the spiral — transforming discovered data into actual training examples that will, hopefully, push the draft model's acceptance rate high enough to overcome speculation overhead and achieve the speedups that speculative decoding promises.## References

[1] "The Dataset Hunter: Deconstructing a Subagent's Mission to Find Agentic Training Data" — Analysis of the user message that launched the dataset discovery mission.

[2] "The Meta-Search: How an AI Assistant Used Tool-Use to Find Tool-Use Datasets" — Analysis of the assistant's targeted follow-up searches on Nemotron-Agentic-v1 and SWE-agent-trajectories.

[3] "The Art of the Parallel Search: How an AI Assistant Launches a Dataset Discovery Mission" — Analysis of the initial five parallel web searches in message 1.

[4] "The Targeted Deep-Dive: How an AI Assistant Refines Its Dataset Search with Precision Queries" — Analysis of the precision detail-gathering searches in message 4.

[5] "Mapping the Agentic Data Landscape: A Deep Dive into Dataset Curation for Speculative Decoding Draft Models" — Analysis of the final comprehensive report in message 5.

[6] "The Methodical Searcher: How an AI Assistant Systematically Discovered Agentic Datasets Through Iterative Web Search" — Analysis of the targeted named-entity searches in message 2.