The Orchestrator's Setup: Creating the Infrastructure for Parallel Dataset Preparation

Introduction

In the complex workflow of training a speculative decoding draft model for a large language model, no step is too small to matter. Message [msg 3667] in this opencode session captures a seemingly trivial moment: the assistant creates two directories on a remote server before launching a fleet of parallel dataset preparation agents. Yet this message, brief as it is, reveals the careful orchestration, the architectural thinking, and the methodical execution that defines successful large-scale ML pipeline construction. It is a moment of transition — the bridge between planning and execution, between writing code and running it at scale.

The message reads in full:

[assistant] Now launch all 10 dataset preps in parallel — these don't need GPU (just CPU + network for HF downloads). For A1/A2 (Kimi-native) they need the tokenizer from /shared/kimi-k2.5-int4 but that's just CPU: [bash] ssh root@10.1.230.174 'mkdir -p /data/eagle3/synth_100k/prepared && mkdir -p /data/eagle3/synth_100k/logs'

On its surface, this is a single mkdir -p command executed over SSH. But to understand why this message exists and what it accomplishes, we must examine the broader context of the EAGLE-3 training pipeline, the state of the system at this moment, and the reasoning that led the assistant to issue this specific command at this specific juncture.

The Context: A Pipeline at a Crossroads

The session leading up to this message had been a rollercoaster of debugging and discovery. The team had been working on deploying an EAGLE-3 speculative decoding drafter for the Kimi-K2.5 language model, running on a server with 8 NVIDIA RTX PRO 6000 Blackwell GPUs. A critical bug had just been resolved: the server had been started with --speculative-algorithm EAGLE instead of EAGLE3, causing the draft model to receive only 7168-dimensional final-layer hidden states instead of the expected 21504-dimensional concatenated states from layers [2, 30, 58]. This single flag mismatch had rendered all trained weights useless, silently bypassing the fusion layer and producing zero acceptance.

After fixing the flag, benchmarking showed the best EAGLE-3 configuration achieving 82.3 tok/s — still 9% slower than the 90 tok/s non-speculative baseline. The acceptance length of ~2.1 tokens was insufficient to overcome speculation overhead. The EAGLE-3 paper's scaling curves suggested the primary lever was more training data. The team had been working with only 10,000 samples; the obvious next step was to scale up by an order of magnitude.

The user had approved a plan to create a 100,000-sample training dataset, combining ten different sources spanning agentic coding trajectories, function-calling conversations, code instructions, reasoning traces, and general chat. The disk had been resized to 11TB to accommodate the ~9.2TB of hidden states this would generate. The assistant had written a comprehensive plan document (train_plan_v4.md), created a unified dataset preparation script (prep_all.py), and transferred it to the remote container.

At the moment of message [msg 3667], the assistant had just killed the EAGLE-3 speculative server (to free GPU memory for the inference pipeline) and was preparing to launch the data preparation phase. The system state was:

  1. The prep_all.py script existed on the container at /root/eagle3-train/datasets/prep_all.py
  2. The old SGLang server had been killed
  3. A baseline non-speculative server needed to be started for response generation
  4. Ten dataset preparation agents needed to run in parallel
  5. The output directories for these agents did not yet exist

Why This Message Was Written: The Reasoning

The assistant's decision to create directories before launching agents reflects a fundamental principle of distributed systems engineering: prepare infrastructure before spawning workers. If the agents were launched without the output directories existing, they would fail immediately — either crashing with a FileNotFoundError or silently writing to locations that would later be impossible to find. By creating the directories first, the assistant ensures that when the agents begin writing their output, the destination is guaranteed to exist.

The choice of directory names is itself revealing:

How Decisions Were Made

Several design decisions are embedded in this single command:

Decision 1: Parallel, not sequential. The assistant could have processed each dataset one at a time, which would be simpler to implement and debug. Instead, it chose to launch all 10 in parallel. This decision is driven by the fact that dataset preparation is CPU-bound and network-bound — downloading from HuggingFace and tokenizing text. These operations are embarrassingly parallel, and running them sequentially would waste hours. The assistant correctly identifies that the bottleneck is I/O, not compute, so parallelism maximizes throughput.

Decision 2: Remote execution, not local. The assistant runs the command via ssh root@10.1.230.174 rather than executing locally. This reflects the architecture of the system: the heavy ML work happens on a remote server (the container with 8 GPUs), while the assistant's host machine is likely a development workstation. The dataset preparation needs access to the tokenizer and the output storage on the remote machine, so running there makes sense.

Decision 3: Infrastructure as a separate step. Rather than having the prep_all.py script create its own output directory (which would be a reasonable design), the assistant creates the directories externally. This separation of concerns means the prep script can assume its output directory exists, simplifying error handling. It also means that if the directory creation fails (e.g., due to permission issues), the error is caught before any agents start, rather than after one has already processed 5,000 samples.

Decision 4: Using mkdir -p. The -p flag (parent) is a defensive choice. It creates the directory if it doesn't exist, and does nothing if it already does. This makes the command idempotent — running it multiple times is safe. This is important in a session where commands might be retried or the system might be restarted.

Assumptions Made

The message and its surrounding context reveal several assumptions:

Assumption 1: The directories don't already exist. The assistant uses mkdir -p rather than mkdir, which would fail if the directories existed. This is a safe assumption — if they do exist, the command is harmless. But the assistant is implicitly assuming a clean state, which is reasonable for a new pipeline phase.

Assumption 2: The remote path /data/eagle3/synth_100k/ is valid. This assumes the disk resize to 11TB has been completed and the /data mount point exists. If the disk hadn't been resized, this command would succeed (creating directories on an existing filesystem) but later stages would fail when they ran out of space.

Assumption 3: The tokenizer path /shared/kimi-k2.5-int4 is accessible. The assistant mentions this path in the message text, confirming it knows the tokenizer location. This assumes the shared filesystem is mounted and readable from the container.

Assumption 4: The prep agents will be launched immediately. The message says "Now launch all 10 dataset preps in parallel" but the actual command only creates directories. The assistant is treating directory creation as the first step of the launch sequence, assuming the actual agent spawning will follow in the next message. This is a reasonable decomposition of a multi-step operation.

Assumption 5: CPU and network bandwidth are sufficient for 10 parallel downloads. The assistant explicitly notes these don't need GPU, but doesn't consider whether 10 simultaneous HuggingFace downloads might saturate the network or overwhelm the CPU. For most setups, this is fine — HuggingFace datasets are typically served via CDN and downloads are bandwidth-limited rather than CPU-limited.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. The EAGLE-3 architecture: Knowledge that EAGLE-3 is a speculative decoding framework where a lightweight draft model predicts the target model's next tokens, and that training requires hidden states from specific intermediate layers of the target model.
  2. The pipeline structure: Understanding that the training pipeline has distinct phases — dataset preparation, response generation (inference), hidden state extraction, vocabulary mapping, and training — and that this message belongs to the first phase.
  3. The remote execution model: Awareness that the assistant operates from a development machine and executes commands on a remote container at 10.1.230.174 via SSH.
  4. The dataset taxonomy: Understanding that datasets A1/A2 (DeepSWE-Agent-Kimi-K2-Trajectories and KimiK2.5-2000x) are "Kimi-native" — they already contain outputs from the Kimi-K2.5 model and can be tokenized directly — while datasets B1-B8 are "prompt-only" and need inference to generate responses matching the target model's distribution.
  5. The resource constraints: Knowledge that the container has 8 GPUs, 11TB of disk, and a shared filesystem at /shared/kimi-k2.5-int4 containing the model weights and tokenizer.
  6. The recent history: Understanding that the EAGLE-3 flag bug was just fixed, that benchmarking showed inadequate acceptance rates, and that the team decided to scale training data from 10K to 100K samples.

Output Knowledge Created

This message creates tangible and intangible outputs:

Tangible: Two directories are created on the remote filesystem:

The Thinking Process Visible in the Message

Even in this brief message, we can see the assistant's reasoning process at work:

Resource-aware scheduling: The assistant explicitly notes which resources each agent needs. The Kimi-native datasets (A1/A2) need the tokenizer (CPU + disk I/O). The prompt-only datasets (B1-B8) need HuggingFace downloads (CPU + network). None need GPU. This analysis justifies the parallel execution — if any agent needed GPU, parallelism would be constrained by the 8 available GPUs.

Risk mitigation: By creating directories before launching agents, the assistant eliminates a common failure mode. If an agent tries to write to a non-existent directory, it either crashes or creates the directory itself (potentially with wrong permissions or in the wrong location). Pre-creating directories gives the assistant control over the filesystem layout.

Sequencing awareness: The message is positioned at a specific point in the execution sequence: after the script is written and transferred, after the old server is killed, but before the new server is started and before the agents are launched. The assistant is working through a mental checklist, ensuring each prerequisite is satisfied before moving to the next step.

Abstraction and delegation: The assistant has abstracted the dataset preparation logic into a single prep_all.py script that takes a --dataset parameter. This allows launching 10 agents with the same codebase, each handling a different dataset. The directory creation is the final infrastructure step before this delegation happens.

Broader Significance

This message, for all its brevity, exemplifies a pattern that appears throughout successful ML engineering projects: the importance of infrastructure before execution. The temptation when scaling up a pipeline is to jump straight to the exciting part — launching the parallel agents, running the inference, training the model. But experienced engineers know that the boring parts — creating directories, setting up logging, validating inputs — are what prevent disasters.

The message also demonstrates the value of explicit reasoning in agent communications. The assistant doesn't just run the command; it explains why it's safe to run these in parallel ("these don't need GPU") and what each type of agent needs ("they need the tokenizer... but that's just CPU"). This transparency builds trust and makes the session auditable — if something goes wrong later, anyone reading the log can trace back to this decision point.

In the broader arc of the EAGLE-3 training project, this message marks the transition from planning to execution. The plan document has been written, the scripts have been created, the infrastructure has been prepared. The next messages will see the 10 agents launched, the baseline server started, and the long process of generating 100,000 training samples begin. This small mkdir -p command is the first domino in a chain that will span days of computation, terabytes of data, and ultimately determine whether the EAGLE-3 drafter can achieve the acceptance rates needed for practical speedup.