The 924-GB Question: A Pivotal Reconnaissance Step in the EAGLE-3 Training Pipeline

Introduction

In the middle of a complex multi-phase machine learning pipeline—training a speculative decoding draft model for a 1-trillion-parameter MoE language model—a single, seemingly mundane bash command can represent a critical decision point. The message at <msg id=4095> in this opencode session is precisely such a moment. The assistant executes a remote SSH command to check the disk usage of two directories on a container running at 10.1.230.174:

[assistant] [bash] ssh root@10.1.230.174 'du -sh /data/eagle3/synth_10k_sglang/hidden_states/ 2>/dev/null; echo "---"; du -sh /data/eagle3/synth_10k_sglang/vocab_mapping/ 2>/dev/null'
924G	/data/eagle3/synth_10k_sglang/hidden_states/
---
840K	/data/eagle3/synth_10k_sglang/vocab_mapping/

On its surface, this is a simple disk usage inquiry. But within the broader context of the session—a multi-day effort to train a high-quality EAGLE-3 draft model for the Kimi-K2.5 architecture—this message is the hinge point between two major phases of work. It confirms a crucial prerequisite for deleting nearly a terabyte of old data, freeing space for a new extraction that will consume over 3.5 TB. This article unpacks the reasoning, assumptions, and strategic significance of this single message.

Context: The EAGLE-3 Training Pipeline

To understand why this message matters, one must understand the pipeline it serves. The assistant and user have been engaged in a weeks-long effort to deploy and optimize the Kimi-K2.5 model—a 1-trillion-parameter Mixture-of-Experts language model running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. A key performance optimization is speculative decoding: using a small "draft" model to predict multiple tokens per forward pass of the large model, amortizing the cost of PCIe-bound inter-GPU communication.

The EAGLE-3 draft model is a single transformer layer (2.6 billion parameters) that predicts future hidden states given the large model's own hidden representations. Training it requires three phases:

  1. Response generation: Using the large model (or OpenRouter API) to generate responses to prompts from diverse datasets.
  2. Hidden state extraction: Running the large model on those responses to capture the hidden states at specific layers (2, 30, and 58 of 61), producing ~42 KB of data per token.
  3. Training: Training the draft model on the extracted hidden states to predict future states. A previous 10K-sample run had produced a draft model with an acceptance length of only ~2.1 tokens—insufficient to beat the 90 tok/s baseline. The team scaled up to 100K samples, generating responses across 10 datasets (A1, A2, B1-B8) totaling 138.4 million tokens. At the time of <msg id=4095>, response generation was complete, and the pipeline was at the threshold between Phase 2 (merge and shuffle) and Phase 3 (hidden state extraction).

Why This Message Was Written

The immediate trigger for this command is the assistant's todo list from <msg id=4093>, which includes the item: "Delete old 10K hidden states (924 GB)" with status "pending." Before executing a destructive rm -rf command on nearly a terabyte of data, the assistant must verify:

  1. That the old hidden states actually exist and are the expected size. The 924 GB figure had been mentioned in planning documents, but disk usage can change due to incomplete cleanup, concurrent processes, or filesystem anomalies. Blindly deleting without verification risks either failing to free space (if the data was already removed) or accidentally targeting the wrong directory.
  2. That the vocab mapping directory is small and should be preserved. The vocab mapping (t2d.pt and d2t.pt files, only 840 KB) maps between the full 163,840-token vocabulary and the reduced 32,000-token draft vocabulary. Rebuilding this mapping would require re-running a separate script (03_build_vocab_mapping.py) and re-validating token coverage. Confirming its small size reinforces the decision to keep it.
  3. That there is sufficient disk space for the upcoming extraction. The new 100K hidden state extraction is estimated to require ~3,517 GB (87.8 million tokens × 42 KB/token). The /data volume has 11 TB total with 10% used (~1,021 GB used per <msg id=4094>). Freeing 924 GB would bring used space down to ~97 GB, leaving over 10 TB for the new extraction. The message thus serves as a pre-flight check before a destructive operation. It is the assistant's way of saying: "I have the information I need to proceed safely."

The Reasoning Process Visible in This Message

Although the message contains only a bash command and its output, the reasoning behind it is visible through the sequence of actions across adjacent messages. In <msg id=4093>, the assistant lays out a prioritized todo list with "Delete old 10K hidden states (924 GB)" as a high-priority pending item. In <msg id=4094>, the assistant checks the state of the merged directories and overall disk usage, establishing the baseline: the /data volume has 1,021 GB used out of 12 TB. Then in <msg id=4095>, the assistant drills down specifically into the old hidden states and vocab mapping.

This reveals a two-level verification strategy: first check the high-level state (what datasets exist, how much disk is free), then check the specific targets of the pending operation. The assistant is building a mental model of the filesystem state before committing to changes. The use of du -sh (disk usage in human-readable summary) rather than ls -la (which would show file counts but not total size) is deliberate: the assistant needs to know the aggregate size to confirm the 924 GB figure, not the individual file listing.

The command also demonstrates defensive scripting: 2>/dev/null suppresses error messages if the directories don't exist, preventing noise from polluting the output. The echo "---" separator makes the two results visually distinct. These are small but meaningful choices that reflect an understanding of how tool output will be parsed and interpreted.

Assumptions Made

Several assumptions underpin this message:

  1. The hidden states directory is safe to delete. The assistant assumes that the old 10K extraction is no longer needed. This is reasonable: the 10K draft model has been benchmarked and found insufficient (accept_len ~2.1), and the new 100K model will supersede it. However, there is an implicit assumption that no other process depends on these files—for example, a running SGLang server that might reference the old hidden states.
  2. The vocab mapping is reusable. The assistant assumes that the vocabulary mapping built for the 10K dataset is identical to what the 100K dataset needs. This is likely true because both datasets use the same tokenizer and the same reduced draft vocabulary (32,000 tokens covering 98.3% of token occurrences). But it is an assumption worth noting: if the 100K dataset introduces new tokens not seen in the 10K set, the mapping might need updating.
  3. The SSH connection will succeed and the remote machine is in a consistent state. The assistant assumes the container at 10.1.230.174 is accessible and that the filesystem state hasn't changed since the previous check moments ago in <msg id=4094>.
  4. Disk space is the binding constraint. The assistant implicitly prioritizes freeing disk space over other concerns (like backup safety or reproducibility). The 924 GB of hidden states could theoretically be compressed or archived, but the decision is to delete them outright. This reflects the practical reality of working with 11 TB of storage and needing 3.5 TB for the next phase.

Mistakes and Incorrect Assumptions

While the message itself is correct and the output is accurate, there is a subtle risk in the assumptions:

The vocab mapping might not be fully reusable. The 10K dataset was drawn from a different distribution than the 100K dataset. If the 100K data contains tokens that were rare or absent in the 10K set, the t2d (token-to-draft) mapping might map them to incorrect draft indices, and the d2t (draft-to-token) mapping might not cover all tokens the model might generate. The assistant's plan to reuse the vocab mapping is noted in <msg id=4091>: "Reuse existing vocab mapping from /data/eagle3/synth_10k_sglang/vocab_mapping/." This is a pragmatic shortcut, but it introduces a potential quality degradation if the token distributions differ significantly.

No verification of file integrity. The du -sh command reports the aggregate size but does not verify that the hidden state files are intact, uncorrupted, or in the expected format. If the old extraction had partial or corrupted files, the 924 GB figure would still be reported correctly, but the assistant wouldn't know. This is a minor concern since the files are being deleted anyway, but it reflects a general pattern of trusting filesystem metadata over content validation.

Input Knowledge Required

To fully understand this message, one needs:

  1. The pipeline architecture: Knowledge that EAGLE-3 training requires hidden state extraction, that the old 10K run produced 924 GB of extracted states, and that the new 100K run will replace them.
  2. The filesystem layout: Understanding that /data/eagle3/synth_10k_sglang/ contains the old extraction, with hidden_states/ being the large artifact and vocab_mapping/ being the small reusable artifact.
  3. The storage constraints: Knowing that /data has 11 TB total with ~1 TB currently used, and that the new extraction needs ~3.5 TB.
  4. The todo list priority: Understanding that deleting the old hidden states is a prerequisite for the next phase, and that the assistant is methodically working through its checklist.
  5. The SSH infrastructure: Knowing that root@10.1.230.174 is the LXC container where inference and training run, and that the assistant has established SSH access.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Confirmed size of old hidden states: 924 GB, matching the planning estimate. This confirms that the full 10K extraction completed successfully and hasn't been partially deleted.
  2. Confirmed size of vocab mapping: 840 KB, trivially small. This reinforces the decision to keep it and reuse it for the 100K training.
  3. Validation of the deletion plan: The assistant now has first-hand confirmation that the deletion target exists and is the expected size, reducing the risk of the rm -rf command.
  4. A record for the conversation log: The output serves as documentation for future reference. If someone later asks "what happened to the old 10K hidden states?" or "how much space did they take?", this message provides the answer.
  5. Confidence to proceed: Perhaps most importantly, this message gives the assistant the confidence to execute the next step—deleting 924 GB of data—without hesitation or error.

The Broader Significance

This message exemplifies a pattern that recurs throughout the opencode session: verification before action. In a pipeline with multi-hour or multi-day phases, where a single mistake (deleting the wrong directory, running the wrong script, misconfiguring a server) can waste days of work, the assistant consistently pauses to check state before proceeding. The du -sh command is cheap (sub-second execution), low-risk (read-only), and provides high-value confirmation.

The 924 GB figure also serves as a tangible reminder of the scale of this project. Each token of training data requires 42 KB of hidden states. The 10K dataset produced 924 GB of extracted states for ~23 million tokens. The 100K dataset will produce over 3.5 TB. These are not abstract numbers—they represent real storage constraints, real extraction time (~72 hours), and real engineering effort to manage.

In the next message (<msg id=4096>), the assistant reads the merge_and_shuffle.py script, preparing to deploy it. The deletion of the old hidden states (confirmed in <msg id=4095>) will free the space needed for the new extraction. The pipeline continues to march forward, one verified step at a time.

Conclusion

The message at <msg id=4095> is a masterclass in operational discipline. A junior engineer might have blindly deleted the old hidden states based on the planning document. A senior engineer—or in this case, a well-trained AI assistant—pauses to verify, to confirm, and to document. The 924 GB of hidden states represent weeks of prior work: the original 10K extraction that took 19 hours, the training runs that proved the 10K model insufficient, and the hard-won lessons about SGLang configuration, CUDA graph capture, and weight key mismatches. Deleting them is not just a disk management operation—it is closing the book on one chapter of the project and opening another.

The message also reveals the assistant's mental model: it thinks in terms of phases, prerequisites, and verification gates. It doesn't just execute commands; it manages state, checks assumptions, and builds confidence incrementally. In a complex ML pipeline where a single wrong rm -rf could destroy days of work, this discipline is not just good practice—it is essential.