The 2.9-Minute Extraction: Monitoring a Long-Running ML Job at Scale

Introduction

In the sprawling narrative of an opencode coding session dedicated to deploying and optimizing trillion-parameter language models on 8× NVIDIA Blackwell GPUs, most messages are dense with code, configuration, and debugging. But some of the most revealing moments are the quiet ones—the progress checks, the status polls, the brief pauses where an agent waits for a long-running job to complete and then peeks at the logs to see what happened. Message [msg 2818] is one such moment. It is a deceptively simple progress check that, when examined in context, reveals the operational rhythm of large-scale ML engineering: the patience required to load a 547 GB model, the relief of watching extraction fly at 2,910 tokens per second, and the quiet satisfaction of a pipeline that finally works after multiple failures.

The Message Itself

The assistant writes:

Model just finished loading (22.5 minutes). Now extraction is starting. Let me check back in a few minutes:

>

``bash sleep 120 && ssh root@10.1.230.174 'tail -15 /root/eagle3-train/extract_1k.log 2>/dev/null' Batch 968-972: 0.8s, Total: 972 samples, 487878 tokens, 5.8 samples/s, 2910 tok/s, ~31.5 GB on disk Batch 972-976: 0.5s, Total: 976 samples, 489464 tokens, 5.8 samples/s, 2911 tok/s, ~16.4 GB on disk Batch 976-980: 0.8s, Total: 980 samples, 492061 tokens, 5.8 samples/s, 2911 tok/s, ~8.3 GB on disk Batch 980-984: 0.9s, Total: 984 samples, 494729 tokens, 5.8 samples/s, 2911 tok/s, ~34.8 GB on disk Batch 984-988: 1.3s, Total: 988 samples, 498572 tokens, 5.8 samples/s, 2911 tok/s, ~71.5 GB... ``

This is the entire message. It consists of an assertion ("Model just finished loading"), a plan ("Let me check back in a few minutes"), a shell command (sleep 120 && ssh ... tail), and the output of that command showing the tail end of a hidden state extraction job that is already 97% complete. The brevity is the point: the assistant is not debugging, not writing code, not making decisions. It is waiting, and it is reporting the status of what it finds when it checks.

The Road to This Moment

To understand why this message matters, we must trace the path that led to it. The assistant has been building an EAGLE-3 speculative decoding training pipeline for the Kimi-K2.5 INT4 model—a 1-trillion-parameter Mixture-of-Experts architecture deployed across 8 GPUs. The pipeline has four steps: prepare a dataset, extract hidden states from the verifier model, build a vocabulary mapping, and train a lightweight draft model. Message [msg 2818] captures the climax of step 2.

The extraction step is the pipeline's bottleneck. It requires loading the full 547 GB verifier model across all 8 GPUs with tensor parallelism, then running every training sample through the model to capture intermediate hidden states from specific layers (layers 2, 30, 58, and 60, as revealed in the subsequent message's data_config.json). These hidden states are the "ground truth" that the EAGLE-3 draft model will learn to predict.

The first attempt at this extraction failed catastrophically. In [msg 2813], the assistant discovered that using --batch-size 2000—attempting to prefill all 1,000 samples (503,000 tokens) in a single batch—caused an out-of-memory (OOM) error on GPU 0. The model itself already consumed ~91 GB per GPU, leaving almost no headroom for activations. The assistant diagnosed the problem, killed the GPU processes ([msg 2814]), cleaned up the output directory ([msg 2815]), and relaunched with --batch-size 4 and --gpu-memory-utilization 0.85 ([msg 2816]). This was the correct fix: with 4 samples per batch (up to 8,192 tokens), the prefill memory footprint stays within the available budget.

What the Output Reveals

The log output in message [msg 2818] tells a remarkable story. Each line shows a batch of 4 samples being processed in under 1.3 seconds, with the cumulative throughput holding steady at 2,910–2,911 tokens per second. At this rate, the full 502,983 tokens across 1,000 samples complete in approximately 173 seconds—under 3 minutes. The "~71.5 GB" at the end of the last line is the cumulative disk usage, confirming that the hidden states are being written to storage in real time.

But there is a subtle discrepancy between the assistant's narrative and the actual timing. The assistant says "Model just finished loading (22.5 minutes). Now extraction is starting." Yet the log output shows extraction already at batch 968 out of 1,000—nearly complete. The model loading finished during the 25-minute sleep from [msg 2817], and the extraction script began processing immediately. By the time the assistant waited an additional 2 minutes and checked the log, the extraction was already 97% done. The assistant's mental model—"extraction is starting"—lagged behind reality by about 2.5 minutes. This is a natural consequence of asynchronous job management: the assistant dispatched a background process, waited, and checked, but the process ran faster than anticipated.

The Significance of 2,910 Tokens Per Second

The extraction throughput of 2,910 tok/s on 8 GPUs is noteworthy. The verifier model is the full Kimi-K2.5 INT4 with 1 trillion parameters, quantized to 4-bit weights and deployed with tensor parallelism across 8 RTX PRO 6000 Blackwell GPUs. Achieving nearly 3,000 tokens per second of prefill throughput on such a massive model requires efficient GPU utilization, well-optimized attention kernels, and minimal communication overhead between GPUs. The fact that the assistant's pipeline can sustain this rate for the full 503,000-token dataset validates both the hardware configuration and the software stack (vLLM with compressed-tensors MoE support).

This throughput number also serves as a sanity check for the training data quality. With 1,000 samples processed in under 3 minutes (after the 22.5-minute model load), the assistant can iterate rapidly on the dataset. If the training results are poor, the bottleneck is not the extraction step—it is the data itself or the training hyperparameters.

The Assistant's Operational Pattern

Message [msg 2818] exemplifies a pattern that recurs throughout the session: dispatch, wait, check, iterate. The assistant does not sit idle during long operations. It dispatches a background job with nohup, sets a timer with sleep, and returns to check progress. This is a form of manual asynchronous programming—the assistant simulates non-blocking execution by issuing a command, sleeping for an estimated duration, and then inspecting the results.

The pattern has risks. If the job finishes faster than expected (as it does here), the assistant wastes time sleeping. If the job fails silently, the assistant may not discover the failure until the next check. The assistant mitigates these risks by checking logs rather than exit codes, and by using tail to see the most recent output. The log-based monitoring approach is robust: even if the process is still running, the assistant can assess progress from the cumulative counters.

Assumptions and Decisions

Several assumptions underpin this message:

  1. The model load completes within 25 minutes. The assistant set a 25-minute sleep in [msg 2817] based on the observed loading rate of ~27 seconds per shard for 64 shards. This was accurate—the load finished in 22.5 minutes.
  2. The extraction script produces meaningful log output. The assistant relies on the script printing per-batch progress lines. If the script had been silent, the tail command would have returned nothing useful.
  3. The batch size of 4 is safe. After the OOM failure with batch_size=2000, the assistant assumed batch_size=4 would fit. This was correct, as the extraction completed without errors.
  4. The SSH connection is reliable. The entire pipeline runs on a remote machine (root@10.1.230.174). The assistant assumes the connection will not drop during the 25-minute wait.
  5. The extraction is CPU-bound or I/O-bound, not GPU-bound. At 2,910 tok/s, the GPUs are likely underutilized during extraction. The assistant does not attempt to increase batch size further to improve throughput.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message creates:

The Thinking Process

The assistant's reasoning in this message is implicit but discernible:

  1. Temporal reasoning: "Model just finished loading (22.5 minutes)" — the assistant tracks elapsed time and reports it, building a mental model of how long each pipeline stage takes.
  2. Progress estimation: "Now extraction is starting" — the assistant believes extraction has just begun, and plans to check in 2 minutes. This estimate is based on the assumption that extraction will take several minutes at least.
  3. Surprise management: The log output reveals extraction is nearly done. The assistant does not express surprise—it simply reports what it sees. The subsequent message ([msg 2819]) confirms completion with precise statistics.
  4. Bottleneck awareness: The assistant knows that model loading (22.5 minutes) dominates extraction (2.9 minutes). This informs future decisions: if the pipeline needs to be re-run, the assistant will try to avoid reloading the model.

Conclusion

Message [msg 2818] is a small but revealing moment in a much larger engineering effort. It captures the transition from waiting to validation—the moment when a long-running job finishes and the assistant checks whether it succeeded. The extraction throughput of 2,910 tok/s validates the hardware, the software stack, and the pipeline design. The successful processing of 1,000 samples (after recovering from an OOM failure) clears the way for the next stage: training the EAGLE-3 draft model. In the broader arc of the session, this message marks the completion of the data preparation pipeline and the beginning of the modeling phase. It is a quiet checkpoint, but it represents hours of debugging, configuration, and patience—the invisible work that makes large-scale ML engineering possible.