Recovery from Catastrophe: How a VM Crash Was Turned Into a Clean Resume

The Message

Extraction resumed cleanly:

>

- Skipped 18,421 existing samples — resume working perfectly - 60 new samples extracted, 0 errors, ~1.12 samples/s - ETA: ~281 min ≈ 4.7 hours for remaining ~18,891 samples - /data on new 15TB NVMe, 9.4 TB free

>

Recovery summary: - vLLM auto-start service stopped and disabled - All 18,421 previously extracted hidden states survived the crash intact - SGLang restarted with HS dump patch, extraction resuming from where it left off

At first glance, this message ([msg 4215]) reads as a routine progress update — a few bullet points confirming that a machine learning data extraction pipeline is back on track. But this seemingly mundane status report is actually the culmination of a tense, multi-hour recovery operation following a catastrophic infrastructure failure. To understand why this message matters, we must reconstruct the crisis that preceded it and examine the chain of decisions that made this clean resume possible.

The Context: A Ceph Cluster Runs Out of Space

The story begins with a disaster. The user's virtual machine was running on a Ceph storage cluster that catastrophically ran out of space ([msg 4192]). When a Ceph cluster exhausts its capacity, the consequences are severe: I/O operations freeze, processes become stuck in uninterruptible sleep (state Dl), and the entire system becomes unresponsive. The user had no choice but to kill the VM entirely.

This was not a simple reboot. The underlying storage architecture was changed: a new 15TB NVMe drive was attached directly to the host machine (bypassing Ceph entirely), and the /data volume — which held the precious hidden state extraction outputs — was migrated to this new disk. The container was then brought back up, but with no guarantee that any of the work-in-progress had survived.

When the assistant first reconnected ([msg 4187]), it found a confusing picture. The GPUs showed ~90GB of memory allocated per device — the Kimi-K2.5 model was loaded — but utilization was at 0%. The extraction process was in state Dl (uninterruptible sleep, waiting on disk I/O), and the log hadn't advanced in hours. The last log line showed sample 18,422, but the server's internal counter was at 22,391 requests. Something was deeply wrong.

The First Discovery: vLLM Had Auto-Started

The assistant's initial investigation revealed that a vLLM inference server had automatically launched on boot ([msg 4195]). This was not a random occurrence — a systemd service called vllm-kimi-k25-int4.service was configured to start at boot, loading the Kimi-K2.5 INT4 model across all 8 GPUs with --gpu-memory-utilization 0.95. This was a leftover from a previous deployment, now consuming all GPU memory and preventing the SGLang-based extraction server from loading its model.

The assistant made a critical decision here: rather than simply killing the vLLM process, it both stopped and disabled the systemd service ([msg 4198]). This was a forward-looking choice — simply killing the process would have allowed it to restart on the next reboot, causing the same problem again. By running systemctl disable, the assistant permanently removed the service from the multi-user.target boot sequence. This decision reflects an understanding that infrastructure automation must be durable, not just reactive.

However, there was a subtle complication. The initial attempt to stop the service via direct SSH to the container IP timed out ([msg 4197]). The assistant had to pivot to an alternative approach: SSH-ing to the Proxmox host (10.1.2.6) and using pct exec 129 to execute commands inside the container directly. This workaround succeeded, and the service was removed. The assistant then killed all remaining Python processes and forcefully released the NVIDIA devices with fuser -k /dev/nvidia*.

Assessing the Damage: What Survived?

With the GPUs freed (confirmed by nvidia-smi showing 0 MiB on all devices, [msg 4199]), the assistant turned to the critical question: had the 2.3 TB of hidden state data survived the disk migration?

The answer was a cautious affirmative. The new 12T NVMe volume showed 2.4 TB used out of 12 TB ([msg 4200]), and the hidden states directory structure was intact. A careful inventory ([msg 4202]) revealed:

The Hidden State Dump Patch: A Persistent Modification

One of the most interesting technical details in this recovery is the hidden state (HS) dump patch. The assistant had previously modified SGLang's source code — specifically the deepseek_v2.py model file — to add a custom hidden state dumping mechanism. This patch intercepts the model's internal representations during inference and writes them to a specified directory (/dev/shm/sglang_hs).

The assistant checked whether this patch survived the reboot ([msg 4203]). Because the SGLang installation lived in /root/sglang/ (on persistent storage, not /tmp), the patch was still intact. This was a fortunate consequence of the installation location — had SGLang been installed in a temporary directory, the patch would have been lost and would need to be re-applied.

However, the extraction script itself — /tmp/02b_extract_hidden_states_sglang.py — was gone. The /tmp directory is ephemeral and does not survive reboots. The assistant had to re-copy this script from the host machine using scp ([msg 4206]).

Restarting the SGLang Server: A Deliberate Configuration

With the GPUs freed and the extraction script restored, the assistant launched a new SGLang server ([msg 4207]). The server command is worth examining in detail, as it reveals several assumptions and decisions:

SGLANG_HS_DUMP_DIR=/dev/shm/sglang_hs \
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 \
NCCL_PROTO=LL NCCL_ALGO=Ring NCCL_P2P_LEVEL=SYS \
NCCL_MAX_NCHANNELS=16 NCCL_BUFFSIZE=16777216 NCCL_NTHREADS=512 \
~/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/kimi-k2.5-int4 \
  --trust-remote-code --tp-size 8 --mem-fraction-static 0.88 \
  --host 0.0.0.0 --port 8000 \
  --disable-cuda-graph --disable-radix-cache --disable-custom-all-reduce

The SGLANG_HS_DUMP_DIR environment variable activates the custom hidden state dumping code path. The NCCL environment variables configure the communication library for the 8-GPU tensor-parallel setup: using LL (Low Latency) protocol, Ring algorithm, and system-level P2P. The --mem-fraction-static 0.88 reserves 88% of GPU memory for the model, leaving headroom for the hidden state buffers. The --disable-cuda-graph, --disable-radix-cache, and --disable-custom-all-reduce flags are all performance trade-offs: they disable optimizations that would interfere with the hidden state extraction or that don't work well with the specific model architecture.

The server took 680 seconds (~11 minutes) to become healthy ([msg 4210]). This is expected for a 1-trillion-parameter MoE model loading across 8 GPUs — the checkpoint has 64 safetensor shards that must be loaded and distributed.

The Resume Mechanism: How Skipping Works

The extraction script's resume capability is a key design feature. The assistant confirmed it was working by checking the log after 60 seconds ([msg 4213]):

Skipping 15002 (already exists)
Skipping 15502 (already exists)
Skipping 16002 (already exists)
Skipping 16502 (already exists)
Skipping 17002 (already exists)
Skipping 17502 (already exists)
Skipping 18002 (already exists)
[18424] 4 extracted (18425 total), 0.91 samples/s

The script checks for the existence of output .pt files before processing each sample. If the file exists, it skips the sample entirely. This is a simple but effective idempotency mechanism — it allows the extraction to be interrupted and resumed arbitrarily without double-processing or data corruption.

The assistant's decision to clear /dev/shm/sglang_hs/req_* before restarting ([msg 4212]) was important. The shared memory directory contained stale request data from the previous extraction run. If these files weren't cleaned, the new extraction process might encounter confusing state or file conflicts.

What Message 4215 Actually Communicates

Returning to the target message itself, we can now appreciate what each bullet point represents:

"Skipped 18,421 existing samples — resume working perfectly" confirms that the idempotency mechanism functioned correctly. Not a single previously-extracted sample was re-processed, saving approximately 4.7 hours of redundant computation.

"60 new samples extracted, 0 errors" shows that the extraction pipeline is healthy and producing clean data. The previous run had 3 errors (visible as missing files in rows 4000-10000), but the new run started error-free.

"ETA: ~281 min ≈ 4.7 hours" provides a realistic timeline. This is based on the observed throughput of ~1.12 samples/second, which translates to roughly 4.7 hours for the remaining ~18,891 samples. This estimate assumes stable conditions — no further disk issues, no GPU problems, and consistent request latency from the SGLang server.

"/data on new 15TB NVMe, 9.4 TB free" confirms that the storage migration was successful and that there is ample space for the remaining extraction. The hidden state files are large (~87 MB per sample for the final hidden state plus three auxiliary layers), so the 9.4 TB of free space provides a comfortable margin.

"vLLM auto-start service stopped and disabled" addresses the root cause of the initial failure. By disabling the systemd service, the assistant ensured that future reboots won't trigger the same conflict.

"All 18,421 previously extracted hidden states survived the crash intact" is perhaps the most important confirmation. The Ceph failure could have easily corrupted the data, but the migration preserved it.

"SGLang restarted with HS dump patch, extraction resuming from where it left off" summarizes the entire recovery operation in a single sentence.

Assumptions and Potential Mistakes

Several assumptions underpin this message, and it's worth examining them critically.

The assistant assumed that the extraction script's file-existence check is sufficient for correct resume. This is true only if the previously written .pt files are complete and uncorrupted. If any of the 18,421 files were truncated or corrupted during the Ceph failure, the script would skip them and produce a gap in the dataset. The assistant did not verify file integrity (e.g., by attempting to load each .pt file and checking that it contains valid tensor data).

The assistant also assumed that the SGLang server's hidden state dump path is deterministic — that the same input produces the same hidden state output. This is generally true for a frozen model with deterministic sampling (temperature=0), but the extraction script uses the model's generate function, which may involve non-deterministic operations depending on the CUDA environment.

The ETA of 4.7 hours assumes linear throughput. If the remaining samples have longer sequences (the max is 8192 tokens), throughput could decrease. The assistant's earlier observation of "1.09 samples/s, 2582 tok/s" ([msg 4188]) suggests that throughput is token-limited rather than sample-limited, so longer samples would reduce the sample rate.

A subtle potential mistake: the assistant cleared /dev/shm/sglang_hs/req_* but did not verify that the HS dump patch's output path matches the extraction script's expectations. If the patch was updated between runs and now uses a different directory structure, the extraction script might fail to find the dumped files.

Knowledge Required to Understand This Message

To fully grasp what message 4215 is communicating, a reader needs knowledge in several domains:

Distributed ML inference: Understanding what tensor parallelism (--tp-size 8) means, how model sharding works across GPUs, and why loading a 1T-parameter model takes 11 minutes.

Storage systems: Understanding Ceph, NVMe, the difference between ephemeral (/tmp) and persistent storage, and why a Ceph cluster running out of space causes processes to hang in state Dl.

System administration: Understanding systemd services, how to stop and disable them, and the implications of auto-start services on server reboots.

The EAGLE-3 training pipeline: Understanding why hidden states need to be extracted, what "auxiliary layers" are in the context of speculative decoding, and why 18,421 samples out of 37,312 represents 49.4% completion.

The HS dump patch: Understanding that this is a custom modification to SGLang's model code that intercepts intermediate representations during inference and writes them to disk.

Knowledge Created by This Message

Message 4215 creates several pieces of actionable knowledge:

  1. The recovery was successful — the pipeline can be trusted to survive infrastructure failures if the data storage is properly migrated.
  2. The resume mechanism works — the file-existence check is a viable idempotency strategy for this use case.
  3. The systemd service must remain disabled — if the VM reboots again, the vLLM service won't interfere.
  4. The extraction rate is ~1.12 samples/second — this provides a baseline for estimating future extraction runs.
  5. The new NVMe storage has sufficient capacity — 9.4 TB free for the remaining ~18,891 samples, with each sample consuming approximately 130 MB (2.3 TB / 18,421 samples).

Conclusion

Message 4215 is a masterclass in concise technical communication. In six bullet points, it conveys the outcome of a complex recovery operation spanning approximately 30 messages and several hours of real time. The message's brevity belies the depth of work behind it: diagnosing a stuck process, discovering an auto-started service, working around a network timeout by pivoting through the Proxmox host, verifying 2.3 TB of data integrity, restoring a missing script, waiting 11 minutes for a model to load, and confirming that the resume mechanism functions correctly.

The message also demonstrates a principle that experienced engineers learn through hard experience: the best status update is one that communicates not just what happened, but what was at stake and why the outcome matters. Every bullet point in message 4215 implicitly answers a question the user would have: "Did my data survive?" (yes), "Will this happen again?" (no, the service is disabled), "How much longer?" (4.7 hours), and "Is the new storage better?" (15TB NVMe, 9.4 TB free).

In the broader narrative of the EAGLE-3 training pipeline, this message marks the transition from crisis recovery to steady-state progress. The extraction would continue for another 4.7 hours, producing the 37,312 samples needed to train the next-generation speculative decoding drafter. But without the decisions captured in this message — the careful data verification, the systemd service cleanup, the script restoration, the server configuration — that training would never have happened.