Disaster Recovery in AI Infrastructure: A Todo List That Saved the Pipeline

In the middle of a high-stakes machine learning pipeline — extracting hidden states from a 1-trillion-parameter Kimi-K2.5 model to train an EAGLE-3 speculative decoding drafter — the infrastructure catastrophically failed. The Ceph cluster underlying the VM ran out of space, forcing a hard kill. A new 15TB NVMe disk was attached directly to the host, the /data volume was migrated, and the container was restarted. When the user returned, they found GPU activity but no extraction progress. Their message was succinct: "asses what happened inside, ideally restart the hidden state extraction."

The assistant's response at <msg id=4193> is deceptively simple — a single todowrite tool call updating a structured task list. But this message represents a pivotal moment in the session: the pivot from crisis to recovery. It is the architectural blueprint for disaster recovery in a complex ML pipeline, encoded as a prioritized todo list.

The Message: A Todo List as a Recovery Plan

The message reads:

[todowrite] {"todos":[{"content":"Assess container state after crash/reboot","priority":"high","status":"in_progress"},{"content":"Kill any auto-started GPU processes","priority":"high","status":"pending"},{"content":"Verify /data volume and extracted data integrity","priority":"high","status":"pending"},{"content":"Re-apply HS dump patch if needed, restart SGLang extraction server","priority":"high","status":"pending"},...]}

At first glance, this is just a structured task list — a tool call that updates a persistent todo tracker. But in the context of the session, it is a carefully reasoned recovery plan compressed into four priority items. Each item encodes a specific hypothesis about what went wrong and what must be verified before any recovery action can be taken.

Why This Message Was Written: The Reasoning and Motivation

The motivation for this message is rooted in the fundamental uncertainty of a crash recovery scenario. When a VM is killed, the container restarted, and a new disk attached, the assistant knows almost nothing about the state of the system. It has four critical unknowns:

  1. What processes are running on the GPUs? The user reported "some auto-started on GPUs, probably vllm/sglang from previous setup." If a model server is occupying GPU memory, the extraction cannot proceed. But the assistant cannot assume — it must verify.
  2. Did the data survive? The hidden state extraction had reached 18,421 samples out of 37,312 (49.4%) when the crash happened, consuming 2.3 TB of disk. The new NVMe disk might have the data intact, partially corrupted, or completely lost. The extraction script had a resume capability (skipping existing .pt files), but only if the data files are valid.
  3. Is the SGLang code patch still applied? The hidden state extraction relied on a custom patch to SGLang's deepseek_v2.py that dumped intermediate hidden states during inference. This patch was applied to files in /root/sglang/, which might or might not have survived the container restart. The extraction script itself was in /tmp/, which is ephemeral and almost certainly wiped.
  4. What is the correct recovery sequence? Even if all components survived, the assistant must carefully sequence the recovery: stop conflicting processes first, verify data, check code integrity, then restart the server and extraction in the right order. The todo list format is not arbitrary — it reflects the assistant's understanding that in a distributed system with multiple GPUs, terabytes of data, and complex software dependencies, recovery must be staged. Each item has a clear dependency on the previous one. You cannot verify data integrity while a vLLM server is consuming 75 GB per GPU. You cannot restart extraction without first confirming the server is running with the patch applied.

How Decisions Were Made: The Implicit Recovery Strategy

The decisions encoded in this message are not explicit — they are embedded in the prioritization and ordering of the todo items. The assistant makes several strategic choices:

First priority: assessment over action. The top item is "Assess container state" — not "restart extraction." This is a deliberate choice to gather information before acting. In a crash scenario, the worst thing you can do is blindly restart processes without understanding what changed. The assistant is choosing reconnaissance over reflex.

Second: free the GPUs before touching data. "Kill any auto-started GPU processes" comes before data verification. This ordering reflects the understanding that GPU memory is a shared, exclusive resource. A vLLM server holding 75 GB per GPU would prevent the SGLang extraction server from loading the model. The assistant must reclaim the GPUs before any other GPU-dependent operation.

Third: data integrity before code patching. "Verify /data volume and extracted data integrity" precedes "Re-apply HS dump patch." This ordering reflects the risk calculus: if the data is corrupted, there is no point in patching code. The assistant wants to know whether the 2.3 TB of extracted hidden states are salvageable before investing effort in restarting the server.

Fourth: patch and restart as the final step. Only after confirming data integrity, GPU availability, and code state does the assistant plan to re-apply the patch and restart the extraction server. This is the last item because it depends on all previous verifications passing.

Assumptions Made by the Assistant

The message makes several implicit assumptions, most of which are reasonable but worth examining:

The data survived the migration. The assistant assumes that the new NVMe disk contains the hidden state files from before the crash. This is not guaranteed — the Ceph cluster failure could have caused data corruption, and the migration to the new disk could have been incomplete. The assistant treats this as a verification task rather than an assumption, which is the correct approach.

The vLLM service is the only GPU consumer. The user mentioned "probably vllm/sglang from previous setup." The assistant assumes this is a single systemd service that can be cleanly stopped. In reality, there could be multiple processes, zombie processes, or GPU memory fragmentation that requires more aggressive cleanup.

The HS dump patch survived in /root/sglang/. The assistant assumes that files in /root/ persist across container restarts. This depends on how the container's storage is configured — /root/ might be on an ephemeral overlay or a persistent volume. The assistant correctly flags this as a verification task ("Re-apply HS dump patch if needed") rather than assuming it survived.

The extraction script in /tmp/ is gone. This is a safe assumption — /tmp/ is almost always ephemeral across reboots. The assistant plans to re-deploy the script via SCP.

The resume capability works correctly. The extraction script skips samples that already have .pt files in the output directory. The assistant assumes that the 18,421 existing files are complete and valid, so the script will correctly resume from sample 18,422. If any of those files are corrupted (truncated writes from the crash), the resume logic might skip them and produce incomplete training data.

Input Knowledge Required to Understand This Message

To fully understand what this message is doing, one needs knowledge of:

The EAGLE-3 training pipeline. The broader session involves training a speculative decoding drafter for the Kimi-K2.5 model. This requires extracting hidden states from the target model during inference — a process that involves patching SGLang's model code to dump intermediate layer outputs.

The crash context. The user explained that the Ceph cluster ran out of space, the VM was killed, a new NVMe disk was attached, and the container was restarted. Without this context, the todo list items would seem disconnected from the ongoing work.

The extraction state before the crash. The assistant knew from prior monitoring that extraction was at ~18,421 samples (49.4%) with 3 errors and 2.3 TB of data. This knowledge informs the "Verify data integrity" task — the assistant knows roughly how many files to expect.

The systemd service infrastructure. The assistant knows that vLLM was previously set up as a systemd service (vllm-kimi-k25-int4.service) that would auto-start on boot. This knowledge is implicit in the "Kill auto-started GPU processes" task.

The SGLang patch mechanism. The hidden state extraction depends on a custom patch to deepseek_v2.py that adds _hs_dump_dir support. The assistant knows where this patch lives and how to verify it.

Output Knowledge Created by This Message

The message creates a structured recovery plan that serves as both a communication tool and a state tracker. The todo list:

  1. Communicates intent to the user. The user can see exactly what the assistant plans to do and in what order. This builds trust and allows the user to correct the plan if needed.
  2. Provides a state machine for the assistant. The todowrite tool persists the todo list across messages, so the assistant can track progress and update status as each item is completed. In subsequent messages, the assistant marks items as "completed" or "in_progress" to show recovery progress.
  3. Encodes dependencies between recovery steps. The ordering of items implicitly encodes the dependency graph: assessment → GPU cleanup → data verification → code patching → server restart → extraction restart.
  4. Creates a shared mental model. Both the user and the assistant now have the same understanding of what needs to happen. This prevents miscommunication about priorities and sequencing.

The Thinking Process Visible in the Reasoning

While the message itself is just a tool call, the reasoning behind it becomes visible when you trace the assistant's behavior in subsequent messages. The assistant immediately:

Conclusion

The message at <msg id=4193> is a masterclass in structured disaster recovery for complex ML infrastructure. It transforms a chaotic crash scenario into a clean, prioritized recovery plan. The todo list format is not a crutch — it is a deliberate architectural choice that enables staged, verifiable recovery in a system with multiple interdependent components. Every item encodes a hypothesis, a verification step, and a dependency. The message is the calm, methodical response to infrastructure failure — a todo list that saved the pipeline.