The Lifecycle Bug: When a "Killed" Instance Lives On

A Moment of Discovery in the vast-manager Code

In the midst of a sprawling coding session to build a distributed GPU proving infrastructure for the Filecoin Curio/cuzk ecosystem, a single message captures a moment of sharp insight: the discovery of a lifecycle bug that would have silently wasted money on undestroyed cloud instances. The message at index 1084 is deceptively brief—a single line of reasoning followed by a grep command—but it represents the culmination of a careful forensic analysis through the vast-manager's codebase.

The Context: A Benchmark That Should Have Triggered Destruction

The story begins with the Norway instance (a 1x RTX 4090, 500GB RAM machine) completing its benchmark at 41.32 proofs per hour—below the 50 proofs/hour minimum threshold. The system was designed to automatically destroy underperforming instances to avoid paying for machines that couldn't meet the proving throughput requirements. The assistant had deployed the vast-manager service, a Go-based management system that orchestrates GPU instances on Vast.ai, handling lifecycle states from registered through params_done to either bench_done (success) or killed (failure).

When the Norway instance reported its benchmark results via the /api/bench-done endpoint, the handler correctly determined that 41.32 < 50 and set the database state to killed. The assistant initially assumed this would trigger automatic destruction: "The Norway instance will be destroyed by the monitor automatically (failed benchmark → killed)," as stated in [msg 1078]. But a subsequent check revealed a troubling reality—the instance was still running on Vast.ai, still accruing charges.

The Investigation: Tracing the Missing Destroy Call

What follows in [msg 1079] through [msg 1083] is a methodical code trace. The assistant uses grep to find the monitor's benchmark failure logic, reads the relevant source file, and reconstructs the intended control flow:

  1. The monitor's Step 4 (line 1110 of main.go) queries for instances where state = &#39;bench_done&#39; AND bench_rate &lt; min_rate.
  2. For any such instances, it logs a message and presumably calls destroyInstance.
  3. However, the handleBenchDone handler (line 471) does not set state = &#39;bench_done&#39; when the rate is too low—it sets state = &#39;killed&#39; directly (line 509). This is the crux of the bug. The monitor's Step 4 queries for bench_done state, but the handler bypasses that state entirely for failing instances, jumping straight to killed. The monitor never sees them. And the handler itself never calls vastai destroy. The instance is marked as killed in the database—a tombstone that satisfies the application logic—but the actual cloud resource continues to run, burning money at \$0.137 per hour. The assistant's realization is precise: "So the Norway instance won't get destroyed automatically! This is a bug. The bench-done handler sets state to killed but never calls vastai destroy."

The Subject Message: Planning the Fix

Message 1084 is the response to this discovery. It reads:

I need to add a vastai destroy call when the benchmark fails. But the bench-done handler doesn't have access to the vast instance ID easily. Let me look at the destroyInstance function and how to look up the vast ID from the label:

>

[grep] func.destroyInstance Found 1 matches /tmp/czk/cmd/vast-manager/main.go: Line 1233: func (s Server) destroyInstance(vastID int) {

This message is not a fix—it is the planning for a fix. The assistant has identified the problem and now faces a second-order challenge: the handleBenchDone handler receives a BenchDoneReq that contains a UUID and a label, but the destroyInstance function requires a vastID (the integer identifier assigned by Vast.ai when an instance is created). The mapping from UUID/label to vast ID is not immediately available in the handler's scope.

The assistant's approach is pragmatic: find the existing destroyInstance function, understand its signature, and then figure out how to bridge the gap. The grep returns a single match at line 1233, confirming the function exists and takes an integer vastID. The next step—not shown in this message but implied—would be to either look up the vast ID from the database using the UUID, or extract it from the label (which follows the pattern C.{vast_id}).

Assumptions and Misconceptions

This message reveals several assumptions that shaped the assistant's thinking:

The monitor-will-handle-it assumption. The assistant initially believed the monitor's periodic polling would catch and destroy failed instances. This was a reasonable assumption given the code's structure—the monitor has a dedicated step for "Kill failed benchmarks." But the assumption failed because it didn't account for the state mismatch between the handler (which sets killed) and the monitor query (which looks for bench_done).

The state-machine consistency assumption. The codebase uses a state machine for instance lifecycle, and the assistant assumed the states would be consistent across all paths. The discovery that handleBenchDone bypasses bench_done for failures is a violation of this expectation—it's a shortcut that creates a dead-end state that nothing ever acts upon.

The handler-completeness assumption. The assistant assumed that setting the database state to killed was sufficient to complete the lifecycle. The discovery that no code path actually destroys the Vast.ai resource reveals that the handler is incomplete—it handles the database state but not the cloud resource lifecycle.

Input Knowledge Required

To understand this message, one needs:

Output Knowledge Created

This message produces several valuable insights:

  1. A confirmed bug: The lifecycle gap is now documented and understood. The handleBenchDone handler for failing benchmarks is incomplete—it marks the instance as killed in the database but never destroys the actual cloud resource.
  2. A fix strategy: The assistant knows it needs to add a vastai destroy call in the failure branch of handleBenchDone. The remaining question is how to obtain the vast ID, which the grep will help answer.
  3. A deeper understanding of the codebase: The assistant has traced through the monitor logic, the handler logic, and the state transitions, building a mental model of how the system should work versus how it actually works.
  4. A cost implication: Without this fix, every failing benchmark instance would continue running indefinitely, silently consuming budget. The Norway instance at \$0.137/hr would cost roughly \$100/month if left running—and with multiple instances being tested, the waste could be significant.

The Thinking Process

What's visible in this message is a mind working through a layered debugging process. The assistant started with a high-level assumption ("the monitor will destroy it"), then verified against reality ("it's still running"), then traced the code path to find the discrepancy, and finally arrived at the root cause. The thinking is systematic:

  1. Observation: Norway instance is still running despite benchmark failure.
  2. Hypothesis: The monitor should destroy it via Step 4.
  3. Verification: Read the monitor code—Step 4 queries for bench_done state.
  4. Contradiction: The handler sets killed, not bench_done.
  5. Conclusion: The handler bypasses the monitor's detection logic.
  6. Resolution: The handler itself must call destroyInstance. The subject message represents step 6—the transition from analysis to action. The assistant doesn't just identify the bug; it immediately begins planning the fix by locating the destroyInstance function and considering the mapping challenge.

The Broader Significance

This message is a microcosm of the entire session's theme: building a robust, self-managing infrastructure requires attention to lifecycle details at every level. The OOM fixes in the same session ([chunk 8.0]) addressed memory management during proving; this fix addresses cost management during benchmarking. Both are essential for a system that can run unattended.

The bug also illustrates a common pattern in distributed systems: the gap between application state (database rows) and infrastructure state (cloud resources). The database said "killed," but the cloud resource said "running." Bridging this gap is a fundamental challenge of infrastructure automation, and this message captures the moment of recognizing that gap.

Conclusion

Message 1084 is a turning point. It marks the shift from passive observation ("the instance should be destroyed") to active intervention ("I need to add a vastai destroy call"). The assistant has identified a bug that would silently waste resources, traced its root cause to a state mismatch between two code paths, and begun planning the fix. It's a small message—barely a paragraph and a grep command—but it encapsulates the essence of debugging: the moment when confusion crystallizes into understanding, and understanding demands action.