The Three-Hour Hard Policy: Engineering Autonomous Safeguards in GPU Fleet Management

Message Overview

[assistant] Now add the hard policy: destroy vast instances that have been inactive (exited/error) for >3 hours. Add this after the existing active-instance checks: [edit] /tmp/czk/cmd/vast-manager/main.go Edit applied successfully.

>

LSP errors detected in this file, please fix: <diagnostics file="/tmp/czk/cmd/vast-manager/main.go"> ERROR [1649:2] no new variables on left side of := </diagnostics>

This is message index 4754 in a long-running coding session building an autonomous fleet management system for GPU-based zero-knowledge proof (ZK) proving infrastructure. At first glance, it appears to be a routine code edit — a few lines added to a Go source file. But this message represents a critical inflection point where the assistant translated a nuanced human directive into an automated safeguard, and the resulting LSP error reveals the messy reality of building production systems under time pressure. The message is the opening move in implementing what the assistant's todo list calls "Add hard policy: monitor kills instances inactive >3hr on vast," and it sits at the intersection of operational reliability, cost governance, and autonomous agent design.

The Context: A Fleet in Crisis

To understand why this message was written, we must step back into the preceding twenty minutes of conversation. The session had been consumed by a cascading production failure. Multiple GPU instances on vast.ai — the cloud marketplace providing the compute for CuZK proof generation — had silently died. The local fleet manager's database showed six instances as "running," but vast.ai's API reported them all as "exited." The monitor loop, which was supposed to detect and clean up dead instances, had a critical blind spot: it only killed instances that disappeared entirely from vast.ai's instance list. Instances with actual_status=exited still existed in the API response, so the lookup succeeded, and the monitor considered them alive. The result was a fleet of ghosts — instances accruing storage charges by the hour, doing no useful work, invisible to the automated cleanup.

The assistant had just fixed this bug ([msg 4733]) by extending the monitor to also kill instances whose vast status was exited or error. After deploying the fix and waiting for the vast cache to refresh ([msg 4748]), the monitor successfully cleared the dead instances from the local database. The fleet summary dropped from "4 running" to "0 running" — honest, but alarming.

Then came the user's directive. The assistant had asked whether to destroy the six lingering vast.ai instances (four exited, two stuck in "loading" for hours) to stop storage charges. The user's answer was nuanced and strategically important:

"Extend agent to have insight into vast state and be able to debug it. Local vast manager (non-agentic should kill unschedulable instances, but it should be up to the agent to keep instances inactive and resume if there is demand; HOWEVER there should be hard policy of >3hr inactive instances are killed."

This single response shaped the next hour of development. It established a clear architectural separation: the non-agentic monitor loop handles mechanical cleanup of obviously dead instances after a timeout, while the autonomous LLM agent retains discretion over instance lifecycle decisions during normal operations. The hard policy was a safety net — a circuit breaker that prevents indefinite storage charges even if the agent or human operator fails to act.

What the Message Actually Does

The edit inserts a new "Step 7" into the monitor loop in main.go, after the existing active-instance checks (Step 6). The code that was written (visible in the subsequent message [msg 4755]) iterates over the cached vast.ai instances and checks their ActualStatus. If an instance has been in exited, error, loading, or scheduling status for more than three hours, the monitor programmatically destroys it via the vast.ai API, stopping all charges.

The reasoning behind the three-hour threshold is pragmatic. Instances can legitimately spend time in loading or scheduling states while vast.ai provisions hardware — a few minutes is normal, but hours indicates a stuck allocation that will never resolve. Similarly, exited instances may have terminated due to transient GPU driver faults or OOM kills; if they haven't been manually resumed or reaped by the agent within three hours, they're dead weight. The policy is deliberately aggressive about cost control: storage charges on vast.ai accumulate continuously regardless of instance state, so every hour a dead instance lingers is money wasted.

The LSP Error: A Window Into Real-Time Development

The edit triggered an immediate LSP diagnostic: ERROR [1649:2] no new variables on left side of :=. This is a compile-time error in Go, indicating that the code at line 1649 attempted to use the := short variable declaration syntax in a context where no new variables were being declared — likely a re-declaration of an existing variable name.

The error is significant for what it reveals about the assistant's working process. The edit was applied "successfully" from the text editor's perspective — the bytes were written to disk — but the language server immediately flagged a semantic problem. The assistant did not catch this error before applying the edit; it relied on the LSP diagnostics as a safety net. This is a common pattern in AI-assisted coding: the model writes code quickly, applies it, and then reacts to compiler or linter feedback in the next turn. The approach prioritizes velocity over correctness, trusting the toolchain to catch mistakes.

The root cause, diagnosed in the very next message ([msg 4756]), was a variable name collision. The assistant had written something like vastInstances := s.vastCache inside the new loop, but vastInstances (or a similarly named variable) was already declared in an enclosing scope — possibly the vastCache field access or an earlier declaration in the same function. The fix was trivial: rename the variable. But the error itself is instructive. It shows how even a straightforward policy addition can stumble on mundane naming conflicts, and how the assistant's workflow of "write, apply, diagnose, fix" creates a tight feedback loop that catches these issues within seconds.

Input Knowledge Required

To understand this message, a reader needs to know several things that are established earlier in the conversation:

The vast.ai instance lifecycle. Instances on vast.ai go through states like loading (provisioning), scheduling (waiting for hardware), running (active and billable), and exited/error (terminated). Crucially, exited instances still appear in the API response and continue accruing storage charges even though no compute is running. The monitor loop periodically fetches this list via vastai show instances --raw and caches it.

The monitor loop architecture. The vast-manager runs a periodic monitor function that performs several steps: checking benchmark timeouts, cleaning up old killed records, and validating that all locally-tracked instances still exist on vast.ai. Step 6 (the active-instance check, just before this edit) iterates over the local database's non-killed instances and kills any that have disappeared from vast.ai's view. The new Step 7 operates in the opposite direction: it iterates over vast.ai's instance list and destroys any that have been stuck in non-running states too long, regardless of whether they're tracked in the local database.

The user's architectural vision. The user explicitly distinguished between the non-agentic monitor (which should enforce hard policies mechanically) and the LLM agent (which should make discretionary decisions about instance lifecycle). This separation of concerns is fundamental to the system's design: the monitor is a safety layer that cannot be overridden, while the agent provides intelligent, context-aware management within those constraints.

The cost sensitivity. Storage charges on vast.ai are a significant operational expense. The entire fleet management system was built with cost optimization as a primary goal — the agent tracks spend per proof, prefers cheaper instances, and aggressively cleans up unused resources. The three-hour policy is a direct expression of this cost-awareness.

Output Knowledge Created

This message produces several kinds of output:

Code. The immediate output is a new code block in main.go — approximately 15-20 lines of Go that implement the hard policy. This code becomes part of the production fleet manager binary.

A todo state transition. The assistant's todo list (visible in [msg 4751]) shows "Add hard policy: monitor kills instances inactive >3hr on vast" transitioning from "in_progress" to eventually "completed." This message is the first concrete step toward that completion.

A diagnostic signal. The LSP error creates knowledge that the edit was not clean — there's a compile error to fix. This error propagates into the next assistant turn, where it will be diagnosed and resolved.

A pattern of interaction. This message establishes a template for how the assistant implements user directives: parse the requirement, identify the insertion point, make the edit, check for errors, iterate. The user can see this pattern and trust that the assistant will catch its own mistakes through the LSP feedback loop.

Assumptions and Their Risks

The message rests on several assumptions, some more visible than others:

The three-hour threshold is appropriate. This is the most consequential assumption. Three hours of storage charges on a GPU instance might cost a few dollars — acceptable as a safety margin. But what if an instance legitimately takes longer than three hours to provision? The assistant assumes that loading/scheduling states lasting >3 hours are always pathological, which is reasonable for vast.ai but not guaranteed. A more sophisticated approach might check the instance's age: a 3-hour-old instance still in loading is almost certainly stuck, but a 10-minute-old instance in loading is normal.

The monitor should have unilateral destroy authority. The user explicitly wanted the non-agentic monitor to enforce this policy, but giving a simple polling loop the power to destroy instances is a trust decision. The assistant assumes the monitor's logic is correct and will not false-positive destroy a legitimately running instance. The ActualStatus check is a coarse filter — what if vast.ai's status reporting is temporarily wrong?

The edit location is correct. The assistant assumes that inserting the new code "after the existing active-instance checks" (after Step 6, before the function's return) is the right place. This is a structural assumption about the code's organization. If the monitor function has cleanup logic or state mutations after Step 6, inserting the destroy loop in the middle could cause subtle ordering bugs.

The variable name won't conflict. This assumption was immediately proven wrong by the LSP error. The assistant assumed vastInstances was available as a local variable name, but it conflicted with an existing declaration. This is a minor mistake with a quick fix, but it illustrates the risk of writing code without full awareness of the surrounding scope.

The Thinking Process

The assistant's reasoning is visible across the conversation leading up to this message. In [msg 4752], the assistant explicitly planned three work items and stated "Let me do the monitor policy and agent tools in parallel." The "monitor policy" item is what this message implements.

The choice to add the policy as a new Step 7 rather than modifying the existing active-instance check (Step 6) reveals deliberate architectural thinking. Step 6 iterates over the local database and checks each instance against vast.ai; Step 7 iterates over the vast.ai cache directly. These are complementary but distinct operations: one protects against the database having stale entries, the other protects against vast.ai having orphaned instances. The assistant recognized that the existing code structure (a numbered sequence of steps in the monitor function) was the natural place to add this new behavior, maintaining consistency with the surrounding code.

The assistant also chose to implement this as a monitor-side policy rather than an agent-side tool. This was a direct response to the user's directive: "Local vast manager (non-agentic should kill unschedulable instances)." The assistant respected this architectural boundary, keeping the hard policy in the Go monitor loop (non-agentic) rather than adding it as an agent tool that the LLM would have to remember to call.

The Broader Significance

This message, for all its brevity, captures a crucial moment in the evolution of an autonomous system. The assistant was not just writing code — it was encoding a human value (cost discipline) into an automated policy that would run indefinitely without human oversight. The three-hour hard policy is a commitment: "We will never pay storage charges for a dead instance for more than three hours, no matter what else goes wrong."

The LSP error that follows is equally significant. It's a reminder that AI-assisted coding is not a magic wand — it's a collaborative process where the AI writes, the toolchain validates, and the human (or the AI in the next turn) corrects. The error was caught within seconds and fixed in the next message, but it existed. The final deployed system would be correct, but the path to correctness included a brief wrong turn.

In the larger arc of the session, this message is the first domino in a chain that will include agent-side tools for vast instance visibility (vast_instances, destroy_vast_instance, resume_vast_instance), a redesigned agent prompt that respects the hard policy boundary, and eventually a fully autonomous fleet management agent that can observe, decide, and act within the safety constraints established here. The three-hour policy is the guardrail that makes the autonomous agent safe to run.