The 429 Guardrail: When LLM Autonomy Meets Hard Code Enforcement

In the development of autonomous LLM agents, one of the most persistent tensions is the balance between giving the model freedom to reason and imposing hard constraints that prevent it from making destructive decisions. Message [msg 4501] captures a pivotal moment in this ongoing negotiation: a single, deceptively simple edit to a Python agent file that adds rate-limit detection to the tool execution loop. The message reads in full:

I need to add rate-limit detection in the tool execution loop. When a launch returns 429, break out of the iteration loop: [edit] /tmp/czk/cmd/vast-manager/agent/vast_agent.py Edit applied successfully.

This message is the capstone of a multi-step debugging and hardening session triggered by the user's request to "Look at how the agent is doing, if what it's doing makes sense, maybe review prompt optimality" ([msg 4486]). What the assistant discovered was a textbook case of LLM-driven automation gone wrong: the agent had been launching GPU instances on vast.ai far beyond what was sensible, burning through its limited budget of LLM inference iterations on futile retries, and ignoring explicit prompt instructions to stop when rate-limited.

The Problem: An Agent That Couldn't Take a Hint

The assistant's analysis in [msg 4489] revealed a cluster of interrelated failures. The agent had launched 8 instances total, with 6 still in a loading/bootstrap state that would take 1–2 hours to become productive. Yet the agent kept launching more, because it saw a gap between current capacity (40 proofs/hour from the single running instance) and the target (500 proofs/hour), without mentally accounting for the ~300 proofs/hour that the loading instances would eventually contribute.

But the more immediately damaging behavior was the rate-limit pathology. The vast.ai API enforces a rate limit of 3 successful instance launches per 15-minute window. Once the agent hit this limit, it didn't stop — it kept calling launch_instance with different offers, receiving HTTP 429 responses, and then trying again in the next LLM iteration. Each retry consumed one of the agent's precious 5 tool-calling iterations per run, wasting tokens and time. The LLM, despite being instructed in the system prompt to respect rate limits, simply could not internalize this constraint when faced with the apparent urgency of unmet demand.

This is a well-known failure mode in LLM agents: the model's training data is full of examples where persistence pays off, and it lacks the visceral understanding that an HTTP 429 means "stop trying." Prompt-level instructions are often insufficient because they compete with the model's ingrained tendency to persevere toward a goal. The assistant recognized that this required not a better prompt, but a hard guardrail in the code.

The Architecture of the Fix

The fix in [msg 4501] was the final piece of a three-part hardening effort that began in [msg 4497] and continued through [msg 4498] and [msg 4499]:

  1. Projected capacity awareness ([msg 4497]): The fast-path logic was rewritten to use projected_proofs_hr (which combines running instance throughput with estimated throughput from loading instances) instead of the raw capacity_proofs_hr. This prevented the agent from seeing a false capacity gap.
  2. System prompt reinforcement ([msg 4498]): The prompt was updated with explicit directives about projected capacity and rate-limit handling.
  3. Rate-limit detection in the tool loop ([msg 4499] and [msg 4501]): The critical piece — a code-level check that intercepts the 429 response from launch_instance and forcibly breaks out of the LLM iteration loop. The specific mechanism works within the agent's tool execution framework. The agent operates in a loop: it calls the LLM, the LLM responds with tool calls (e.g., launch_instance), the agent executes those tools, and feeds the results back to the LLM for the next iteration. The rate-limit detection intercepts the result of execute_tool() and checks for a 429 status code. When detected, it breaks the loop immediately, preventing the LLM from seeing the result and deciding to try again.

Input Knowledge Required

To understand this message, one must grasp several layers of context:

Assumptions and Design Decisions

The fix embodies several important assumptions:

The LLM cannot be trusted with rate limits. This is a significant design decision. The assistant had already tried the prompt-level approach in [msg 4498], adding explicit instructions to stop when rate-limited. But the observed behavior proved that this was insufficient. The assumption is that LLMs, no matter how well-prompted, will prioritize task completion over constraint satisfaction when the two conflict. This is a pragmatic, empirically-driven stance that prioritizes system reliability over model autonomy.

A hard break is better than a soft signal. Rather than injecting a warning message into the LLM's context and letting it decide, the fix forcibly terminates the iteration loop. This means the LLM never gets to "think about" the rate limit — it simply stops. This is a loss of autonomy, but a gain in reliability.

The rate limit is non-negotiable. The fix doesn't attempt to implement a retry-with-delay mechanism or a backoff strategy. It assumes that if you've hit the rate limit, the correct action is to stop entirely for this run. The next run (in 5 minutes) will naturally have a fresh rate-limit window.

Mistakes and Corrected Assumptions

The original design contained a subtle but critical mistake: it assumed that the LLM would naturally handle rate limits because the prompt told it to. This assumption was rooted in an overly optimistic view of LLM reasoning. In practice, the model's training data — which is full of examples where persistence, retrying, and working around obstacles is rewarded — overwhelmed the specific instruction to stop.

This is a recurring lesson in LLM agent development: prompts are not code. A directive in a system prompt is a suggestion, not a constraint. The model will weigh it against all its other priors and goals, and when those goals are strong (e.g., "scale up to meet demand"), the prompt loses.

The assistant's own analysis in [msg 4489] shows this realization in real time. The initial approach was to "update the prompt to clearly state... and include a directive to stop immediately when rate-limited." But by [msg 4499], the assistant had already recognized that prompt changes alone wouldn't suffice, and began adding code-level enforcement. Message [msg 4501] completes this evolution.

Output Knowledge Created

This message produces several forms of output knowledge:

The Broader Significance

This message, for all its brevity, represents a fundamental insight about building reliable LLM agents: the model is a reasoning engine, not a policy enforcer. The LLM excels at understanding complex situations and generating creative solutions, but it is terrible at following hard constraints that conflict with its goals. The job of the engineer is to identify which constraints are non-negotiable and enforce them at the code level, reserving the LLM's autonomy for the decisions that genuinely benefit from reasoning.

The 429 guardrail in [msg 4501] is a small piece of code — likely just a few lines — but it embodies a philosophy that will shape the entire agent architecture going forward. It is the difference between an agent that wants to do the right thing and an agent that must do the right thing. In production systems, only the latter is acceptable.