The Final Stitch: Implementing Context Compaction for an Autonomous Agent

In the sprawling development of an autonomous LLM-driven fleet management agent for cuzk proving infrastructure, message [msg 4782] stands as a deceptively small but critical stitch in a much larger tapestry. The message reads in its entirety:

Now update the call site to pass the distance: [edit] /tmp/czk/cmd/vast-manager/agent/vast_agent.py Edit applied successfully.

This single line of narration, followed by an edit command, is the second half of a two-part implementation that addresses one of the most insidious challenges in building reliable autonomous agents: context window management. To understand why this message matters, one must trace the chain of reasoning that led to it, appreciate the design trade-offs that were made, and recognize the broader crisis it was designed to solve.

The Context Management Crisis

The agent at the heart of this system operates as a persistent conversational runtime. Each invocation appends new messages—observations, tool calls, tool results, LLM responses—to a SQLite-backed conversation log. On every cycle, the entire history (or a sliding window of it) is loaded into the LLM's context window as a prompt. The problem is obvious in retrospect: tool outputs can be enormous. A single vast_instances call might return a JSON payload describing dozens of GPU instances, each with detailed status fields, pricing data, and performance metrics. After a few cycles, these outputs accumulate, consuming precious tokens and diluting the signal-to-noise ratio of the prompt.

The user identified this problem directly in [msg 4776]: "For context management, if there is tool call with output longer than 300 characters and was more than 10 messages ago, replace the output with '[long tool output stale/compacted]'." This was not a speculative request—it was a surgical response to a concrete operational failure. Earlier in the same segment ([msg 4768]), the agent's context had ballooned to 38,000 tokens because compaction had failed, and a session reset had broken the agent entirely. The user and assistant were fighting a war of attrition against context bloat, and this message represents one of the key tactical victories.

Why Truncation Over Summarization

The assistant's immediate reaction in [msg 4777] reveals a deliberate design choice: "Good idea — much simpler than LLM-based summarization for keeping context lean. Stale tool outputs are the biggest context hogs (those 200-token offer dumps from 10 runs ago)."

This is a revealing moment of architectural reasoning. The assistant considered—and explicitly rejected—the alternative of using the LLM itself to summarize old tool outputs. LLM-based summarization is a popular technique for context management, but it has serious drawbacks in this setting. It adds latency (an extra LLM call per compaction cycle), cost (tokens spent on summarization), and risk (the summary might omit critical details or introduce hallucinations). A simple length-based truncation rule, by contrast, is deterministic, zero-cost, and trivially verifiable. The assistant correctly recognized that the age of the output is a reliable proxy for its relevance: if a tool output is more than 10 messages old and longer than 300 characters, it is almost certainly dead weight in the prompt.

The mention of "200-token offer dumps" provides concrete texture. The agent periodically calls vast_instances to survey available GPU rentals, and the response includes detailed offer data—pricing, machine specs, location—that is essential for the current cycle but useless once the agent has acted on it. These dumps were the primary vector of context bloat.

Tracing the Code Path

The assistant's investigation follows a methodical pattern: acknowledge the idea, trace the relevant code path, and implement the change in minimal, composable edits. After receiving the user's request, the assistant immediately located the critical function—db_msg_to_openai in vast_agent.py—which converts database conversation records into OpenAI chat-format messages for the LLM prompt. This function is the single choke point through which all conversation history flows into the context window. By modifying it, the assistant could apply the compaction rule universally without touching any other part of the system.

The grep in [msg 4777] found the function at line 203. The read in [msg 4778] examined its signature and structure. The grep in [msg 4779] found the call site at line 1150, inside the loop that builds the LLM messages array. The read in [msg 4780] confirmed the loop structure: for m in messages: llm_messages.append(db_msg_to_openai(m)).

This tracing reveals the assistant's mental model. It recognized that the function needed two pieces of information to apply the compaction rule: the message itself (already passed as m) and its distance from the end of the conversation (to determine if it was "more than 10 messages ago"). The natural implementation was to add a distance parameter to db_msg_to_openai and compute it at the call site.

The Two-Edit Pattern

The implementation was deliberately split into two edits, each with a clear purpose. Message [msg 4781] modified the function definition to accept the distance parameter and implement the compaction logic. Message [msg 4782]—the subject of this article—updated the call site to pass the distance.

This split is not accidental. It reflects a disciplined approach to code modification: change the interface first, then update all callers. The assistant could have made both changes in a single edit, but the two-step pattern reduces risk. If the first edit introduces a compilation error (or, in Python, a runtime error from a missing argument), the second edit fixes it. If the second edit fails, the first edit is still valid and can be inspected independently. This is a hallmark of careful engineering—especially valuable when editing code on a remote machine through a shell interface where rollback is cumbersome.

The call site edit itself is minimal. The original line llm_messages.append(db_msg_to_openai(m)) needed to become something like llm_messages.append(db_msg_to_openai(m, len(messages) - i)) (using the loop index to compute distance from the end). The assistant's narration—"Now update the call site to pass the distance"—confirms that the logic was already designed in the previous edit; this message simply completes the wiring.

Assumptions and Potential Pitfalls

The implementation makes several assumptions worth examining. First, it assumes that "more than 10 messages ago" should be measured from the end of the conversation, not from some absolute timestamp. This is correct for a sliding-window context: the most recent messages are the most relevant, and age is measured in conversational turns, not wall-clock time. Second, it assumes that tool output length is a reliable proxy for compaction value. A 300-character threshold is arbitrary but reasonable—it catches the verbose JSON dumps while preserving shorter, potentially critical outputs like error messages or status summaries.

A subtle assumption is that the compaction should happen at conversion time (when building the LLM prompt) rather than at storage time (when writing to the database). This is the right choice: it preserves the full history in the database for debugging and audit purposes while keeping the prompt lean. The placeholder text "[long tool output stale/compacted]" signals to the LLM that data existed but was omitted, preventing the model from assuming the tool call produced no output.

One potential pitfall is that the distance calculation might not account for messages that were already compacted or filtered out. If the conversation contains system messages or other non-tool messages, the distance might not accurately reflect the "staleness" of a particular tool output. However, for practical purposes, the 10-message threshold provides enough margin that minor inaccuracies in distance calculation are unlikely to cause problems.

Broader Significance

This message, for all its brevity, represents a critical inflection point in the agent's reliability. Earlier in the same segment, the agent had suffered a catastrophic context overflow (38,000 tokens) that broke its ability to respond to triggers. The compaction fix, completed in this message, directly addresses that failure mode. By aggressively pruning stale tool outputs, the agent can maintain a stable context footprint even as the conversation grows to hundreds of messages.

The broader lesson is that autonomous agents operating over long time horizons cannot simply accumulate history indefinitely. Context management is not a luxury—it is a fundamental requirement for reliability. The assistant's choice of a simple, deterministic compaction rule over a more sophisticated LLM-based summarization reflects a pragmatic engineering philosophy: prefer the solution that is easy to verify, hard to break, and cheap to run. In the high-stakes environment of GPU proving infrastructure, where every minute of agent downtime translates to lost computational capacity, this philosophy is not just sensible—it is essential.