The Art of Context Compaction: A Surgical Fix for LLM Context Bloat in Autonomous Agents

Introduction

In the complex world of autonomous LLM-driven systems, few challenges are as insidious as context management. As an agent accumulates conversation history, tool outputs, and system observations, its prompt grows inexorably toward the model's context window limit—at which point the agent begins to forget, hallucinate, or fail entirely. Message 4785 in this opencode session captures a deceptively simple but profoundly effective solution to this problem: a heuristic-based compaction strategy that replaces stale, verbose tool outputs with compact placeholders while preserving full fidelity in the database for human review.

The message reads:

Done. Any tool message (role=tool) with content >300 chars that's more than 10 messages from the end of the conversation gets replaced with [long tool output stale/compacted] when building the LLM prompt. The full output stays in the DB for UI visibility — only the LLM context gets compacted.

>

This handles the main context bloat source: get_offers returns ~50k chars, get_fleet returns ~2k chars. After 10 messages (roughly 2 runs), those get compacted from ~200 tokens to ~6 tokens each.

This short confirmation message, written after the implementation was already coded, tested, and deployed to production, represents a critical architectural insight: the difference between what the LLM needs to see and what the human operator needs to see is not the same thing. By decoupling the persistence layer from the prompt construction layer, the assistant created a system that preserves full auditability while keeping the agent's cognitive load lean.

Why This Message Was Written: The Context Crisis

To understand the significance of this message, one must appreciate the context management crisis that preceded it. The autonomous fleet management agent (built in earlier chunks of this segment) operated on a 5-minute cron cycle, each run accumulating tool outputs from get_offers, get_fleet, diagnose_instance, and other data-gathering endpoints. These outputs were stored in a SQLite conversation log and fed into the LLM prompt on every subsequent run.

The problem was exponential growth. The get_offers endpoint, which queries vast.ai for available GPU instances, returns approximately 50,000 characters of JSON—a firehose of pricing data, hardware specs, and availability information. The get_fleet endpoint returns roughly 2,000 characters of fleet state. After just a few runs, the accumulated tool outputs could easily consume 10,000–20,000 tokens of the model's context window, leaving insufficient room for the system prompt, conversation history, and the agent's own reasoning.

The user recognized this problem and, in message 4776, issued a direct directive: "For context management, if there is tool call with output longer than 300 characters and was more than 10 messages ago, replace the output with '[long tool output stale/compacted]'." This was not a suggestion—it was a specific, actionable requirement that the assistant executed faithfully.

The assistant's response in message 4777 revealed its immediate agreement: "Good idea — much simpler than LLM-based summarization for keeping context lean. Stale tool outputs are the biggest context hogs (those 200-token offer dumps from 10 runs ago)." This acknowledgment is telling: the assistant had been considering more complex approaches (LLM-based summarization) and recognized the user's simpler heuristic as superior. The user's domain expertise—understanding that old pricing data is irrelevant to current decision-making—cut through the complexity.

How the Decision Was Made: From Requirement to Implementation

The implementation path was remarkably linear. The assistant identified the critical function db_msg_to_openai() in vast_agent.py, which converts database conversation records into OpenAI-format messages for the LLM prompt. The modification was surgical: pass the message's distance from the end of the conversation as a parameter, and within the function, check two conditions:

1. Is the message role tool? (Only tool outputs are candidates for compaction.)
2. Is the content longer than 300 characters? (Short outputs pass through unchanged.)
3. Is the message more than 10 positions from the end? (Recent outputs remain visible.) If all three conditions are met, the content is replaced with the placeholder string [long tool output stale/compacted]. The database record itself is never modified—only the in-memory copy used for prompt construction is affected. The assistant's choice of 10 messages as the threshold was not arbitrary. In the agent's operational rhythm, each run produces approximately 5 messages (user request, assistant response with tool calls, tool results, final assistant answer). Two runs thus produce roughly 10 messages. The assistant explicitly stated this reasoning: "After 10 messages (roughly 2 runs), those get compacted." This means that after two observation cycles—about 10 minutes of real time—old pricing data and fleet snapshots are pruned from the agent's active context.

Assumptions Embedded in the Design

The compaction strategy rests on several assumptions that deserve examination.

First, the assumption that old tool outputs are irrelevant to current decision-making. This is generally true for the specific tools identified—get_offers returns current market pricing, which changes rapidly and has no historical value. But it may not hold for all tool types. A diagnose_instance output that reveals a persistent hardware fault, for example, might be relevant for many cycles. The compaction heuristic is indiscriminate: it treats all tool outputs equally based on age and length, not on semantic importance.

Second, the assumption that 300 characters is the right threshold for "long." This value was specified by the user, but it's worth questioning. A 300-character tool output might contain critical diagnostic information (e.g., "GPU 0: ECC error count 127, temperature 84°C, memory utilization 97%") that the agent needs to see. Conversely, a 280-character output might be safely compacted. The threshold is arbitrary and could benefit from tuning.

Third, the assumption that 10 messages (approximately 2 runs) is the right retention window. The agent's operational cycle is 5 minutes, so 10 messages represents about 10 minutes of history. For a system making scaling decisions based on demand trends, 10 minutes of pricing data may be sufficient—but it may also be too short if the agent needs to detect patterns across longer timeframes.

Fourth, the assumption that compaction is purely beneficial. By removing verbose tool outputs, the assistant frees context window for more conversation history and reasoning. But it also removes evidence that the agent might use to ground its decisions. A compacted output is indistinguishable from an error or a missing tool call—the agent sees only the placeholder and must proceed without the data.

Input Knowledge Required to Understand This Message

A reader needs several pieces of context to fully grasp this message.

The architecture of the agent system: The agent runs as a Python script (vast_agent.py) on a management host, triggered by a systemd timer every 5 minutes. It maintains a SQLite database of conversation history, builds an LLM prompt from that history, calls the OpenAI API, executes tool calls, and stores results back in the database. The db_msg_to_openai() function is the bridge between the persistence layer and the prompt construction layer.

The context management challenges: Earlier in the session, the agent suffered from context overflow (38k tokens), duplicate runs, and session reset bugs. The compaction strategy is one of several measures taken to control context growth, alongside no-op run pruning, duplicate message suppression, and conversation summarization.

The specific tools and their output sizes: get_offers returns ~50k characters of GPU pricing data. get_fleet returns ~2k characters of instance state. These are the primary context hogs that the compaction targets.

The user's role and authority: The user is the system operator and domain expert. The directive in message 4776 was a direct instruction, not a suggestion. The assistant's role was to implement it faithfully, which it did.

Output Knowledge Created by This Message

This message creates several forms of knowledge for the reader.

A documented design decision: The compaction strategy is now explicitly recorded in the conversation history, with its rationale, thresholds, and expected impact. Future developers or operators can understand why tool outputs are compacted and under what conditions.

A quantified benefit: The assistant estimates that each compacted message saves approximately 194 tokens (from ~200 tokens to ~6 tokens). With get_offers producing ~50k characters (roughly 12,500 tokens) per call, even a single compaction saves significant context window space.

A precedent for context management: The message establishes a pattern for handling context bloat: preserve full data in the database, compact aggressively in the prompt, and use simple heuristics rather than LLM-based summarization. This pattern can be applied to other verbose tool outputs as the system evolves.

A boundary between persistence and presentation: The explicit statement "The full output stays in the DB for UI visibility — only the LLM context gets compacted" establishes a critical architectural principle. The database is the system of record; the prompt is a derived view optimized for the LLM's consumption.

The Thinking Process Visible in the Message

Although the message is a confirmation rather than a planning document, it reveals significant reasoning.

The assistant identifies the "main context bloat source" by name: get_offers and get_fleet. This indicates an understanding of the system's operational profile—the assistant knows which tools produce the most verbose outputs and can prioritize compaction efforts accordingly.

The quantification of savings—"from ~200 tokens to ~6 tokens each"—shows a concrete understanding of the token economics. The assistant has internalized the cost of verbose tool outputs and can articulate the benefit of compaction in measurable terms.

The phrase "roughly 2 runs" as an approximation for 10 messages reveals an understanding of the agent's operational rhythm. The assistant knows that each run produces approximately 5 messages and can estimate the temporal meaning of the message count threshold.

The choice of the word "stale" in the placeholder is deliberate. It communicates to the LLM that the data was once present but has been removed due to age, not due to an error. This is important for the agent's reasoning—it should understand that the information existed but was compacted, not that the tool failed to return data.

Potential Limitations and Unaddressed Questions

The compaction strategy, while effective, leaves some questions unanswered.

What about tool outputs that are semantically important but old? A diagnostic report from 15 messages ago might contain the root cause of a recurring crash. Under the current heuristic, it would be compacted and invisible to the agent. The system has no mechanism for promoting semantically important outputs above the age/length threshold.

What about partial compaction? A 50,000-character offer dump might contain a few critical data points (e.g., "RTX 5090 available at $0.43/hr on machine 49452") amidst thousands of irrelevant entries. The current approach compacts the entire output rather than extracting the salient information.

What about the interaction with other context management strategies? The compaction runs alongside no-op run pruning, duplicate suppression, and conversation summarization. The combined effect of these strategies on context window utilization has not been measured.

What about the 300-character threshold for non-tool messages? Assistant messages with long reasoning traces or tool call definitions are not compacted. If the agent's own responses become verbose, they could still consume significant context window space.

Broader Significance

This message represents a mature approach to a fundamental challenge in autonomous LLM systems: the tension between comprehensive history and limited context. Rather than attempting to summarize or compress all information, the system makes a pragmatic trade-off: keep everything in the database, but feed the LLM only what it needs.

The heuristic approach—simple, deterministic, and transparent—contrasts with more complex alternatives like LLM-based summarization (which adds latency and cost) or vector database retrieval (which adds architectural complexity). For a system operating on a 5-minute cycle with predictable tool output patterns, the heuristic is appropriate and effective.

The message also illustrates a productive human-AI collaboration pattern: the user identifies a problem and proposes a specific solution; the assistant validates the approach, implements it efficiently, and articulates the expected benefits. The user's domain expertise (knowing that old pricing data is irrelevant) combines with the assistant's implementation skill (modifying the prompt construction pipeline) to produce a clean, surgical fix.

In the broader arc of the autonomous agent's development, this compaction strategy is one of several context management improvements that transformed the system from a fragile, context-overflow-prone prototype into a stable, production-ready fleet manager. The message captures a moment of architectural clarity—the recognition that the database and the prompt serve different purposes and should be managed accordingly.