The 12k Token Elephant: Truncating Tool Results to Save an Autonomous Agent's Context

The Message

In message 4612 of this coding session, the assistant wrote:

The problem is that result_str for get_offers is ~50k chars (12k tokens). I'll truncate tool results when persisting to the conversation — the LLM already got the full result in the current call, we just need a summary for history:

>

[edit] /tmp/czk/cmd/vast-manager/agent/vast_agent.py Edit applied successfully.

This short message — barely three lines of reasoning and one edit command — represents a critical inflection point in the construction of an autonomous LLM-driven fleet management agent. It is the moment the assistant recognized that a naive approach to conversation history would inevitably collapse under its own weight, and made a surgical fix that preserved the agent's ability to maintain long-term context without drowning in irrelevant detail.

The Context: Building a Persistent Conversational Agent

To understand why this message matters, we must understand what was being built. The assistant was in the midst of a major architectural pivot: transforming the vast.ai fleet management agent from an ephemeral, stateless cron job into a persistent conversational system. Previously, each agent invocation was a fresh start — the LLM received the current fleet state, made a decision, and that was it. There was no memory, no learning from past actions, no ability to incorporate human feedback.

The new design, implemented over the preceding messages, introduced a rolling conversation log stored in SQLite. Each agent run would load the conversation history, append a new observation, and present the entire thread to the LLM as context. This gave the agent genuine memory: it could reference its own past decisions, learn from mistakes, and incorporate operator preferences injected as user messages into the thread.

But this design carried an implicit assumption: that the conversation history would remain within a manageable size. The user had explicitly set a budget of 30k tokens for the context window ([msg 4577]), and the assistant had dutifully implemented token estimation and planned for summarization. However, the first real test of the system — run #2 — revealed a problem that no amount of clever summarization could solve.

The Discovery: A 12k Token Leak

In message 4607, the assistant inspected the conversation state after run #2 completed successfully. The agent had loaded one message from run #1, appended its observation, called get_offers to find available GPU instances, launched an RTX 5090, and reported its reasoning. The conversation now held 11 messages totaling 12,862 tokens.

The breakdown was shocking:

r#2       tool ~12297tok: {   "offers": [     {       "id": 32871859,       "host_id": 90137,       "machine_id": 21223,

A single tool response — the raw JSON output of get_offers — consumed 12,297 tokens, or roughly 96% of the entire conversation budget. The get_offers tool queries the vast.ai marketplace for available GPU rentals, returning a detailed list of offers with pricing, hardware specs, location, reliability scores, and more. This data is essential for the agent to make informed launch decisions, but it is also enormous — approximately 50,000 characters of dense JSON.

The assistant immediately recognized the crisis ([msg 4608]):

The big problem: get_offers returns ~12k tokens of raw JSON. This will blow up the context fast. Let me truncate tool results before storing them.

The Reasoning: Why Truncation Was the Right Call

The assistant's decision to truncate tool results at the point of persistence, rather than at the point of LLM submission, reveals a nuanced understanding of the system's architecture. The key insight is stated directly in the subject message:

the LLM already got the full result in the current call, we just need a summary for history

This is the critical distinction. The agent's execution loop works as follows:

  1. Load conversation history from SQLite
  2. Append the current observation
  3. Send the full message list (system prompt + history + observation) to the LLM
  4. The LLM responds with tool calls
  5. Execute each tool, getting the full result
  6. Persist the tool result to the conversation history in SQLite
  7. Send the tool result back to the LLM as part of the same turn
  8. The LLM produces its final response The tool result serves two distinct purposes. In step 7, it is fed back to the LLM in the current call, where the LLM needs the full detail to make decisions. But in step 6, it is stored in the conversation history for future calls, where the LLM only needs to remember that a tool was called and what the outcome was — not every field of every offer. By truncating at the persistence step, the assistant preserved the full data for the immediate decision while preventing the conversation history from bloating with irrelevant detail. This is a classic trade-off in LLM application design: the model needs complete information for the current reasoning step, but only summary-level context for historical awareness.

The Implementation: A Surgical Edit

The assistant had already located the relevant code in message 4611, reading lines 1105-1116 of vast_agent.py:

# Execute
result_str = execute_tool(tc_name, tc_args, agent_config)

# Persist tool result to conversation
append_message(
    run_id,
    "tool",
    content=result_str,
    tool_call_id=tc_id,
    name=tc_name,
)

The fix was to truncate result_str before passing it to append_message. The exact edit applied is not shown in the subject message (it says "Edit applied successfully" without revealing the diff), but the intent is clear: cap the stored tool result to a reasonable length, perhaps a few hundred characters that capture the key outcome (e.g., "Launched instance 33014980" or "Found 12 offers, selected cheapest RTX 5090") rather than the full JSON dump.

Assumptions and Their Validity

The assistant made several assumptions in this decision, most of which were sound:

Assumption 1: The LLM does not need historical tool results in full detail. This is generally correct. When the agent reviews past runs, it needs to know what happened — "I launched instance X because demand was Y" — not the raw marketplace data that informed that decision. The LLM's reasoning about past events is based on its own previous analysis, not on re-analyzing raw data.

Assumption 2: Truncation at persistence is safe because the LLM already saw the full result in the current turn. This is correct for the standard execution path. However, there is an edge case: if the LLM's response in step 8 references specific details from the tool result (e.g., "I chose offer 32871859 because it has 48GB VRAM"), those details are preserved in the assistant's own response, which is also persisted. The truncated tool result serves as a fallback reference.

Assumption 3: A simple character or token limit on stored tool results is sufficient. This assumption proved partially correct in the short term, but the assistant would later implement more sophisticated context management, including smart compaction that replaces old tool outputs with placeholders ([chunk 32.3]). The truncation fix was a tactical solution; the strategic solution came later.

What This Message Reveals About the Thinking Process

The subject message is remarkable for what it reveals about the assistant's diagnostic process. The chain of reasoning is visible across messages 4607-4612:

  1. Measurement (msg 4607): The assistant checks the conversation state empirically, discovering the 12k token tool response. This is not speculation — it is data-driven diagnosis.
  2. Identification (msg 4608): The assistant names the problem: "The big problem: get_offers returns ~12k tokens of raw JSON."
  3. Hypothesis formation (msg 4608): The assistant proposes the solution: "Let me truncate tool results before storing them."
  4. Code location (msgs 4609-4611): The assistant searches for the relevant code, reading the exact lines that need modification.
  5. Execution (msg 4612): The assistant states the problem clearly, articulates the rationale, and applies the edit. This pattern — measure, identify, hypothesize, locate, execute — is characteristic of effective debugging. The assistant did not guess at the problem or apply a generic fix. It inspected the actual conversation state, saw the exact token breakdown, traced the code path that produced the bloated message, and made a targeted intervention.

The Broader Significance

This message sits at the intersection of two fundamental challenges in building autonomous LLM agents: context management and information relevance.

Context management is the problem of keeping the LLM's input within its context window while preserving enough information for coherent reasoning. The 30k token budget set by the user was generous by many standards, but a single tool call consuming 40% of that budget demonstrated how quickly naive approaches fail.

Information relevance is the problem of distinguishing what the LLM needs to know right now from what it needs to remember for later. The assistant's insight — that the full tool result is needed for the current decision but only a summary is needed for history — is a general principle that applies to any agent system that persists conversation history.

The fix also reveals an implicit design philosophy: the conversation history is for the LLM's future self, not for the developer's audit log. If the developer needed full tool results for debugging, they would need a separate logging mechanism. The conversation history is optimized for the LLM's consumption, not for human inspection.

Output Knowledge Created

This message created several forms of knowledge:

  1. A fixed codebase: The edit to vast_agent.py prevented context overflow from large tool results, keeping the conversation within the 30k token budget for many more runs.
  2. A design pattern: The principle of truncating tool results at persistence, while keeping full results for the current turn, became a reusable pattern that the assistant would later extend with more sophisticated compaction strategies.
  3. Empirical validation: The assistant learned (and demonstrated) that get_offers returns approximately 50k characters / 12k tokens, a fact that informed subsequent decisions about tool design and context budgeting.
  4. A diagnostic precedent: The method of inspecting conversation state via the API endpoint and analyzing token distribution became a standard diagnostic technique used throughout the rest of the session.

Conclusion

Message 4612 is a small edit with outsized consequences. In three lines of reasoning and one edit command, the assistant averted a context crisis that would have crippled the autonomous agent within a handful of runs. The fix was not flashy — it did not introduce new capabilities or architectural changes — but it was essential. Without it, the agent's conversation history would have ballooned past 30k tokens in run #3 or #4, forcing either a hard reset (losing all context) or a costly summarization pass that would have degraded the quality of historical information.

The message also exemplifies a broader lesson about building LLM applications: the most impactful interventions are often the simplest ones that align the system's data flow with the model's cognitive constraints. The LLM does not need to re-read 12,000 tokens of GPU offers to remember that it launched an instance. Giving it only what it needs — and nothing more — is the essence of effective context engineering.