The Debugger's Grep: Tracing a 400 Error in an Autonomous LLM Agent

Message at a Glance

[assistant] [grep] def call_llm_chat
Found 1 matches
/tmp/czk/cmd/vast-manager/agent/vast_agent.py:
  Line 853: def call_llm_chat(messages: list[dict], tools: list[dict] | None) -> dict:

This is message [msg 4598] in a sprawling coding session where an autonomous LLM-driven fleet management agent is being built from scratch. On its surface, the message is unremarkable — a simple grep command searching for a function definition in a Python file. But this grep sits at a critical inflection point in the session: the moment a major architectural rewrite was deployed, failed on its very first invocation, and the assistant began a systematic debugging descent into the bowels of the HTTP request construction that feeds prompts to the agent's underlying language model.

The Debugging Context

To understand why this grep was issued, we must trace the events of the preceding twenty minutes. The user had asked a penetrating architectural question at [msg 4575]: "Is the agent in one compactable 'conversation' with feedback (Pi agent runtime style) or ephemeral per cron? How are actions linked together?" The assistant's honest answer at [msg 4576] revealed a fundamental limitation: the agent was entirely ephemeral. Each five-minute cron invocation started a fresh Python process, made 1–5 LLM calls, and exited. There was zero conversational continuity between runs. The LLM had no memory of its own reasoning, could not do multi-step planning, and actions were linked only by temporal proximity in a database table.

The user immediately greenlit a redesign at [msg 4577]: "Yeah, keep context to up to 30k tokens." What followed was a rapid, multi-threaded implementation spanning Go backend changes, a Python subagent rewrite, and UI updates. A conversation SQLite table was added. The Go API gained GET/POST/DELETE /api/agent/conversation endpoints. The Python agent was rewritten from the ground up by a subagent task ([msg 4584]) to append observations, decisions, and tool results to a rolling conversation thread. The UI gained a "Conversation" tab. Everything was built, compiled, and deployed in a matter of minutes.

Then came the moment of truth. At [msg 4595], the assistant triggered the first agent run to seed the conversation. The logs showed the agent starting, loading the conversation (0 messages), appending an observation, writing the perf file — and then a truncated line ending with 2026-03-17 1.... The run had hit a 400 error from the LLM API.

Why This Message Was Written

The assistant's reasoning at [msg 4597] reveals the hypothesis driving this grep: "The 400 error is from the LLM call. Let me check what the call_llm_chat sends — probably a content: null or format issue with the tools param when there's only a system prompt + one user message."

This is a classic debugging maneuver. The assistant had just deployed a completely new conversational architecture. The observation had been stored successfully (confirmed at [msg 4596]), the perf file had been written, but the LLM call itself failed. The most likely culprit was the HTTP request format sent to the OpenAI-compatible chat completions endpoint. The assistant needed to examine the function that constructs and dispatches that HTTP request — call_llm_chat — to verify its implementation and trace the format issue.

The grep was the first step in a systematic trace: locate the function, read its implementation, trace the message construction backward to db_msg_to_openai (which converts database conversation records to OpenAI message format), and then trace forward to run_agent where the messages array is assembled. Each step would reveal whether the format was correct or whether some edge case — a null content field, a missing tools parameter, an empty messages array — was triggering the API rejection.

Input Knowledge Required

To understand this message, one must know several things that are not stated in the grep itself. First, the broader architecture: the agent is a Python script that runs on a five-minute timer, fetches fleet state from a Go-based vast-manager API, and makes decisions by calling an LLM (qwen3.5-122b) via an OpenAI-compatible endpoint. Second, the recent architectural change: the agent was just converted from ephemeral per-run invocations to a persistent conversational model where each run appends to a rolling thread stored in SQLite. Third, the failure mode: the first run after deployment returned a 400 HTTP error from the LLM API, meaning the request payload was malformed. Fourth, the debugging hypothesis: the assistant suspected a format issue with how messages were being serialized, particularly around content: null or the tools parameter.

One must also understand the assistant's tool-use conventions. The [grep] prefix indicates a file-search tool that scans for a regex pattern. The output shows the match location. This is not a bash command but a specialized search tool that returns file paths and line numbers. The assistant uses it to navigate large codebases efficiently without reading entire files.

Output Knowledge Created

The grep produced a single line of output: the location of def call_llm_chat at line 853 of /tmp/czk/cmd/vast-manager/agent/vast_agent.py, along with the function signature showing it takes messages: list[dict] and tools: list[dict] | None and returns dict. This knowledge immediately guided the next steps. The assistant now knew exactly where to read. At [msg 4599], it read lines 853 onward to examine the HTTP request construction. At [msg 4600], it grepped for db_msg_to_openai to trace the message conversion logic. At [msg 4602], it grepped for run_agent to see how the messages array was assembled. Each grep built on the previous one, creating a chain of discovered locations that formed a complete picture of the data flow from database to API call.

The Thinking Process

The assistant's reasoning, visible in the surrounding messages, follows a clear pattern. First, observe the symptom: the LLM API returned 400. Second, form a hypothesis based on domain knowledge about OpenAI-compatible APIs: the issue is likely a format problem with content: null or the tools parameter. Third, trace the code path: start at the function that makes the HTTP call (call_llm_chat), examine the message conversion function (db_msg_to_openai), and then check how the messages are assembled in run_agent. Fourth, if the format looks correct, add better error logging to capture the actual API response body — which is exactly what happens at [msg 4604]–[msg 4605].

This is methodical debugging. The assistant does not jump to conclusions or randomly edit files. It reads the relevant code, verifies its understanding, and only then makes changes. The grep is the first step in a directed traversal of the call graph, not a blind search.

Assumptions and Potential Mistakes

The assistant made several assumptions in this debugging session. It assumed the 400 error was a message format issue rather than an authentication problem, a rate-limit rejection, a model availability issue, or a network error. This assumption was reasonable given that the LLM API had been working previously with the old ephemeral agent and the only change was the message construction logic. The assistant also assumed that the OpenAI-compatible endpoint followed the standard chat completions schema, which was validated by the earlier subagent research at [msg 4584] that confirmed qwen3.5-122b passed all tool-calling tests.

A potential mistake was not immediately logging the full API response body. The initial call_llm_chat implementation at line 890–893 only logged the exception on requests.RequestException but did not capture the response body for HTTP errors (400, 422, etc.). The assistant recognized this gap at [msg 4604] and added response body logging, which would reveal the exact error message from the API. This is a common debugging blind spot: handling HTTP errors as exceptions rather than inspecting the error payload that APIs typically return.

The Broader Significance

This message, for all its brevity, captures a universal moment in software engineering: the transition from building to debugging. The conversational agent rewrite was ambitious and executed rapidly. The deployment succeeded — the Go backend compiled, the Python script parsed, the API responded, the UI rendered. But the system failed at the moment of actual use, when the agent tried to think. The grep at [msg 4598] is the first step of the diagnostic journey that follows every deployment of complex systems. It is the moment when theory meets reality, when the clean architecture diagram encounters the messy details of HTTP serialization, and when the engineer must descend from the high-level design to the low-level wire format.

What makes this message particularly interesting is what it reveals about debugging methodology in an AI-assisted coding session. The assistant does not have a debugger, cannot set breakpoints, and cannot inspect variables at runtime. It must reason about the code statically, trace data flows by reading source files, and form hypotheses based on its knowledge of API conventions. The grep tool is its primary instrument for navigating code, and each grep result is a breadcrumb leading deeper into the system. The assistant's debugging is a process of reading, not running — a form of static analysis that relies on the assistant's ability to simulate the execution mentally and spot inconsistencies between the intended format and the actual implementation.

Resolution

The debugging chain continued after this message. The assistant read call_llm_chat at [msg 4599] and found the format looked correct. It then examined db_msg_to_openai at [msg 4600]–[msg 4601], which handles the content: null edge case by explicitly setting msg["content"] = None for assistant messages with tool calls — a required OpenAI format. It then checked the message assembly in run_agent at [msg 4602]–[msg 4603] and found the construction correct. Unable to find a static format error, the assistant added response body logging at [msg 4605] and re-ran the agent at [msg 4606]. The re-run showed more detail, and the debugging continued into subsequent messages, ultimately revealing the root cause and leading to a fix that got the conversational agent operational.

This grep, then, is not just a search command. It is the opening move in a diagnostic campaign that would trace through every layer of the agent's communication with its LLM, from the database schema to the HTTP headers, to understand why a system that should have worked did not. It is a reminder that in complex systems, the gap between "it compiles" and "it works" is often filled with grep commands.