The 400 That Wasn't: Debugging a Conversational Agent's First Breath

The Message

The message construction looks correct. The 400 might be from the qwen3.5-122b model rejecting some format detail. Let me add error response body logging to diagnose: [read] /tmp/czk/cmd/vast-manager/agent/vast_agent.py

This seemingly innocuous message, <msg id=4604>, sits at a critical inflection point in a much larger story: the birth of a persistent conversational runtime for an autonomous LLM-driven fleet management agent. The assistant had just completed a sweeping architectural transformation—rewriting an ephemeral, stateless cron-based agent into a stateful system with rolling conversation history, SQLite-backed memory, and human feedback injection. And then, on its very first run, the agent crashed with a 400 error from the LLM API. This message captures the moment the assistant paused, examined the code, formed a hypothesis, and decided how to proceed.

Context: The Conversational Rewrite

To understand why this message matters, we must understand what preceded it. The agent had originally been designed as a simple cron job: every five minutes, a fresh Python process would start, fetch live fleet state from APIs, build a system prompt from scratch, make a few LLM calls, and exit. Each run was a blank slate. The agent had no memory of its own reasoning, no ability to plan across time horizons, and no way to learn from human feedback beyond a flat list of knowledge strings.

The user recognized this limitation and asked a pointed question at <msg id=4575>: "Is the agent in one compactable 'conversation' with feedback (Pi agent runtime style) or ephemeral per cron? How are actions linked together?" The assistant's honest answer—"Ephemeral per cron"—sparked a major redesign. The user gave the go-ahead with a pragmatic constraint: keep context to 30k tokens maximum.

Over the next several messages, the assistant implemented a complete conversational infrastructure. A conversation_log table was added to the SQLite schema. A Go API was built with GET, POST, and DELETE endpoints for conversation messages. The Python agent was rewritten in a subagent task (see <msg id=4584>) to use a persistent rolling conversation: each run appends its observation, the LLM's response, tool calls, and tool results to a shared thread. Human feedback from alert acknowledgments and knowledge changes is injected as user messages. A context window management system was designed to summarize or prune older messages when the token count approaches 30k.

The deployment at <msg id=4594> confirmed the infrastructure was in place: the conversation API returned zero messages, the UI had a new Conversation tab. Everything was ready for the agent's first breath.

The First Breath Fails

At <msg id=4595>, the assistant triggered the first agent run manually. The logs showed promising activity: the agent loaded an empty conversation, appended an observation for run #1, wrote the performance file. Then—a 400 error from the LLM API. The log was truncated; the assistant couldn't see the error response body.

This is where the debugging chain begins. At <msg id=4596>, the assistant checked the conversation state and confirmed the observation was stored correctly. At <msg id=4597>, the assistant formed an initial hypothesis: "probably a content: null or format issue with the tools param when there's only a system prompt + one user message." Over the next several messages (<msg id=4598> through <msg id=4603>), the assistant traced through the code: the call_llm_chat function, the db_msg_to_openai converter, the run_agent message construction at line 1028.

The Subject Message: A Diagnostic Pivot

Message <msg id=4604> is the culmination of this code-reading phase. The assistant has examined the message construction logic and concluded: "The message construction looks correct." This is a significant statement—it means the assistant has traced through the entire pipeline from database messages to OpenAI-format API call and found no obvious formatting errors, no missing fields, no structural problems.

Having ruled out a code bug in the message formatting, the assistant pivots to a new hypothesis: "The 400 might be from the qwen3.5-122b model rejecting some format detail." This is a reasonable guess. Different LLM providers and models have subtle quirks in their API expectations. Some models reject requests with content: null even when tool_calls are present. Some require specific ordering of messages. Some have undocumented constraints on tool definition formats. The qwen3.5-122b model, while powerful, might have particular sensitivities.

But here's the crucial insight: the assistant cannot verify this hypothesis without more information. The current error handling in call_llm_chat (around line 890 of the agent file) catches requests.RequestException and logs the exception message, but it does not log the response body. A 400 error from an OpenAI-compatible API typically includes a detailed error message in the response JSON—something like {"error": {"message": "..."}}. Without that body, the assistant is flying blind.

The decision to add error response body logging is the core action of this message. It's a textbook debugging maneuver: when you can't see the error, instrument the code to capture it. The assistant reads the file to find the exact location where the error response would need to be logged, preparing to make the edit.

What the Assistant Got Wrong

The assistant's hypothesis was incorrect. The 400 error was not caused by the qwen3.5-122b model rejecting some format detail. The real cause was more subtle and more interesting.

As subsequent messages reveal, the actual bug was a stale local variable. The run_agent function at <msg id=4617> loads the conversation from the database at step 1, storing it in a local messages list. At step 5 (around line 982), it appends the observation to the database via the API. But the local messages list is never updated. When step 8 builds the LLM call, it iterates over the stale local messages list—which, on the first run, contains zero messages. The result is an API call with only a system prompt and no user message, which the LLM API rightfully rejects with a 400.

The assistant's assumption that "the message construction looks correct" was wrong because it examined the formatting code (which was indeed correct) but missed the data flow bug. The messages being passed to the formatter were incomplete because the local list had not been refreshed after the database append.

This is a classic class of bug in stateful systems: the local cache and the persistent store fall out of sync. It's particularly insidious because both the formatting code and the database code look correct when examined independently. The bug only emerges in the interaction between them.

The Deeper Pattern: Debugging Under Uncertainty

What makes this message fascinating is not the bug itself but the debugging methodology it reveals. The assistant is operating under significant uncertainty: it deployed a complex, multi-component system (Go backend, Python agent, SQLite database, LLM API) and observed a failure at the boundary between components. The error message is opaque—just "400" with no body. The assistant must reason backward from effect to cause.

The assistant's approach follows a systematic pattern:

  1. Verify the obvious: Check that the observation was stored correctly (msg 4596). It was.
  2. Trace the code path: Read the message construction logic, the format converter, the API call function (msg 4597-4603).
  3. Form a hypothesis: The formatting looks correct, so the model might be rejecting some detail (msg 4604).
  4. Instrument for more data: Add error response body logging to see the actual error message (the action in msg 4604).
  5. Deploy and test: Push the fix, re-run the agent (msg 4605-4606). Step 4 is the critical insight of this message. The assistant recognizes that without the error response body, it cannot distinguish between competing hypotheses. The 400 could mean "bad request format," "missing required field," "model not found," "rate limited," "content policy violation," or a dozen other things. Adding response body logging transforms an opaque error into a diagnostic signal.

Input Knowledge Required

To understand this message, one needs knowledge of:

Output Knowledge Created

This message creates several forms of knowledge:

The Broader Significance

This message is a microcosm of the challenges involved in building reliable autonomous agents. The agent is not a simple API wrapper—it's a complex system with state management, context windows, tool execution, and human feedback integration. Each of these components can fail in ways that manifest as opaque errors at the system boundary.

The assistant's debugging approach—verify, trace, hypothesize, instrument, test—is the same methodology that human engineers use when faced with similar failures. But the assistant does it faster, reading hundreds of lines of code in seconds and forming hypotheses across multiple components simultaneously.

What's particularly notable is that the assistant's initial hypothesis was wrong, but the debugging approach was correct. Adding error response logging was the right move regardless of the root cause. The actual bug (stale local variable) was discovered only after the assistant deployed the logging fix, ran the agent again, and got a new error message ("No user query found in messages") at <msg id=4615>, which led to the real diagnosis at <msg id=4617>.

This is a powerful lesson about debugging complex systems: you don't need to guess the root cause correctly on the first try. You need to set up the feedback loops that will reveal the root cause when you test your hypotheses. Message <msg id=4604> is the moment the assistant chose to build that feedback loop rather than continue guessing in the dark.