The 404 That Revealed a Missing API: Debugging the Bad Hosts List

In the sprawling ecosystem of a distributed GPU proving platform, the smallest messages often carry the most weight. Message 1544 in this opencode session is a case in point: a single bash command, a four-word response, and yet it encapsulates an entire debugging journey. The message reads:

[assistant] [bash] ssh 10.1.2.104 "curl -s http://127.0.0.1:1235/api/bad-hosts" | head -c 500 404 page not found

This is the entire content of the message — a tool call and its result. On its surface, it is almost banal: an API endpoint returned a 404. But to understand why this message was written, what it reveals about the assistant's reasoning, and where it led, we must trace the thread of investigation that preceded it.

The Context: Mysterious Instance Deaths

The assistant had been managing a fleet of GPU instances on vast.ai, deploying machines to benchmark their proof-generation performance. In the messages immediately before this one ([msg 1542] and [msg 1543]), the assistant had deployed five new instances: an RTX 5070 Ti in Quebec, a 2× RTX 5060 Ti in the UK, a cheap RTX 5060 Ti in Texas, an RTX 5090 in Illinois, and a 2× RTX 4080 Super in Denmark. These were chosen strategically to gather performance data across diverse hardware configurations.

But something went wrong. Two of the newly deployed instances were killed almost immediately after creation. The assistant's first hypothesis was that these instances had been preemptively blocked by the bad_hosts table — a database table that tracks machines previously deemed unsuitable for proving work. The deploy API had recently been updated to reject offers on known-bad hosts before creating instances (as noted in the chunk summary), so it was plausible that the instances were being rejected at deployment time rather than dying after startup.

The assistant's first attempt to check this hypothesis came in [msg 1543], where it ran:

ssh 10.1.2.104 "curl -s http://127.0.0.1:1235/api/bad-hosts" | python3 -m json.tool

This returned a cryptic error: Extra data: line 1 column 5 (char 4). The python3 -m json.tool utility had failed to parse the response as JSON, suggesting the response was either malformed JSON or not JSON at all. This is where the assistant's debugging instincts kicked in.

The Message: Peeling Back the Abstraction Layer

Message 1544 is the assistant's second attempt to inspect the bad_hosts data. The key difference from the first attempt is the addition of head -c 500 — a Unix pipe that limits output to the first 500 bytes. This is a deliberate debugging technique: when a tool fails to parse a response, the correct next step is to look at the raw response without any parsing layer. By stripping away the JSON parser, the assistant hoped to see exactly what the server was returning.

The result — 404 page not found — was both clarifying and surprising. It confirmed that the response was not JSON at all, but a standard HTTP 404 error page. The /api/bad-hosts endpoint simply did not exist.

This is a classic example of the "leaky abstraction" debugging pattern. The assistant had assumed an API endpoint existed because similar endpoints existed for other data tables (/api/host-perf, /api/dashboard, /api/offers). The pattern of RESTful API design suggested that if there was a host_perf table with a read endpoint, there should also be a bad_hosts table with a read endpoint. But this assumption was incorrect — the developer had implemented the bad_hosts table and the deploy-side rejection logic, but had not created a dedicated API endpoint to query it.

The Assumption and Its Correction

The assistant's assumption was reasonable but wrong. It assumed API completeness — that the set of database tables and the set of API endpoints were in one-to-one correspondence. In reality, software systems are rarely so tidy. The bad_hosts table existed in SQLite, and the deploy API used it for validation, but there was no GET endpoint to list its contents. This is a common oversight in rapid development: the write path (checking bad hosts during deployment) is implemented, but the read path (inspecting the bad hosts list) is neglected because it's not needed for the core workflow.

The mistake was not in the assumption itself, but in the failure to verify it earlier. The assistant had previously interacted with the bad_hosts system — in [msg 1543] it referenced instances being on the "bad hosts list" — but had never actually queried the endpoint to confirm it existed. The first attempt with python3 -m json.tool failed, but the error message was ambiguous: "Extra data" could have meant the JSON had unexpected content, not that the entire response was HTML. It took the second attempt with head -c 500 to reveal the true nature of the problem.

The Thinking Process: Systematic Debugging in Action

What makes this message instructive is the thinking process it reveals, even though the message itself contains no explicit reasoning text. The assistant's behavior shows a clear debugging methodology:

1. Form a hypothesis: The instances were killed because they were on the bad_hosts list.
2. Test the hypothesis: Query the bad_hosts API endpoint.
3. Encounter an unexpected error: The JSON parser fails.
4. Simplify the toolchain: Remove the JSON parser and look at raw output.
5. Interpret the raw result: A 404 means the endpoint doesn't exist.
6. Adapt the approach: Switch to a different method of accessing the data. This sequence — hypothesis, test, unexpected error, strip abstractions, adapt — is the essence of systematic debugging. The assistant did not panic, did not guess, and did not try random fixes. It methodically peeled back layers until it understood the failure mode.

The Resolution and Its Implications

The immediate resolution came in the very next message ([msg 1545]), where the assistant switched tactics entirely:

ssh 10.1.2.104 "sudo sqlite3 /var/lib/vast-manager/state.db 'SELECT * FROM bad_hosts;'"

This bypassed the missing API endpoint entirely and queried the SQLite database directly. The result revealed six entries in the bad_hosts table, including machine IDs 10400 (RTX 5060 Ti, Texas) and 10686 (another RTX 5060 Ti, Texas) — precisely the machines that had been deployed and immediately killed. The hypothesis was confirmed.

But the deeper implication is architectural. The missing API endpoint represents a gap in the vast-manager's interface design. The system had grown organically: the bad_hosts table was added to support deploy-side validation, but the corresponding read endpoint was never created because the developer (the assistant itself, in earlier messages) had been querying the database directly during development. The API was incomplete because the development workflow had not yet demanded completeness. This is a natural consequence of building systems incrementally — features are added where they are needed, and the interface grows to match the immediate use cases, not some idealized specification.

Input and Output Knowledge

To understand this message, the reader needs several pieces of input knowledge: familiarity with the vast-manager system and its API endpoints; awareness that a bad_hosts table existed and was used during deployment; understanding of the HTTP 404 status code; and knowledge of basic Unix debugging tools like head -c and curl.

The output knowledge created by this message is equally important: confirmation that the /api/bad-hosts endpoint does not exist; the realization that the bad_hosts data is only accessible through direct SQLite queries; and the insight that the assistant's mental model of the API surface was incomplete. This knowledge directly shaped the next action — querying the database directly — and informed future decisions about whether to add the missing endpoint or continue using direct database access.

Conclusion

Message 1544 is a study in the power of minimal debugging. In just two lines — a curl command and its 404 response — it encapsulates the entire process of forming a hypothesis, testing it, encountering an unexpected error, stripping away abstraction layers, and adapting. The message is short, but the reasoning behind it is deep. It reminds us that in complex systems, the most valuable debugging tool is not any particular command or technique, but the discipline of systematically peeling back layers until the truth is revealed.