The 404 That Revealed a Routing Mystery: A Diagnostic Probe in the Vast.ai Manager

Introduction

In the course of building a distributed GPU proving infrastructure for the Filecoin network, a seemingly trivial HTTP response—"404 page not found"—became the focal point of a methodical debugging session. The message at <msg id=1038> consists of a single bash command and its output:

[assistant] [bash] ssh 10.1.2.104 'curl -s http://127.0.0.1:1236/api/instances' 2>/dev/null | head -5
404 page not found

On its surface, this is just a failed HTTP request. But within the broader narrative of the opencode session, this probe represents a critical juncture where the assistant's mental model of the deployed system collided with reality. The assistant had built a sophisticated management service—the vast-manager—complete with a web UI, REST API, instance lifecycle management, and benchmark orchestration. But when it tried to query the API to check the status of remote GPU instances, the service responded with a 404, forcing a re-examination of assumptions about how the code had been compiled, deployed, and configured.

The Broader Context: Orchestrating a Distributed Proving Fleet

To understand why this single curl command matters, one must appreciate the complexity of the system being built. The assistant was constructing an automated pipeline for running cuzk PoRep (Proof of Replication) benchmarks on rented GPU instances from Vast.ai, a peer-to-peer GPU marketplace. The system involved:

• A vast-manager service running on a controller host (10.1.2.104) that tracked instances, managed their lifecycle, and collected benchmark results.
• Remote GPU instances (in Norway, British Columbia, Czechia, Belgium) running a custom Docker image with the cuzk proving engine.
• A web UI served on port 1235 for human monitoring.
• A JSON API (presumably on port 1236) for programmatic access by other components. The assistant had just resolved a critical Out-of-Memory (OOM) bug that was killing low-RAM instances during benchmark warmup. New instances had been deployed with a hardened Docker image, and the assistant needed to check their status—were they running? Had they started their benchmarks? Were they producing acceptable proof rates? The API was the designated channel for this information.

The Diagnostic Trail: A Methodical Search

The subject message did not arise in isolation. It was the fourth probe in a systematic investigation spanning messages <msg id=1034> through <msg id=1038>. The assistant's reasoning, visible in the preceding messages, reveals a careful narrowing-down process:

1. First probe (msg 1034): The assistant tried http://127.0.0.1:1235/api/instances and got a JSON decode error—the response body was not valid JSON. This was the first hint that something was wrong, but the error was ambiguous: it could have been a malformed response, a network issue, or a routing problem.
2. Second probe (msg 1035): The assistant tried the same URL again and got "404 page not found". This was clearer: the path /api/instances did not exist on port 1235. But the assistant noted that the Norway instance had just finished downloading parameters and was starting its benchmark, providing a useful status update from a different channel.
3. Third probe (msg 1036): The assistant checked the root URL on port 1235 and confirmed the service was alive—it returned the HTML UI for "Vast Worker Manager". Then it tried /instances (without the /api prefix) on port 1235, which also returned 404. The assistant also checked the Norway instance directly via SSH and confirmed the benchmark was running. This was important: it established that the instances were operational, but the management API was broken.
4. Fourth probe (msg 1037): The assistant switched to port 1236, trying http://127.0.0.1:1236/api/instances. This returned a JSON parse error again—the same ambiguous result as the first probe. The assistant hypothesized that "the deployed binary might have the UI and API on the same port or the routes are different."
5. Fifth probe (msg 1038, the subject): The assistant tried port 1236 again, but this time piped the output through head -5 to see the raw response without trying to parse it as JSON. The result: "404 page not found". This progression shows a clear debugging methodology: each probe eliminates a hypothesis. The assistant first established that the service was running (port 1235 root returned HTML). Then it tested specific paths (/api/instances, /instances). Then it tested the other port (1236). Finally, it removed the JSON parsing layer to see the raw HTTP response, revealing that port 1236 also had no matching route for /api/instances.## Assumptions Made and Broken The assistant operated under several assumptions that this 404 response challenged: Assumption 1: The API lived on port 1236. The assistant's codebase had been designed with a separation between the web UI (port 1235) and the JSON API (port 1236). This is a common architectural pattern—keep human-facing and machine-facing interfaces separate. But the deployed binary might have been compiled from a version where this separation was not yet implemented, or where the routes were registered differently. Assumption 2: The path /api/instances existed. The assistant had written the vast-manager code and knew what routes it should expose. But the 404 response indicated that either the route was never compiled into the binary, or the binary running on the controller host was a different version than expected. This is a classic deployment mismatch: the code on disk may not match the code running in production. Assumption 3: The API was the correct channel for instance status. The assistant was trying to use the JSON API to check on the Norway, BC Canada, Czechia, and Belgium instances. But the failure forced it to fall back to alternative methods: SSHing directly into the remote instances, checking process lists, and reading log files. This revealed that the system was not yet as self-contained as intended—the management plane had a gap. Assumption 4: The 404 was a routing problem, not a service problem. The assistant correctly deduced that the service was alive (the root URL returned HTML), so the issue was with specific routes. But it could also have been a reverse proxy configuration issue, a missing dependency, or a crash in the API handler that caused the framework to return a catch-all 404.

Input Knowledge Required

To understand this message, one needs knowledge of:

• HTTP semantics: The 404 status code means "Not Found"—the server is running but cannot find a resource matching the request URI.
• The vast-manager architecture: That it exposes a web UI on port 1235 and a JSON API on port 1236, and that /api/instances is the expected endpoint for listing managed instances.
• The SSH tunneling setup: The controller host (10.1.2.104) is accessed via SSH, and the assistant is running curl against its loopback interface (127.0.0.1), meaning the service is expected to be listening locally.
• The deployment pipeline: That a Go binary was compiled, pushed as part of a Docker image, and deployed to the controller host. The version of that binary determines which routes are available.
• The broader proving infrastructure: That remote GPU instances in various geographic locations are running benchmarks, and the manager needs to track their progress.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

1. Port 1236 also returns 404 for /api/instances. This ruled out the hypothesis that the API was simply on a different port. Both ports 1235 and 1236 lacked the expected route.
2. The raw response is "404 page not found", not a JSON error body. This is important because it tells us the HTTP framework being used (likely a Go web framework like Gin or Echo) returns a plain-text 404 by default when no route matches. A JSON API would typically return {"error":"not found"} or similar.
3. The assistant needs to inspect the actual binary. The next logical step (which occurs in subsequent messages) would be to check the binary's version, look at the compiled routes, or rebuild and redeploy with the correct API endpoints.
4. The system is not yet self-monitoring. The assistant had to resort to SSHing into remote instances directly to check their status, revealing that the management plane was incomplete. This would likely trigger work to either fix the API routes or add alternative monitoring channels.

The Thinking Process: A Window into Debugging Methodology

The reasoning visible across messages <msg id=1034> through <msg id=1038> reveals a structured debugging approach:

1. Observe symptom: API call fails with JSON decode error.
2. Refine probe: Remove JSON parsing to see raw response → "404 page not found".
3. Check service health: Verify the service is running (root URL returns HTML).
4. Test alternative paths: Try /instances without /api prefix.
5. Test alternative ports: Try port 1236.
6. Eliminate parsing artifacts: Use head -5 to see raw response on port 1236. This is textbook troubleshooting: isolate the failure mode, verify the infrastructure layer, then narrow down to the specific component. The assistant did not panic or restart services—it methodically gathered data until the failure mode was clearly characterized. The 404 response itself, while seemingly trivial, was a significant data point. It told the assistant that the code running on the controller host did not match the code it had written. This would necessitate either a rebuild/redeploy cycle or a code inspection to find out why the routes were missing. In a production debugging context, a 404 on an expected endpoint is often more informative than a 500 (internal server error) because it points to a configuration or deployment issue rather than a runtime bug.

Conclusion

The message at <msg id=1038> is a single curl command that returned "404 page not found". But in the context of the broader session, it represents the culmination of a diagnostic probe that revealed a fundamental mismatch between the assistant's mental model of the deployed system and reality. The vast-manager service was running, but its API routes were not as expected—a classic deployment drift problem.

This moment is a reminder that in complex distributed systems, the simplest debugging tools (curl, SSH, head) are often the most effective. The assistant did not need a sophisticated monitoring dashboard or an APM tool; it needed to ask the service directly "do you have this route?" and listen to the answer. The 404 was not a failure—it was information. And that information would drive the next round of fixes, whether rebuilding the binary, checking the code for route registration bugs, or redesigning the API architecture entirely.