The Missing Route: Debugging a 404 with a Single Grep

In the middle of a sprawling infrastructure session — spanning Docker builds, GPU benchmark deployments, and a vast.ai management system — the assistant encounters a sudden, puzzling failure. Message 1512 is deceptively simple: a single grep command searching for API route registrations in a Go source file. But this one-line investigation is the culmination of a frustrating debugging sequence, and it reveals a subtle but critical gap between what the assistant expected the API to provide and what it actually offered.

The Message

The assistant's message reads, in full:

[assistant] [grep] mux\.HandleFunc.*api
Found 8 matches
/tmp/czk/cmd/vast-manager/main.go:
  Line 1718: 	mux.HandleFunc("/api/log-push", s.handleLogPush)
  Line 1721: 	mux.HandleFunc("/api/dashboard", s.handleDashboard)
  Line 1722: 	mux.HandleFunc("/api/manager-logs", s.handleManagerLogs)
  Line 1723: 	mux.HandleFunc("/api/instance-logs/", s.handleInstanceLogs)
  Line 1724: 	mux.HandleFunc("/api/kill", s.handleKill)
  Line 1725: 	mux.HandleFunc("/api/offers", s.handleOffers)
  Line 1726: 	mux.HandleFunc("/api/deploy", s.handleDeploy)...

There are no secrets to redact here — this is purely code structure. The assistant ran a regex search for mux.HandleFunc.*api across the vast-manager source tree and received eight matches, seven of which are fully displayed before the output is truncated.

The Context: A Deploy That Broke Something

To understand why this grep was necessary, we need to trace backward through the preceding messages. The assistant had been iterating on the vast-manager — a custom management service that orchestrates GPU proving instances on vast.ai. In [msg 1499], the assistant rebuilt the vast-manager binary with go build and deployed it to the controller host at 10.1.2.104. The deployment was slightly awkward — an initial scp failed with a permission error, then a second attempt with scp as root failed with a "Received message too long" SSH error, forcing the assistant to pipe the binary through cat | ssh instead. Despite these hiccups, the service restarted successfully and showed active (running) in its systemd status ([msg 1502]).

Immediately after deployment, the assistant continued working. It queried /api/offers successfully ([msg 1507]), checked the bad-hosts list and host performance data via /api/bad-hosts and /api/host-perf ([msg 1508]), and everything appeared normal. These endpoints returned valid JSON that the assistant parsed with Python scripts.

Then came the crash. In [msg 1509], the assistant tried to query /api/instances — an endpoint it had clearly used before, since it had a sophisticated Python script ready to parse and display instance state. But instead of JSON, it got something that caused Python 3.14's JSON decoder to throw an exception. The traceback was cut off, leaving the assistant with an unparseable response. In [msg 1510], the assistant tried again, this time piping the raw output through head -c 500 to see what the server was actually returning. The answer was stark: 404 page not found.

A third attempt at /api/data in [msg 1511] confirmed the pattern — another 404.

The Reasoning: "What Routes Actually Exist?"

This is the precise moment captured in message 1512. The assistant has just discovered that a critical API endpoint — /api/instances — no longer exists on the newly deployed binary. The natural debugging question is: What endpoints do exist? The assistant could have read the source file manually, scrolled through hundreds of lines of main.go, or checked the git history to see what changed. Instead, it chose a surgical approach: a single grep for mux.HandleFunc.*api.

The choice of grep over manual reading is itself a decision worth examining. The assistant is operating in a terminal-driven environment where every action must be explicit. Reading the full main.go file would consume bandwidth and cognitive load. Grepping for the route registration pattern is efficient: it extracts exactly the information needed — the complete API surface — in one line. The regex mux\.HandleFunc.*api targets lines where the Go HTTP mux registers a handler function for a path containing "api". This catches all the standard REST endpoints.

The assumption embedded in this grep is that all API routes are registered using mux.HandleFunc with a literal string path containing "api". This is a reasonable assumption for a well-structured Go HTTP server, but it could miss routes registered via other patterns (e.g., route groups, subrouters, or handler functions that register routes dynamically). In this codebase, however, the assumption holds — the grep returns all the active endpoints.

What the Grep Revealed

The output lists seven fully visible routes:

  1. /api/log-push — for instances to push their logs to the manager
  2. /api/dashboard — returns dashboard summary data
  3. /api/manager-logs — returns the manager's own logs
  4. /api/instance-logs/ — returns logs for a specific instance (trailing slash suggests a path prefix)
  5. /api/kill — kills an instance
  6. /api/offers — returns available vast.ai offers
  7. /api/deploy — deploys a new instance Notably absent: /api/instances, /api/bad-hosts, /api/host-perf, and /api/data. Yet the assistant had successfully queried /api/bad-hosts and /api/host-perf just moments earlier in [msg 1508]. How is this possible? This discrepancy is the article's central puzzle. The grep output is truncated after line 1726 with an ellipsis (...), and the header says "Found 8 matches" but only seven lines are shown. The eighth match — likely on line 1727 or later — is not displayed. It's plausible that /api/bad-hosts and /api/host-perf are registered on lines after 1726, or that they are registered through a different mechanism (perhaps a separate mux or a route group). The grep's truncation means the assistant has an incomplete picture.

Assumptions and Their Consequences

The assistant made several assumptions during this debugging sequence. First, it assumed that /api/instances was a valid endpoint. This assumption was based on earlier usage — the assistant had queried it before the binary rebuild and expected it to still exist. But the rebuild may have been based on a different branch, a stale checkout, or a version of the code where that endpoint hadn't been implemented yet. The assistant never verified the endpoint existed before the rebuild; it only discovered the gap when the 404 appeared.

Second, the assistant assumed that the new binary was functionally identical to the old one, just with updated UI strings. The build command in [msg 1499] was straightforward — go build -o vast-manager ./cmd/vast-manager/ — and produced no errors beyond sqlite3 compiler warnings. But the assistant did not run tests or compare the binary size against the previous version. The 404 suggests either that the /api/instances endpoint was never committed to the codebase, or that it was removed in a recent edit.

Third, the assistant assumed that a successful systemctl restart and active (running) status meant the service was fully functional. But a service can start successfully and still serve a different set of routes than expected — especially if the binary was built from different source code than the assistant assumed.

Input and Output Knowledge

The input knowledge required to understand this message includes familiarity with Go's net/http mux pattern (mux.HandleFunc), the concept of REST API route registration, and the structure of the vast-manager codebase. The reader must also understand the preceding debugging sequence — the deploy, the 404s, and the growing frustration as a familiar endpoint vanished.

The output knowledge created by this message is the list of registered API routes. This list serves as a ground-truth inventory of the manager's API surface. It confirms that /api/instances does not exist, and it provides the assistant with the information needed to either add the missing endpoint or find an alternative way to query instance state (perhaps through /api/dashboard or by parsing the database directly).

The Thinking Process

The assistant's thinking process in this message is a model of systematic debugging. The sequence is:

  1. Observe symptom: /api/instances returns 404.
  2. Verify symptom: Try the endpoint again with raw output capture — still 404.
  3. Test hypothesis: Maybe the endpoint was renamed to /api/data — also 404.
  4. Audit the code: Grep for all registered routes to see what actually exists.
  5. Analyze results: Compare the list of existing routes against the list of expected routes.
  6. Identify gap: /api/instances is not in the registered routes. This is textbook debugging: reproduce the failure, narrow the scope, inspect the system's actual state, and compare against expectations. The grep is the inspection step — it replaces speculation with evidence.

Broader Significance

Message 1512 is a small but pivotal moment in the session. It marks the transition from "things are working" to "something is broken" and forces the assistant to confront the gap between its mental model of the API and the API's actual implementation. The missing /api/instances endpoint is not just a 404 — it's a failure of the deployment process, a gap in testing, and a reminder that rebuilding and restarting a service does not guarantee it provides the same functionality.

The message also illustrates a broader truth about infrastructure debugging: the most powerful tool is often the simplest. A single grep — one line of text — can resolve a mystery that minutes of trial-and-error HTTP requests could not. The assistant did not need to read the entire source file, trace through the handler logic, or compare git diffs. It asked the code a direct question — "what routes do you have?" — and the code answered.