The 404 That Revealed a Routing Bug: Debugging Go's ServeMux in the vast-manager

In the middle of a complex deployment session for a distributed GPU proving infrastructure, a seemingly trivial HTTP 404 error became the catalyst for an important debugging moment. Message [msg 838] captures the precise instant when an assistant, having just deployed and tested a fleet management service called vast-manager, discovers that one of its REST API endpoints silently returns an HTML error page instead of the expected JSON response. The message is deceptively brief — a single observation followed by a file read — but it encapsulates a moment of diagnostic clarity that reveals the subtle pitfalls of Go's HTTP routing in the standard library.

The Context: A Fleet Management Service Under Test

To understand the significance of this message, one must first appreciate the broader system being built. The assistant had been working on a sophisticated infrastructure for managing GPU instances rented from Vast.ai, a marketplace for cloud GPU compute. The vast-manager service was designed to track instances through a state machine (registration, parameter download, benchmarking, running), monitor their health, and automatically destroy unregistered or unhealthy instances. It also maintained a "bad hosts" list — a mechanism to blacklist problematic GPU providers whose hardware had proven unreliable.

By the time we reach [msg 838], the assistant has already deployed the service to a controller host (10.1.2.104), configured it as a systemd service, installed the vast CLI with API credentials, and tested the core API endpoints: registration, state transitions, status queries, and the bad-host management endpoints. The background monitor had even been tested in production — it correctly destroyed unregistered instances, including one deliberately labeled "kill-me" to verify the mechanism. Only one registered instance (C.32705217) remained running.

The bad-host API was the final piece being validated. The assistant had just tested adding a bad host entry via POST /bad-host (which succeeded) and listing entries via GET /bad-host (which also succeeded). Then came the DELETE test.

The Discovery: A 404 That Shouldn't Exist

The preceding message, [msg 837], shows the assistant running a sequence of curl commands against the vast-manager API:

=== Bad Host API ===
{
  "ok": true
}
[
  {
    "host_id": "99999",
    "reason": "test entry",
    "added_at": "2026-03-11T23:30:22Z"
  }
]
404
jq: parse error: Invalid numeric literal at line 1, column 9

The POST and GET endpoints returned clean JSON. But DELETE /bad-host/99999 returned a raw 404 — not a JSON object with an error message, but a plain HTTP 404 status code. The jq parser then choked on the response, producing a parse error. The subsequent GET /bad-host still showed the entry, confirming the delete had not gone through.

This is the moment captured in [msg 838]. The assistant's reasoning is immediate and precise: "The DELETE returned a 404 HTML page — the routing for DELETE /bad-host/99999 isn't working. Let me check the code. The issue is that Go's http.ServeMux doesn't route /bad-host/99999 to the /bad-host handler — it needs a separate pattern."

The Root Cause: Go's ServeMux Routing Semantics

The assistant's diagnosis reveals a deep understanding of Go's HTTP routing behavior. In Go 1.22, the standard library introduced a new http.ServeMux with enhanced pattern matching, but its routing semantics are not as flexible as those of third-party routers like chi or gorilla/mux. When a handler is registered for the pattern DELETE /bad-host, the mux only matches requests to that exact path. A request to DELETE /bad-host/99999 — where 99999 is a path parameter — does not match the /bad-host pattern. Instead, it falls through to the default handler, which returns a 404 HTML page (the standard Go net/http behavior for unmatched routes).

This is a classic pitfall for developers transitioning from third-party routers to the new standard library mux. The http.ServeMux requires explicit patterns for sub-paths, or a catch-all pattern with manual path parsing. The assistant had likely registered the DELETE handler with a simple /bad-host pattern, expecting it to match any sub-path — an assumption that the standard library does not honor.

The Fix: A Catch-All Pattern with Manual Path Parsing

The assistant's next step, visible in [msg 839], is to read the code to understand the current routing setup. The file read reveals the existing DELETE handler implementation, which expects to extract the host_id from the URL path. The fix, applied in [msg 840], involves switching the mux registration to a catch-all pattern (likely /bad-host/ with a trailing slash) and manually parsing the path to extract the host_id parameter. This approach gives the handler full control over path parsing while remaining within the standard library's routing constraints.

The subsequent messages ([msg 841] and [msg 842]) show the fix being compiled, deployed, and verified. The rebuild produces only compiler warnings about const qualifiers in the SQLite binding (harmless), and the verification test confirms that DELETE /bad-host/99999 now returns {"ok": true} and the entry is removed from the list.

What This Message Reveals About the Debugging Process

Message [msg 838] is a masterclass in efficient debugging. The assistant demonstrates several key practices:

Immediate symptom recognition: The 404 HTML page is not dismissed as a transient glitch. The assistant immediately recognizes it as a routing problem.
Domain-specific knowledge applied instantly: The assistant knows Go's ServeMux behavior well enough to name the exact cause without reading the code first. The file read is confirmatory, not exploratory.
Contextual awareness: The assistant knows that POST and GET work but DELETE does not, narrowing the problem to the DELETE handler's routing registration rather than the handler logic itself.
Proportional response: The fix is minimal and targeted — change the routing pattern, not the handler logic. The assistant does not over-engineer a solution.

Assumptions and Their Validity

The assistant's analysis rests on several assumptions, all of which prove correct:

The 404 is from Go's default handler, not from the application: The HTML nature of the 404 (as opposed to a JSON error response) confirms this. The application's error handler returns JSON; the default mux returns HTML.
The handler logic is correct: The assistant assumes the DELETE handler itself works fine when reached. This is validated after the fix — the same handler code produces the correct JSON response once routing works.
The routing pattern is the only issue: The assistant does not suspect database issues, permission problems, or handler bugs. The symptom pattern (one endpoint failing while others work) correctly isolates the problem to routing.

Input and Output Knowledge

To understand this message, the reader needs knowledge of Go's HTTP routing semantics, particularly the behavior of http.ServeMux with path patterns. Familiarity with REST API design (the convention of DELETE /resource/{id}) and the vast-manager's architecture (the bad-host list, the JSON API design) is also helpful.

The message creates new knowledge about a specific bug in the vast-manager's routing configuration. It documents the symptom (404 on DELETE), the root cause (ServeMux pattern mismatch), and the fix direction (catch-all pattern with manual parsing). This knowledge becomes the basis for the immediate fix and the verification in subsequent messages.

The Broader Significance

While this message describes a small, contained bug fix, it illuminates a recurring theme in software engineering: the gap between developer expectations and framework behavior. Go's http.ServeMux is designed for simplicity and explicitness, but that simplicity can surprise developers accustomed to more flexible routers. The assistant's ability to bridge this gap — to recognize the framework's behavior, diagnose the mismatch, and apply the correct pattern — is the hallmark of an engineer who understands not just what the code says, but what the framework expects.

The fix also demonstrates a pragmatic trade-off: rather than introducing a third-party router dependency for path parameter support, the assistant opts for a manual parsing approach that stays within the standard library. This keeps the binary small, avoids dependency management, and aligns with Go's philosophy of simplicity — even if the resulting code is slightly more verbose.

In the end, message [msg 838] is about more than a 404 error. It is about the moment of insight that transforms an opaque failure into a clear, actionable diagnosis — the kind of moment that separates effective debugging from aimless tinkering.