The Missing Curl: A Debugging Micro-Moment in Distributed Systems

The Message

[assistant] [bash] cd /home/theuser/gw/test-cluster && FGW_DATA_DIR=/data/fgw2 docker compose exec kuri-1 curl -s http://localhost:8078/ 2>&1
OCI runtime exec failed: exec failed: unable to start container process: exec: "curl": executable file not found in $PATH

At first glance, this is a trivial failure: a developer tries to run curl inside a Docker container, and curl isn't there. The error message is straightforward, the fix is obvious (use wget or install curl), and the whole exchange takes seconds. But this micro-moment, buried in a long debugging session for a horizontally scalable S3 storage system built on YugabyteDB and IPFS-based storage nodes, reveals far more than a missing binary. It exposes the assumptions, pressures, and reasoning patterns that define real-world distributed systems debugging.

The Context: A Cluster Under Investigation

To understand why this message was written, one must understand the crisis that preceded it. The assistant had been building and iterating on a test cluster for a distributed S3 architecture—a system where stateless S3 frontend proxies route requests to Kuri storage nodes, which in turn store data in a RIBS (Remote Indexed Block Store) layer backed by YugabyteDB for metadata. The architecture is complex: three layers (S3 proxy → Kuri nodes → database), each with its own configuration, health checking, and failure modes.

The immediate trigger for this message was a "No healthy backends" error from the S3 proxy. After rebuilding the Docker image with a new CQL batcher optimization and restarting the cluster, the assistant ran load tests and found that all write operations were failing. The docker compose ps output showed all containers running, but the S3 proxy reported zero healthy backends. The proxy's logs confirmed it had registered both kuri-1 and kuri-2 as backends, but health checks were failing.

The assistant's reasoning chain at this point was methodical and familiar to anyone who has debugged distributed systems:

  1. Check the proxy — is it responding at all? Yes, curl -s http://localhost:8078/ returns "Not Found" (404), which is expected for the root path.
  2. Check the proxy's backend pool — logs show both backends registered but marked unhealthy.
  3. Check the Kuri nodes directly — can they serve S3 requests? This is where the target message enters. The assistant's hypothesis was clear: either the Kuri nodes' S3 API wasn't listening on the expected port, or the health check endpoint was failing for some reason. The natural next step was to bypass the proxy entirely and hit the Kuri node's S3 endpoint directly. Since the architecture exposes Kuri's S3 port (8078) on the host as port 7001, the assistant could have curled localhost:7001 from the host—and indeed, a previous message (msg 1120) shows exactly that attempt, which returned a raw HTTP response. But that test was inconclusive because the response headers didn't clearly indicate whether the S3 API was functional. So the assistant pivoted to a more precise diagnostic: run curl inside the Kuri container itself, hitting localhost:8078 from within the container's network namespace. This would eliminate any Docker networking or port mapping issues from the equation. If the S3 API was listening inside the container, curl localhost:8078 would work. If not, the problem was in the Kuri process itself.

The Assumption That Failed

The command failed not because of a networking issue or a misconfigured S3 API, but because curl was not installed in the container. The Docker image was built from alpine:3.22.0, a minimal base image that does not include curl by default. The Dockerfile (visible in msg 1103) shows a two-stage build: a Go builder stage compiles the binaries, and a minimal Alpine runtime stage contains only the compiled binaries (kuri, gwcfg, s3-proxy). No debugging tools like curl, wget, netstat, or bash are included.

This assumption—that curl would be available inside a container—is one of the most common pitfalls in containerized development. Developers working on host machines with full package managers naturally reach for curl as a universal HTTP testing tool. Containers, especially those built from scratch or minimal Alpine images, often lack these utilities. The assistant's mental model of the container environment was shaped by the development context (a Go project with full tooling) rather than the production context (a minimal runtime image).

The mistake is not a sign of carelessness; it is a natural consequence of context-switching between development and operations. The assistant had been writing Go code, building binaries, and running load tests from the host—all environments where curl is a given. The Docker container, however, was designed for minimal footprint, not for debugging.

The Debugging Pattern: Systematic Isolation

What makes this message interesting is not the error itself but what it reveals about the assistant's debugging methodology. The assistant was following a textbook isolation strategy:

Input Knowledge Required

To understand this message, a reader needs:

  1. Docker Compose fundamentals: The docker compose exec command runs a process in an already-running container, as opposed to docker compose run which starts a new container. The FGW_DATA_DIR environment variable is used by the compose file for volume mounts.
  2. The architecture under test: The test cluster has three tiers—S3 frontend proxy (port 8078 on host), Kuri storage nodes (port 7001/7002 on host, 8078 inside container), and YugabyteDB. The proxy performs health checks against the Kuri nodes' internal S3 endpoints.
  3. The debugging state: The proxy reports "No healthy backends" despite both Kuri nodes being registered. This means the health check HTTP requests from the proxy to the Kuri nodes are failing, even though the containers are running.
  4. Container image composition: The fgw:local image is built from Alpine, and the assistant had just rebuilt it (msg 1104) to include the CQL batcher changes. The assistant knows the image was rebuilt but hasn't verified what tools are inside.
  5. The Go toolchain context: The assistant has been working in /home/theuser/gw, the Go project root, where go run and go build are the primary interaction modes. The shift to Docker-based debugging is a context change.

Output Knowledge Created

The failed command produced two pieces of knowledge:

Direct output: curl is not available in the Kuri container. This immediately informs the next debugging step: try an alternative tool (wget, as seen in msg 1122) or check the container's listening ports via other means (netstat/ss, as seen in msg 1124).

Inferred knowledge (from subsequent steps): The assistant's next attempt used wget, which returned "Connection refused" (msg 1122). This was the actual diagnostic target: the Kuri node's S3 API was not listening on port 8078 inside the container. This led the assistant to check the container's listening ports with netstat (msg 1124), which confirmed that Kuri was listening on ports 34431, 44091, and 2112—but not 8078. The S3 API had failed to start.

This cascade of discoveries—curl missing, then wget connection refused, then netstat showing no S3 listener—traces directly back to the initial failed curl command. Each failure forced a different diagnostic approach, ultimately revealing that the Kuri node's configuration was preventing the S3 API from binding. The root cause was a configuration validation error: RetrievableRepairThreshold > MinimumReplicaCount (3 > 1), which caused the Kuri process to skip initializing its S3 API.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the sequence of commands and their results. The thought process is not explicitly stated in the target message (it is a single bash command), but it is inferable from the surrounding context:

  1. "The proxy says no healthy backends. I need to check if the Kuri nodes are actually serving S3 requests." This is the core motivation. The proxy's health check mechanism is opaque—it could be failing because of a network issue, a configuration mismatch, or because Kuri's S3 API genuinely isn't running.
  2. "I'll eliminate network complexity by testing from inside the container." The docker compose exec approach is deliberately chosen over host-level curl to avoid Docker port mapping, DNS resolution, or firewall issues. This is a sound isolation strategy.
  3. "I'll use curl because it's the standard HTTP debugging tool." This is the assumption that fails. The assistant's muscle memory defaults to curl without checking the container's tooling.
  4. "The error says curl is missing. Let me try wget instead." This is the immediate adaptation, visible in the next message (msg 1122). The thinking is rapid and pragmatic. There is no panic, no over-analysis. The assistant treats the failed command as data, not as a blocker. Within seconds, the next command (wget) is issued, and the debugging continues. This is characteristic of experienced engineers working under time pressure: try something, observe the result, adapt, and move on.

Broader Significance

This message, for all its apparent triviality, is a microcosm of distributed systems debugging. It illustrates: