The Missing Curl: A Debugging Dead End in a Distributed S3 Cluster

The Message

[bash] docker exec test-cluster-kuri-1-1 curl -s -X PUT --data "direct test" http://localhost:8078/test-bucket/direct.txt 2>&1
OCI runtime exec failed: exec failed: unable to start container process: exec: "curl": executable file not found in $PATH

Context and Motivation

This message, sent by an AI assistant debugging a horizontally scalable S3 architecture test cluster, captures a moment of investigative friction. The assistant is deep in a debugging session, trying to understand why S3 PUT operations through the stateless frontend proxy (port 8078) are not persisting objects into the shared YugabyteDB metadata store. Earlier in the session, the assistant successfully uploaded a test object via curl -X PUT http://localhost:8078/test-bucket/hello.txt, but when attempting to retrieve it with a GET request, the proxy returned a 404 Not Found. A direct query of the filecoingw_s3.S3Objects table confirmed the worst: the table was empty. The object had vanished into the network ether.

The assistant's reasoning at this point follows a classic debugging bifurcation: either the S3 frontend proxy is failing to route the PUT request to a backend Kuri storage node, or the Kuri node is receiving the request but failing to persist it. To isolate which layer is broken, the assistant needs to test the backend directly, bypassing the proxy. This is the motivation behind message 630.

The Debugging Strategy

The assistant had already confirmed network connectivity between the S3 proxy container and the Kuri backend containers using wget to hit the /healthz endpoint. That worked: the backend responded "OK." But health checks only verify that the server is listening, not that its business logic (S3 object storage) functions correctly. The next logical step was to issue an actual S3 PUT request directly to the Kuri node's HTTP endpoint, bypassing the frontend proxy entirely.

The command chosen was:

docker exec test-cluster-kuri-1-1 curl -s -X PUT --data "direct test" http://localhost:8078/test-bucket/direct.txt

This attempts to execute curl inside the test-cluster-kuri-1-1 container, targeting localhost:8078 — which, from inside the container, is the Kuri node's own S3 HTTP server (the same port the frontend proxy would forward to). If this direct PUT succeeded and the object appeared in YugabyteDB, the bug would be isolated to the proxy layer. If it failed, the Kuri node's S3 handler was the culprit.

The Failure and Its Implications

The command failed immediately with an OCI runtime error: exec: "curl": executable file not found in $PATH. The container image — a minimal Docker build based on the project's fgw:local image — does not include the curl binary. This is a common constraint in containerized environments: production-oriented images strip out debugging tools to minimize size and attack surface. The Kuri container likely ships only the compiled Go binary and its runtime dependencies, not general-purpose utilities.

This failure is not a code bug but a tooling limitation. It reveals an assumption the assistant made: that the container would have curl available for ad-hoc debugging. In many development Docker images, this assumption holds — developers often include curl, wget, ping, and other diagnostic tools. But this project's Dockerfile, which the assistant had been iterating on throughout the session, was optimized for production deployment, not interactive debugging.

Assumptions and Mistakes

The assistant made several implicit assumptions in this message:

Curl availability: The most obvious assumption was that curl would be present in the Kuri container. This was incorrect. The container image was built from a multi-stage Dockerfile that copies only the compiled Go binaries (kuri, gwcfg, s3-proxy) into the final stage, not any OS-level debugging tools.
Network namespace equivalence: The assistant assumed that localhost:8078 inside the container would reach the Kuri node's S3 server. This is correct — the Kuri process listens on port 8078 within its own container network namespace. However, the assistant may not have fully considered that the S3 proxy container and the Kuri container are separate network stacks; the proxy reaches Kuri via Docker's internal DNS (e.g., http://kuri-1:8078), not via localhost.
The PUT would succeed if the backend was healthy: The assistant assumed that a successful health check (which returned "OK") implied the S3 PUT handler would also work. In reality, the health check endpoint and the S3 object handler are separate code paths with separate dependencies (database connections, authentication, routing logic). A healthy health check does not guarantee a functional PUT handler.
The debugging approach was sound: Despite the tooling failure, the assistant's debugging methodology was correct — isolate the proxy from the backend by testing directly. The mistake was in the tool selection, not the strategy.

Input Knowledge Required

To fully understand this message, a reader needs:

Docker fundamentals: Understanding that docker exec runs a command inside a running container, and that containers have isolated filesystems that may lack tools present on the host.
OCI runtime behavior: Knowing that the error "executable file not found in $PATH" means the binary simply doesn't exist in the container's filesystem, not that there's a PATH issue.
The architecture under test: The three-layer design of S3 frontend proxy → Kuri storage nodes → YugabyteDB, and the distinction between the proxy's public port (8078 on the host) and the Kuri node's internal S3 port (also 8078, but inside the container network).
The debugging context: The preceding PUT that appeared to succeed but left no trace in the database, and the assistant's need to isolate the failure to either the proxy routing layer or the Kuri storage layer.

Output Knowledge Created

This message creates negative knowledge — it tells the assistant (and the reader) what doesn't work. Specifically:

The Kuri container cannot be debugged with curl: Any future debugging of the Kuri node's HTTP endpoints must either install curl into the image, use an alternative tool available in the container (like wget), or issue requests from outside the container (e.g., from the host or from another container that has curl).
The debugging path must change: Since direct container-internal testing is blocked, the assistant must pivot to other strategies — testing through the proxy with more detailed logging, examining the proxy's routing code, or adding instrumentation to the Kuri node's S3 handler.
Container image design affects debuggability: This is an implicit lesson about the trade-off between minimal production images and debuggable development environments. The assistant may need to create a separate debug image or add a debug mode that includes diagnostic tools.

The Thinking Process Visible

The message reveals the assistant's reasoning through its command construction:

The target selection: test-cluster-kuri-1-1 — the assistant chose the first Kuri node, assuming both nodes are symmetric. This is a reasonable assumption for a horizontally scalable architecture.
The endpoint selection: http://localhost:8078/test-bucket/direct.txt — the assistant targets the Kuri node's own S3 port, using a distinct object key ("direct.txt") to distinguish this test from the earlier proxy-mediated upload ("hello.txt"). This shows careful experimental design: the assistant wants to avoid confusing the results of the two tests.
The data payload: --data "direct test" — a simple string, deliberately different from the earlier "Hello from test cluster!" payload, again to aid traceability.
The error handling: The assistant captured both stdout and stderr (2>&1), ensuring the OCI error would be visible in the output. This is a standard shell practice that indicates the assistant expects possible errors.

The Aftermath

Following this dead end, the assistant pivots to a different approach. In message 631, it tests PUT through the proxy from the host (not inside the container), which reveals a 500 Internal Server Error — a more informative failure than the silent disappearance of the earlier upload. This leads to examining the S3 handler code and discovering that the PUT handler requires a X-Amz-Content-Sha256 header for unsigned requests, which plain curl PUTs don't provide. The assistant then adds auto-injection of this header into the proxy, fixing the issue.

Message 630, then, is a necessary dead end — a debugging hypothesis that was correct in logic but blocked by tooling constraints. It forced the assistant to change tactics and ultimately find the real bug. In the narrative of the debugging session, it's the moment where "test directly on the backend" becomes unavailable, and "test through the proxy with better instrumentation" becomes the only viable path forward. The missing curl binary, frustrating as it was, redirected the investigation toward the actual root cause.