The Missing Header: How a Single S3 Compatibility Fix Resolved a Loadtest Failure

Introduction

In the course of building and debugging a horizontally scalable S3-compatible storage system built on Kuri storage nodes and YugabyteDB, a seemingly small but critical insight emerged in message 1427 of the coding session. The assistant, after an extensive debugging chain that began with a simple user question about whether the single-node Docker Compose configuration still worked, arrived at a concise realization: the loadtest tool was sending raw HTTP requests to the S3 endpoint without the required x-amz-content-sha256 header, causing every PUT operation to be rejected. The message itself is deceptively brief—a single sentence of analysis followed by an edit command—but it represents the culmination of a multi-step investigation that traversed configuration management, container lifecycle, external storage module initialization, and S3 protocol semantics. Understanding this message requires reconstructing the full debugging journey that preceded it.

The Trigger: A Simple Question

The chain began when the user asked a straightforward question: "Does ./docker-compose.yml still work for single-node mode?" (message 1397). This was not a casual inquiry—it reflected a legitimate concern about whether the root-level Docker Compose configuration, which had been modified during earlier architectural refactoring, remained functional as a standalone development and testing environment. The assistant's investigation revealed that the configuration was missing a settings.env file, a prerequisite for the Docker Compose setup to pass validation. Creating this file from scratch required the assistant to reconstruct the minimal environment variables needed for a single Kuri storage node to operate, drawing on knowledge from the multi-node test cluster configuration stored at /data/fgw2/config/kuri-1/settings.env.

The First Obstacle: External Offload Module Initialization

After creating the initial settings.env and starting the containers, the assistant encountered a failure in the Kuri node's startup sequence. The error log showed that the node could not construct the S3 server because it failed to build the arguments for MakeS3Server, which in turn failed to build a ribsbstore.BlockStore dependency. The root cause was an uninitialized "external offload module"—a subsystem within the Kuri architecture that provides external storage backends. Tracing through the code, the assistant discovered that the LocalWebInfo.maybeInitExternal function required the EXTERNAL_LOCALWEB_URL environment variable to be set. This was a non-obvious dependency: the single-node configuration template had included EXTERNAL_LOCALWEB_PATH, EXTERNAL_LOCALWEB_BUILTIN_SERVER, and EXTERNAL_LOCALWEB_SERVER_TLS, but not the URL variable. Adding EXTERNAL_LOCALWEB_URL=http://localhost:8443 and force-recreating the container resolved the startup failure.

The Second Obstacle: Invalid Content SHA256

With the Kuri node now running, the assistant ran the loadtest to verify end-to-end functionality. The loadtest immediately produced write errors, and the server logs revealed the cause: every PUT request was being rejected with "invalid content sha256:". The S3 handler in the Kuri storage node's getBodyReader function (in server/s3/handlers.go) requires the x-amz-content-sha256 HTTP header to be present on PUT and POST requests. This header is part of the AWS S3 signature mechanism, used to verify that the request body has not been tampered with in transit. The Kuri storage node's S3 implementation, being a full S3-compatible server, enforces this requirement even when authentication is disabled.

The assistant correctly identified that the multi-node test cluster worked because it used a separate S3 frontend proxy (the s3frontend component) that sits between clients and Kuri storage nodes. This proxy automatically adds the x-amz-content-sha256 header to requests before forwarding them to the storage nodes. The single-node Docker Compose configuration, however, exposed the Kuri storage node's S3 server directly on port 8078, bypassing the proxy entirely. The loadtest, which had been developed and tested exclusively against the proxy, had never needed to send this header.

The Decision: Fix the Loadtest, Not the Infrastructure

At this point, the assistant faced a design decision with two viable paths:

Option 1: Add a proxy layer to the single-node Docker Compose. This would make the single-node configuration architecturally consistent with the multi-node test cluster, but it would add complexity to what was meant to be a simple development environment. It would also require maintaining an additional container and configuration.

Option 2: Fix the loadtest to send the x-amz-content-sha256 header. This would make the loadtest compatible with any S3-compatible endpoint, including both the proxy-mediated multi-node cluster and the direct storage node single-node setup. It would also make the loadtest more standards-compliant.

The assistant chose Option 2, reasoning that "the loadtest should work with any S3-compatible endpoint." This was the correct architectural decision: a load testing tool should be a well-behaved S3 client, not one that depends on a specific proxy's header injection behavior. The fix required adding the x-amz-content-sha256: UNSIGNED-PAYLOAD header to the loadtest's HTTP PUT requests, which tells the S3 server that the client is not providing a content hash and the server should skip body verification.

The Subject Message: Insight and Action

The subject message (index 1427) captures the moment of insight:

[assistant] I see - the loadtest is using raw HTTP, not the AWS SDK. It's missing the x-amz-content-sha256 header. Let me add it: [edit] /home/theuser/gw/integrations/ritool/loadtest.go Edit applied successfully.

This message is remarkable for its economy. The assistant had already done the investigative work—reading the loadtest source code, confirming that it used a custom s3Client struct with raw HTTP calls rather than the AWS SDK, and tracing the header requirement in the server's handler code. The message itself is the synthesis: the loadtest's custom HTTP client was missing a header that the AWS SDK would have included automatically. The edit was applied, and subsequent testing (message 1428) confirmed that the fix resolved the issue.

Assumptions and Their Consequences

Several assumptions shaped this debugging journey. The assistant initially assumed that the single-node Docker Compose configuration would work with the same environment variables as the multi-node test cluster, but this assumption was only partially correct—the multi-node cluster's configuration files were generated by a script (gen-config.sh) that included additional variables like RIBS_RETRIEVALBLE_REPAIR_THRESHOLD. The assistant also assumed that restarting the container would pick up the new environment variables, but Docker Compose's restart command reuses the existing container; a --force-recreate was needed to rebuild it with the updated configuration.

More subtly, the assistant assumed that the loadtest's raw HTTP client was functionally equivalent to the AWS SDK for the purposes of testing. While this was true for the proxy-mediated path, it broke down when hitting the storage node directly because the proxy had been silently compensating for the missing header. This is a classic leaky abstraction problem: the proxy's header injection was an invisible dependency that the loadtest had unknowingly relied upon.

Knowledge Created

This message and its surrounding context produced several valuable pieces of knowledge. First, it documented that the Kuri storage node's S3 handler requires the x-amz-content-sha256 header even when authentication is disabled—a protocol-level requirement that is independent of the auth configuration. Second, it established that the loadtest tool must be a compliant S3 client, not just a tool that happens to work against the test cluster's proxy. Third, it revealed that the single-node Docker Compose configuration exposes the storage node directly, which is simpler but requires clients to speak the full S3 protocol. Finally, the fix itself—adding the UNSIGNED-PAYLOAD header—became a reusable pattern for any other tools or scripts that might interact with the storage node directly.

Conclusion

Message 1427 is a case study in how distributed systems debugging often works: a simple question leads to a chain of investigations that surface hidden dependencies and assumptions. The assistant's ability to trace the error from "invalid content sha256" through the server handler code, the proxy architecture, the loadtest implementation, and finally to a one-line header fix demonstrates the depth of understanding required to maintain complex systems. The fix itself was small, but the insight behind it—that the proxy and the storage node have different S3 protocol requirements, and that the loadtest must be compatible with both—was the product of careful, methodical investigation. In the end, the single-node Docker Compose was working, the loadtest was more standards-compliant, and the commit was amended to include the fix, leaving the codebase in a cleaner state than before.