The Case of the Wrong Header: Debugging CIDgravity API Authentication in a Filecoin Gateway

In distributed storage systems, the difference between a working integration and a silent failure can be as small as a single HTTP header name. This article examines a pivotal debugging message in a Filecoin Gateway (FGW) coding session where an assistant identified that the CIDgravity API expected an X-API-KEY header rather than the more common Authorization: Bearer scheme—a discovery that unblocked the stalled deal flow and revealed a subtle but critical mismatch between the application code and the external API's expectations.

The Message

The subject message, written by the assistant at index 2260 in the conversation, reads as follows:

I see - the code uses X-API-KEY header (line 77), not Authorization: Bearer. Let me try with the correct header:

>

``bash # Try the actual GOCD endpoint with X-API-KEY header TOKEN=$(ssh 10.1.232.83 "sudo cat /home/fgw/.ribswallet/cidg.token") echo "=== Testing GOCD endpoint with X-API-KEY ===" curl -v --max-time 30 -X POST "https://service.cidgravity.com/private/v1/get-on-chain-deals" \ -H "Content-Type: application/json" \ -H "X-API-KEY: $TOKEN" \ -d '{}' 2>&1 | tail -40 === Testing GOCD endpoint with X-API-KEY === * issuer: C=US; O=Let's Encrypt; CN=R13 * Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption * Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption * subjectAltName: "service.cidgravity.com" matches cert's "service.cidgravity.com" * SSL c... ``

The output is truncated at the SSL certificate chain, but the subsequent message (index 2261) confirms the critical result: the API returned a 200 OK response in approximately 2.6 seconds, delivering 5.3 megabytes of deal state data. The authentication problem was solved.

Context: Why Deals Were Not Flowing

To understand the significance of this message, one must trace the debugging journey that preceded it. The user had asked the assistant to investigate why "rbdeal deals are not flowing" before deploying a new binary with repair workers. The assistant began by examining the kuri node logs and immediately spotted the culprit: every two minutes, the deal check loop logged an error: context deadline exceeded (Client.Timeout exceeded while awaiting headers) when calling https://service.cidgravity.com/private/v1/get-on-chain-deals.

This was a complete blockage. The deal tracker in rbdeal/deal_tracker.go runs a two-phase loop: first it calls runCidGravityDealCheckLoop to reconcile on-chain deal states, and only after that succeeds does it proceed to runSPDealCheckLoop and the cleanup loop that creates new deals. Because the CIDgravity call was timing out, the entire deal-making pipeline was frozen. No new deals could be proposed, no groups could transition from "ready for deals" to "in-progress," and the storage system sat idle despite having 374 TiB of datacap and sufficient wallet balance.

The assistant's initial diagnostic steps were methodical. First, connectivity was confirmed—the node could resolve service.cidgravity.com and establish a TCP connection on port 443. Then, the assistant tested the API with an Authorization: Bearer header using the token stored in /home/fgw/.ribswallet/cidg.token. This returned a 401 Unauthorized response with the message "No API key found in request." The token itself appeared valid in format, but the authentication mechanism was wrong.

The Discovery: Reading the Source Code

At this point, the assistant made a crucial decision: instead of guessing at the correct authentication method, it read the source code. The assistant searched for the CIDgravity API client code and found it in cidgravity/get_deal_states.go. Reading line 77 of that file revealed the truth—the application was sending the API token via the X-API-KEY header, not the Authorization: Bearer header that the assistant had used in the manual curl test.

This is the moment captured in the subject message. The assistant's statement "I see - the code uses X-API-KEY header (line 77), not Authorization: Bearer" represents a synthesis of two information sources: the manual curl test that failed with Bearer auth, and the source code that revealed the correct header name. The assistant then immediately validated this discovery by running the same endpoint with the X-API-KEY header.

Assumptions and Their Consequences

This debugging episode reveals several assumptions—some correct, some incorrect—that shaped the investigation.

The assistant initially assumed that the CIDgravity API used standard Bearer token authentication, which is the most common pattern for REST APIs in 2026. This was a reasonable default assumption, but it was wrong. The CIDgravity API chose a custom X-API-KEY header instead, a pattern more common in older API designs or systems that want to avoid confusion with OAuth2 flows.

The assistant also assumed that the timeout issue was related to authentication. This turned out to be partially correct—the 401 responses from the wrong header would have been fast failures, not timeouts. The actual timeout (30 seconds in the application's HTTP client) was likely caused by the API's slow response time. The subsequent message (index 2261) shows the API took 2.6 seconds to respond with 5.3 MB of data, which is well within the 30-second timeout. This suggests the timeout in production may have been caused by network conditions, request payload differences, or server-side load rather than authentication.

A more subtle assumption was that the token file contained the complete credential in the correct format. The assistant read the token from the filesystem and passed it directly as the header value, which worked. This validated that the token file format was correct for the X-API-KEY scheme.

Knowledge Required and Created

To understand this message, a reader needs several pieces of background knowledge. First, familiarity with HTTP authentication methods—specifically the difference between Authorization: Bearer (standardized in RFC 6750) and custom headers like X-API-KEY. Second, understanding of the Filecoin ecosystem and CIDgravity's role as a deal-making service that matches storage providers with clients. Third, knowledge of the codebase structure: that cidgravity/get_deal_states.go contains the API client implementation, and that rbdeal/deal_tracker.go orchestrates the deal check loop. Fourth, familiarity with command-line debugging tools like curl, ssh, and journalctl.

The message creates several important pieces of knowledge. It confirms that the CIDgravity API at service.cidgravity.com uses the X-API-KEY header for authentication. It validates that the stored token in /home/fgw/.ribswallet/cidg.token is correctly formatted and functional. It demonstrates that the API endpoint get-on-chain-deals responds successfully in approximately 2.6 seconds from the QA environment, returning several megabytes of JSON data. Most importantly, it establishes that the authentication layer is not the root cause of the timeout issue—since the API responds correctly with the right header, the production timeouts must stem from other factors such as network latency, request volume, or server-side performance.

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic pattern. The first sentence—"I see - the code uses X-API-KEY header (line 77), not Authorization: Bearer"—is a moment of insight, connecting the observed 401 error to the discovered source code. The assistant does not merely note this fact; it immediately formulates a hypothesis: if the correct header is used, the API will respond successfully.

The next action is to test this hypothesis. The assistant constructs a curl command that mirrors exactly what the application does: same endpoint, same content type, same header name, same token source. The --max-time 30 flag matches the application's timeout, ensuring the test replicates production conditions. The -v (verbose) flag captures SSL certificate details, which serves double duty: it confirms the TLS handshake completes successfully and provides forensic evidence of the connection path.

The output shown in the message is the SSL certificate chain from Let's Encrypt, indicating the TLS handshake succeeded and the HTTP request was sent. The truncation at "SSL c..." is an artifact of the tail -40 command, but the critical information—that the connection progressed past TLS negotiation—is visible. The full result, revealed in the next message, confirms the hypothesis: 200 OK, 2.6 seconds, 5.3 MB of data.

Broader Significance

This message exemplifies a fundamental debugging principle: when an external API behaves unexpectedly, verify your assumptions against the actual code rather than guessing. The assistant could have spent hours trying different authentication methods, tweaking timeout values, or investigating network configuration. Instead, a two-minute source code reading identified the exact issue.

The message also highlights the importance of integration testing. The application code was correct—it had been sending X-API-KEY all along. But the manual debugging test used Authorization: Bearer, creating a false negative that could have misled the investigation. By reading the code before running the test, the assistant avoided this trap.

Finally, the message demonstrates how a single debugging step can reframe an entire investigation. Before this message, the problem appeared to be a network timeout. After this message, the problem was narrowed to a specific API behavior—the endpoint works with the correct header, so the production timeout must have another cause. This reframing allowed the assistant to move forward with deployment while the team investigated the remaining timeout issue separately.