From Dead Code to Dead Endpoints: A Distributed Storage Debugging Odyssey

Introduction

In the practice of building and operating distributed systems, the boundary between development and operations is not a line — it is a fog. Code changes ripple into deployment configurations; deployment configurations collide with filesystem permissions; filesystem permissions interact with external API availability; and external APIs fail for reasons that no amount of local debugging can reveal. The engineer working in this fog must hold multiple threads simultaneously, switching between the role of code archaeologist, systems architect, production debugger, and infrastructure operator — sometimes within the span of a single session.

Segment 12 of the Filecoin Gateway (FGW) coding session is a case study in exactly this kind of multi-threaded production engineering. Over the course of roughly 150 messages, an AI assistant and a human operator worked together to diagnose a stalled Filecoin deal pipeline, remove a legacy dependency that had been dead weight for months, build and enable a new HTTP-only repair worker subsystem, migrate a Lotus API endpoint away from a rate-limited public gateway, fix a repair staging path that pointed to a read-only partition, and deploy the resulting changes to a two-node QA cluster — only to discover that the new API endpoint was not yet operational.

This article tells the complete story of that session: the decisions made, the code changed, the deployments executed, and the lessons learned when a debugging journey that began with a timeout ended with a connection refused.

The Crisis: A Silent Gatekeeper

The production cluster consisted of two Kuri storage nodes running a horizontally scalable S3-compatible storage system built on IPFS and Filecoin. At the heart of the system's deal management pipeline was the deal tracker loop — a periodic background process responsible for checking on-chain deal states via the CIDgravity API, identifying data groups ready for new Filecoin storage deals, and initiating storage proposals. This loop is the system's lifeline to the Filecoin network: if it stalls, no new deals are made, no repairs are triggered, and the system gradually falls out of sync with the decentralized storage layer it depends on.

When the assistant began investigating, the deal tracker loop was failing consistently. Every cycle produced the same error: Post "https://service.cidgravity.com/private/v1/get-on-chain-deals": context deadline exceeded (Client.Timeout exceeded...). The HTTP client had a 30-second timeout configured, but the API was taking between 110 and 160 seconds to respond. The loop would start, attempt the CIDgravity call, and abort approximately 162 seconds later without ever reaching the deal-making phase.

The architectural root cause was traced to the sequential design of runDealCheckLoop in rbdeal/deal_tracker.go. The function runs three sub-loops in strict order: runCidGravityDealCheckLoop first, then runSPDealCheckLoop, then runDealCheckCleanupLoop (which actually proposes new deals). If the CIDgravity check fails — for any reason — the entire loop returns an error and the deal-making step never executes. The CIDgravity API was acting as a silent gatekeeper, and when it timed out, the entire pipeline stalled.

This design is a textbook example of brittle control flow. A transient failure in an external API — something that should be handled with retries, caching, or graceful degradation — was cascading into a complete halt of the deal-making subsystem. The system was not resilient to the very failure mode it was most likely to encounter.

Thread One: The Lassie Archaeology

Before the assistant could address the CIDgravity timeout, the user intervened with a redirect that would shape the entire session. At message 2147, the user asked: "Is there legacy Lassie/Graphsync in the repair path still? If so we should remove it and also come up with a plan to integrate into the appliance mode."

This question was prompted by the assistant's milestone gap analysis, which had identified the repair worker as the last remaining gap in Milestone 4 (Data Lifecycle Management). The user, with the instincts of an experienced engineer, recognized that before enabling a dormant subsystem, it was worth checking whether it carried dead weight. The Lassie dependency — a library for retrieving data from Filecoin storage providers using the Graphsync protocol — had been part of the codebase since an earlier architectural phase when the system supported dual retrieval paths (HTTP and Graphsync). But the Graphsync path had been abandoned, and the Lassie integration was now little more than a compile-time artifact.

The investigation that followed was a methodical exercise in code archaeology. The assistant began by reading the header of deal_repair.go, where a comment block documented the intended design: "Fetch sector: Http if possible, lassie if not." This comment was a time capsule from an earlier architectural phase, preserving an intention that had long since been abandoned. The assistant then traced the Lassie dependency through every file that referenced it — retr_provider.go, retr_checker.go, deal_diag.go — and discovered that the Lassie integration was almost entirely dead code. The FindCandidates function was defined but never called. The cs slice of types.RetrievalCandidate structs was constructed but never consumed. The Lassie import in go.mod was a compile-time artifact with no runtime significance.

The removal proceeded in careful stages. First, the FindCandidates function was deleted from retr_provider.go, along with its Lassie import and the now-unused go-multiaddr and go-libp2p-core/peer dependencies. Then, the cs variable construction was removed from retr_checker.go, and the types.RetrievalCandidate usage was replaced with a simpler direct call. The fetchGroupLassie stub in deal_repair.go — which already contained the line err = errors.New("lassie is gone") — was removed entirely. Finally, go mod tidy was run to purge Lassie from go.mod and go.sum, and the build was verified to compile cleanly.

Throughout this process, the assistant maintained a disciplined verification cadence. After each batch of edits, a build command was run to confirm compilation. When the build failed due to a filesystem issue — the Docker-based build environment had a different module cache than the local filesystem — the assistant adapted by using the correct Go module path. This attention to the build pipeline, not just the source code, is a hallmark of professional engineering practice. By message 2207, the Lassie removal was complete. The dependency was gone, the dead code was removed, and the codebase was ready for the next phase.

Thread Two: Building HTTP-Only Repair Workers

With the Lassie dependency cleared, the assistant pivoted from archaeology to construction. The repair workers — background goroutines responsible for fetching data from Filecoin storage providers when local copies fall below a retrievability threshold — had been disabled for an unknown period, commented out in ribs.go with the note: "XXX: no repair worker for now, we don't have a staging area to repair to."

The user had provided clear guidance at message 2156: remove the Lassie dependency, enable repair workers when a staging path exists (which should always be the case), and default to four workers. The assistant translated this guidance into code.

The centerpiece of this effort was a complete rewrite of deal_repair.go. The old file was entirely commented out — a skeletal design document rather than executable code. The new implementation implemented HTTP-only group retrieval from storage providers, with PieceCID (CommP) verification to ensure data integrity before re-importing into local storage. The retrieval path was simplified to a single protocol: booster-http, the HTTP-based retrieval protocol exposed by Filecoin storage providers. No more Lassie, no more Graphsync, no more dual-path complexity. The repair path was now purely HTTP-based, with integrity verification built in.

The assistant then wired the repair workers into the startup path. In ribs.go, the commented-out goroutine launches were replaced with a single call to r.startRepairWorkers(ctx), which conditionally starts workers based on whether the staging path is configured. The configuration layer was extended with RIBS_REPAIR_WORKERS and RIBS_REPAIR_STAGING_PATH environment variables, with sensible defaults.

But code changes alone are not enough in a production system managed by Ansible. The assistant extended the Ansible role for Kuri nodes with the new configuration variables, added a directory creation task to ensure the repair staging path exists with correct permissions, and verified that the systemd service template's ReadWritePaths covered the new directory. This full-stack thinking — from Go source to Ansible template to systemd unit — is what separates a feature that works in development from one that works in production.

The commit at message 2233 captured the culmination of both threads: "feat: enable HTTP-only repair workers and remove Lassie dependency." The diff showed 207 insertions and 478 deletions — a net simplification of the codebase. The repair workers were alive.

Thread Three: The CIDgravity Timeout

The third thread began when the user, seeing the assistant preparing to deploy the new binary, intervened with a critical redirect: "First — look at logs and see why rbdeal deals are not flowing, then deploy." This seven-word message halted the deployment momentum and reoriented the session toward diagnosis.

The assistant's investigation revealed the CIDgravity timeout issue in full detail. The assistant tested the API directly from the Kuri node using curl. The first attempt used Authorization: Bearer and received a 401 Unauthorized — "No API key found in request." By reading the source code in cidgravity/get_deal_states.go, the assistant discovered that the API expected an X-API-KEY header instead. This subtle authentication mismatch was a critical finding. With the correct header, the API responded with HTTP 200 in approximately 2.6 seconds, returning 5.3 MB of deal state data.

This created a puzzle. The API worked from the command line in 2.6 seconds, but the Go service was timing out at 30 seconds. The assistant hypothesized that the discrepancy might be due to connection reuse issues, semaphore contention (the code used a weighted semaphore to limit concurrent CIDgravity requests to 4), or some other state accumulated over the three days the service had been running.

The decision was made to deploy the new binary (with repair workers and Lassie removal) and restart both Kuri nodes, in the hope that clearing the process state would resolve the timeout. The deployment succeeded — systemctl status confirmed both services were active (running). But the post-deployment checks revealed two new issues that would dominate the remainder of the session.

The Two Production Failures

After restarting the Kuri nodes with the corrected binary, the assistant observed fresh startup logs. Two new problems emerged simultaneously.

Issue 1: 429 Too Many Requests from the Lotus Gateway. The wallet balance manager was receiving HTTP 429 (Too Many Requests) responses from api.chain.love, the public Lotus gateway used for Filecoin blockchain queries. This was rate-limiting the wallet balance checks that the deal tracker needed to function. Even if the CIDgravity timeout could be resolved, the deal pipeline would still be blocked by the Lotus gateway's rate limiting.

Issue 2: Repair Staging Path Error. The repair worker subsystem was failing at startup because it tried to create a staging directory at /data/repair-staging. This path fell outside the writable /data/fgw/ partition. The root filesystem was read-only for the service user, causing an immediate startup error that prevented repair workers from launching.

The assistant's diagnosis was crisp and actionable: two bullet points that cut through hours of debugging to identify the root causes blocking the system. The assistant immediately gathered configuration data, running ssh commands to check the current settings. The output revealed that RIBS_REPAIR_STAGING_PATH was not explicitly set in the environment file, and RIBS_DATA was /data/fgw — confirming that the default path was being used and that it pointed to a read-only location.

The User's Directive: Two Sentences That Changed Everything

The user's response was concise and authoritative: "API: use pac-l-gw.devtty.eu, also set that as default in the gateway. repair-staging put in ribsdata yes."

This single 24-word message contained two precise directives that would trigger a cascade of code changes, configuration updates, binary rebuilds, and redeployments across the two-node QA cluster. The first directive — switching the Lotus API endpoint — addressed the rate-limiting issue. The second — placing the repair staging path under RIBS_DATA — addressed the read-only filesystem error.

The user's message reveals a sophisticated mental model of the system architecture. The first directive reflects an understanding that api.chain.love was a public, shared resource that was rate-limiting the cluster's requests, and that a dedicated gateway (pac-l-gw.devtty.eu) existed as a replacement. The phrase "set that as default in the gateway" is particularly significant — it tells the assistant not merely to update the runtime configuration on the two QA nodes, but to change the compiled-in default in the Go source code itself. This is a design decision: the new endpoint should be the canonical default for all future deployments.

The second directive is equally deliberate. The user recognized that the repair staging path default was hardcoded to an absolute path (/data/repair-staging) that happened to be on a read-only partition. Rather than suggesting a different absolute path, the user instructed the assistant to resolve it relative to RIBS_DATA — the configurable data directory that was already set to /data/fgw (a writable partition).

The Code Changes: Three Layers of a Fix

The assistant executed the user's instructions with methodical thoroughness, making changes across three distinct layers of the system.

Layer 1: The Code Default. In configuration/config.go, the assistant located the FilecoinApiEndpoint field, which had a default value of "https://api.chain.love/rpc/v1", and changed it to "https://pac-l-gw.devtty.eu/rpc/v1". This was a single-line string replacement in an envconfig struct tag, but its implications were significant — every future build of the Kuri binary would use the new endpoint unless explicitly overridden by environment variables.

The assistant's approach was deliberate. Before making the edit, it first ran a grep to locate the exact line, then read the file with [read] to see the code in its full context. This is a pattern familiar to any experienced developer: look before you leap. When making a surgical change to a production system, especially to a configuration default that affects every deployment, the cost of a mistake is high. Reading the file first is an investment of a few seconds that can save hours of debugging a broken build.

Layer 2: The Runtime Path Resolution. The repair staging path fix was more complex than a simple string replacement. The assistant read rbdeal/deal_repair.go to understand how the staging path was resolved. The code showed that startRepairWorkers() checked cfg.Ribs.RepairStagingPath and, if it was empty, disabled the workers entirely with a log message. If it was set to an absolute path like /data/repair-staging, the code would try to create it — and fail on the read-only partition.

The assistant's initial approach was to modify startRepairWorkers() to resolve the path relative to RIBS_DATA when the configuration value was empty. But then, while reading more of the repair code, the assistant realized that fetchGroupForRepair() also needed the staging path. Simply fixing the path resolution in one function would leave the other still pointing at the wrong location.

This triggered a moment of architectural recognition. The assistant ran a grep on rbdeal/ribs.go and discovered that the ribs struct already had a field called repairDir, initialized in the constructor as filepath.Join(root, "repair"), where root was the RIBS_DATA path. This field existed but was being bypassed by the repair worker code, which read from the configuration directly instead of using the struct field.

The assistant's response was a textbook refactoring: instead of duplicating path-resolution logic in multiple functions, it updated the repair code to use r.repairDir — the field that was already correctly computed from RIBS_DATA. This change rippled through both startRepairWorkers() and fetchGroupForRepair(), ensuring consistency and eliminating the read-only filesystem error.

Layer 3: The Deployment Configuration. The assistant recognized that changing the default in source code was not sufficient. The QA cluster was deployed via Ansible, and the Ansible inventory file at ansible/inventory/qa/group_vars/all.yml contained its own override for the endpoint. If that override was not updated, the deployed systems would continue using the old, rate-limited gateway regardless of the new source default.

The assistant read the Ansible file, confirmed the old endpoint, and applied the edit. This three-layer approach — code default, runtime behavior, deployment configuration — ensured that the fix was complete at every level of the system.

The Deployment: From Theory to Reality

With the code changes complete, the assistant rebuilt the binary using make kuboribs and prepared for deployment. The deployment itself was done manually with scp and ssh commands rather than through the Ansible pipeline. This was a deliberate choice for speed: the deal check loop was stalled in production, and running an Ansible playbook would involve inventory parsing, fact gathering, and orchestration overhead. A direct scp followed by ssh could execute in seconds.

The assistant copied the binary to both nodes, moved it to /opt/fgw/bin/kuri, set execute permissions, and restarted the kuri service via systemctl. The commands used && chaining to ensure that if any step failed, the subsequent steps would not execute — preventing a partial deployment where the service is restarted without the new binary in place.

Before restarting, the assistant also updated the runtime configuration files on both nodes using sed commands over SSH, replacing api.chain.love with pac-l-gw.devtty.eu in /data/fgw/config/settings.env. The sed command used the alternate delimiter | to avoid escaping slashes in the URL — a small but telling detail that shows clear thinking about shell quoting.

The Moment of Verification: Mixed Results

After a 10-second wait, the assistant checked the startup logs on both nodes. The results were mixed.

Good news: The repair workers started successfully. The log showed: starting repair workers {"workers": 4, "stagingPath": "/data/fgw/repair"}. The path resolution fix worked perfectly — the staging directory was now under the writable RIBS_DATA partition, and four repair workers initialized without error. The Lassie dependency was gone, the HTTP-only repair path was operational, and the staging path was correctly resolved.

Bad news: The deal check loop failed again. This time, the error was not a 429 rate-limiting error but a "connection refused" error to pac-l-gw.devtty.eu. The hostname resolved to IP 45.33.141.226, but port 443 was not accepting connections.

The assistant immediately began investigating, testing ports 1234 and 2346 (common ports for Lotus RPC gateways) with curl. All returned "Connection refused." The new gateway endpoint — the one the user had specifically directed the team to use — was simply not operational.

This is the bitter irony at the heart of this debugging session. The assistant had executed every step perfectly: the code default was updated, the runtime path resolution was refactored, the Ansible configuration was aligned, the binary was rebuilt and deployed, and the environment files were patched on both nodes. Yet the system remained broken because the external service it depended on was not running.

The Silence and the Resolution

The conversation captures the moment of recognition. One message is empty — no text, no commands, no output. This silence in the logs speaks volumes. It marks the boundary between "we have a new problem" and "I know why." The assistant had reached the limit of what it could diagnose from its position. It could confirm the hostname resolved and the ports were closed, but it could not know why the gateway was down or when it would return. That knowledge belonged to the user, who operated the infrastructure.

In the very next message, the user writes: "pac-l-gw had an issue, running now." The gateway had a temporary problem and was restored. The empty message was the fulcrum between diagnosis and resolution — the moment when the debugging session pivoted from "our configuration is wrong" to "the external service is down."

Lessons for Distributed Systems Engineering

Several enduring lessons emerge from this session.

First, dead code is a liability. The Lassie dependency had been sitting in go.mod and multiple source files for what appears to be months, imposing a cognitive burden on anyone reading the code and creating the illusion of a capability that did not actually exist. The user's instinct to ask about it — and the assistant's methodical approach to removing it — is a model for dependency hygiene.

Second, configuration defaults are assumptions about the runtime environment. The repair staging path default of /data/repair-staging assumed a writable filesystem layout that did not match the production deployment. The CIDgravity client timeout of 30 seconds assumed an API response time that was empirically false. The Lotus endpoint default of api.chain.love assumed a public gateway that would remain available without rate limits. Every default encodes an assumption about the runtime environment, and those assumptions must be validated against production reality.

Third, the most impactful fixes require changes at multiple levels of the system. A rate-limited API endpoint is not just a code configuration problem; it is a deployment configuration problem, a runtime behavior problem, and potentially a documentation problem. The assistant's systematic approach — code default, runtime path resolution, deployment config — reflected an understanding that robustness comes from consistency across layers.

Fourth, the best code is not the code that solves today's problem most quickly; it is the code that solves today's problem while making tomorrow's problems easier to solve. The assistant's decision to refactor the repair staging path to use the existing repairDir struct field, rather than adding new configuration logic, exemplifies this principle. It eliminated a potential source of inconsistency, reduced cognitive load for future readers, and achieved the user's directive without adding a single new configuration variable.

Fifth, verification is not optional. The assistant's habit of running grep after sed, checking startup logs after restart, and testing endpoints independently is what separates effective operators from those who make changes and walk away. The discipline of "change then verify" is the only defense against the hidden assumptions that plague every production deployment.

Sixth, control flow dependencies create fragility. The sequential design of the deal check loop — where a failure in the CIDgravity check blocks the entire deal-making pipeline — is a textbook example of a brittle architecture. A more resilient design might decouple the CIDgravity state sync from the deal proposal logic, or use a separate timeout for each sub-loop, or cache the CIDgravity response to tolerate transient API slowness.

Finally, configuration changes are only as good as the services they reference. A perfectly implemented code change is worthless if the external dependency it relies on is unavailable. The assistant's careful work — updating defaults, fixing path resolution, rebuilding binaries, deploying across two nodes — was all contingent on the gateway being reachable. The entire deployment cycle was a necessary precondition for discovering that the gateway was down.

Conclusion

Segment 12 of the FGW coding session is a microcosm of distributed systems engineering in practice. It contains moments of careful code archaeology (the Lassie removal), moments of construction (the repair worker rewrite), moments of forensic diagnosis (the CIDgravity timeout), moments of operational triage (the staging path fix), and moments of architectural recognition (the endpoint migration). The three threads — removal, enablement, diagnosis — are not separate stories but a single narrative about the work of making a complex system function correctly in production.

The Lassie dependency is gone. The repair workers are running. The staging path is correctly resolved under RIBS_DATA. The Lotus endpoint has been migrated to a dedicated gateway. And the deal flow — after a brief moment of uncertainty when the new gateway was unreachable — is operational again.

What makes this session instructive is not any single achievement but the pattern of work it exemplifies: diagnose, implement, deploy, verify, iterate. Each cycle reveals the next problem. Each fix is a hypothesis waiting to be tested against production reality. And sometimes, the most important thing an engineer can do is recognize the boundary of their control and wait for the human with the infrastructure knowledge to fill the gap.

The session ends not with a triumphant resolution but with a clear understanding of what remains to be done — which, in production engineering, is often the best outcome one can hope for.## References

[1] "The Three Threads of a Distributed Storage Debugging Session: Lassie Removal, Repair Workers, and CIDgravity Timeouts" — Segment 12, Chunk 0 article. Covers the Lassie code removal, HTTP-only repair worker implementation, and CIDgravity API timeout diagnosis across messages 2147–2267.

[2] "The Endpoint That Wasn't There: A Case Study in Iterative Production Debugging" — Segment 12, Chunk 1 article. Covers the Lotus endpoint migration from api.chain.love to pac-l-gw.devtty.eu, the repair staging path fix, and the discovery that the new gateway was not yet operational.