From Architecture Blueprint to Production Readiness: The Complete Filecoin Gateway Engineering Journey

Introduction

This is the story of a distributed storage system's evolution from an architectural blueprint to a production-ready deployment, told across nineteen segments of intensive engineering work. The Filecoin Gateway (FGW) is a horizontally scalable S3-compatible storage system built on the Filecoin decentralized storage network, designed to bridge the gap between standard cloud storage APIs and the decentralized Filecoin market. Over the course of this session, the system underwent a transformation that touched every layer: architecture, infrastructure, observability, data lifecycle management, deployment automation, testing, and performance optimization.

What makes this journey remarkable is not just the breadth of work accomplished—the distributed S3 architecture, the monitoring dashboard, the Ansible deployment pipeline, the multi-tier caching system, the garbage collection engine, the enterprise-grade observability stack, the AI support system, the QA cluster deployment, the 164 new unit tests, the documentation overhaul, and the write-path optimization—but the patterns of engineering discipline that emerge across every phase. The same systematic debugging methodology that resolved a missing database column in the test cluster also uncovered a missing removeUnsealedCopy field in the CIDgravity API. The same iterative refinement that produced a working Ansible pipeline also produced a generation-based sync coalescing design. The same commitment to verification that validated the enterprise dashboards also drove the creation of 164 unit tests.

This article synthesizes the entire session into a coherent narrative, organized by phase of work rather than chronological order, to reveal the underlying engineering principles that guided the system from concept to production readiness.

Phase 1: The Distributed S3 Architecture (Segments 0–5)

The Architecture That Sparked It All

The session began with an architecture specification that was remarkable for its density of design thinking [1]. The requirements were precise: horizontally scalable S3-compatible storage, stateless frontend proxy nodes routing to independent Kuri storage nodes, a shared YCQL database tracking object placement, round-robin writes with directed reads, multipart uploads assembled from parts distributed across multiple nodes, and a read-after-write guarantee. This was not a request for incremental improvement—it was a demand for architectural transformation.

The assistant responded by creating a comprehensive scalable-roadmap.md document and immediately beginning implementation. Phase 1 modified the existing Kuri node codebase to support node identification, adding NodeID and ExpiresAt fields to the S3Object interface and propagating them through the CQL persistence layer, bucket operations, and dependency injection. Phase 2 created an entirely new S3 frontend proxy package (server/s3frontend/) with a stateless HTTP server, round-robin request distribution, and a backend pool with health checking. Phase 3 began implementing YCQL-based read routing, enabling the frontend to query the shared database for object location before directing GET requests [1].

The Dashboard That Revealed Nothing

While the architecture was being built, the user requested a monitoring dashboard—a live cluster overview with performance charts, topology diagrams, and real-time metrics [2]. The assistant designed and implemented a comprehensive React-based monitoring UI with components for cluster topology, throughput charts, latency distributions, error rates, active requests, and data flow visualization. The design specified five new backend RPC methods and a tiered polling strategy ranging from 100ms for active requests to 5s for topology data.

But the dashboard had a fundamental problem: there was no cluster to monitor. The beautiful instrument panel had no engine to connect it to. This gap between observability and infrastructure would become a recurring theme—you cannot monitor what you cannot deploy.

The Test Cluster That Wouldn't Run

The user's practical question—"How do I run a test cluster with 2x2 nodes?"—triggered the construction of a Docker Compose-based test infrastructure [2][3]. What followed was a cascade of debugging that revealed the gap between architectural design and operational reality.

The first failure was a missing database: the Kuri nodes crashed because the PostgreSQL database filecoingw had never been created. The fix was a db-init service that ran after YugabyteDB started. But then the db-init container failed on subsequent runs because the database already existed—an idempotency problem fixed with error suppression. Then the startup script hung because docker-compose ps doesn't show exited containers by default—fixed with the -a flag. Then the Kuri nodes failed with "no external module configured" because CAR file staging was not configured.

Each fix revealed the next problem. The permission errors on YugabyteDB data files revealed the database idempotency issue. The database fix revealed the container detection bug. The container detection fix revealed the CAR staging configuration error. And that error revealed the fundamental architectural misunderstanding [3].

The Architecture Correction

The user looked at the test cluster configuration and recognized that it violated the documented architecture. The assistant had configured Kuri nodes to expose S3 APIs directly, with a single shared configuration file, treating them as public-facing S3 endpoints rather than internal storage nodes. The scalable-roadmap.md clearly specified that S3 frontend proxies are a separate stateless node type, and Kuri nodes are storage backends with independent configurations [3].

The correction was comprehensive. The assistant created gen-config.sh to generate separate settings.env files per node, each with distinct EXTERNAL_LOCALWEB_URL values and port allocations. The Docker Compose file was restructured into a proper three-layer hierarchy: S3 proxy on port 8078 → Kuri storage nodes internally → shared YugabyteDB. The routing layer was implemented as specified in the roadmap, completing Phase 3 [3].

This moment crystallized a lesson that would echo through the entire session: configuration is architecture. The Docker Compose file was not just a developer convenience—it was the concrete expression of the system's topology. Getting the configuration wrong meant building a different system than the one described in the roadmap.

Keyspace Segregation and Dual CQL Connections

The next major challenge was database contention. When both Kuri nodes ran simultaneously, they deadlocked on shared group resources because they were using the same database keyspace [4]. The solution was to segregate database keyspaces: each Kuri node gets its own filecoingw_{node_id} keyspace for deals, groups, and blockstore data, while a shared filecoingw_s3 keyspace handles object routing metadata.

This required a significant architectural change: each Kuri node needed two separate CQL connections—one for its per-node RIBS keyspace and one for the shared S3 metadata keyspace. The assistant added a S3CqlConfig to the configuration system, created a S3CqlDB wrapper type, and wired dual CQL connections through the Kuri plugin's dependency injection. All changes were staged into 14 logical git commits covering configuration, interfaces, the Kuri S3 plugin, dual CQL connections, the S3 frontend proxy package, build system, test cluster infrastructure, documentation, CQL schema migrations, and endpoint fixes [4].

Debugging the Monitoring Dashboard

With the test cluster running, the assistant turned to debugging the monitoring dashboard. Several issues emerged: a Go 1.22 HTTP route conflict between HEAD / and GET /healthz required replacing the standard ServeMux with a custom handler. The web UI container needed an Nginx reverse proxy. The S3 proxy returned internal server errors because the S3Objects table lacked the node_id column—fixed by updating the schema and db-init script [5].

The assistant then implemented real cluster monitoring metrics. A ClusterMetrics collector tracked throughput, latency, error rates, active requests, and I/O bytes with a rolling 10-minute window. The ClusterTopology RPC was upgraded to parse FGW_BACKEND_NODES and perform health checks. The frontend was enhanced to visually distinguish S3 frontend proxies (blue) from Kuri storage nodes (green), with new I/O throughput charts and latency components. The user requested renaming "SLA" to "SLO" with a 350ms threshold, which was applied and verified [5].

Performance Optimization: Data Generator and CQL Batcher

With the cluster operational, the focus shifted to performance. The load test data generator was analyzed through benchmarks, revealing three performance tiers: WithMD5 at ~700–800 MB/s (MD5 bottleneck), DataOnly with allocation at ~3–6.5 GB/s, and FillBuffer without allocation at ~50–85 GB/s [6]. The assistant updated the loadtest worker to use pre-allocated buffers, bypassing both the MD5 bottleneck and allocation overhead.

The next performance investigation revealed that apparent data corruption during load testing was actually context deadline timeouts—no real corruption existed [7]. The assistant then implemented a CQLBatcher in the database/cqldb package that collects individual CQL INSERT calls and flushes them in batches (default 15,000 entries or within 10–30ms), using a worker pool with exponential backoff retries. This batcher was integrated into ObjectIndexCql.Put() and significantly improved write throughput.

Docker networking was also optimized. Host networking mode was attempted to eliminate the Docker userland proxy bottleneck, but port conflicts with existing services forced a revert to bridge networking. The test cluster was stabilized with proper configuration, the loadtest header fix (x-amz-content-sha256: UNSIGNED-PAYLOAD), and all improvements committed [8].

Phase 2: Deployment Automation (Segments 6–7)

The Ansible Pipeline That Wouldn't Deploy

With the test cluster working, the team turned to production deployment automation. The assistant committed Ansible deployment scripts for FGW clusters—7 roles, 5 playbooks, and inventory structure—and created a comprehensive Docker-based test harness to validate them [9].

The test environment included a YugabyteDB container, three target hosts running Ubuntu 24.04 with systemd and SSH, and an Ansible controller container. Running the test suite revealed a cascade of failures that required iterative debugging:

The Pipeline That Still Wouldn't Deploy

The second round of Ansible debugging revealed an even deeper set of issues [10]:

Phase 3: Enterprise Features (Segments 8–10)

From Infrastructure to Research-Driven Planning

With the validated deployment pipeline complete, the session pivoted from execution to planning [11]. The user launched multiple research agents to investigate state-of-the-art approaches for three ambitious milestones:

Multi-Tier Caching and Passive Garbage Collection

The implementation of Milestones 03 and 04 was completed in a concentrated burst of engineering [12]. For Milestone 03, the assistant built:

Enterprise-Grade Observability and Operations

Milestone 02 was finalized with a comprehensive set of enterprise features [13]. The assistant built:

Phase 4: Production Deployment (Segments 11–14)

The QA Cluster on Physical Nodes

The enterprise features were validated, but the system had never been deployed on physical hardware. The next phase addressed this gap by deploying a QA test cluster across three physical nodes: a head node (10.1.232.82) and two Kuri nodes (10.1.232.83, 10.1.232.84) [14].

The assistant built the deployment from scratch using Ansible, installing a single-node YugabyteDB, creating both SQL and CQL keyspaces, building and deploying kuri and s3-proxy binaries, and securing sensitive credentials (CIDgravity token, wallet files) in restricted-access vault files rather than plaintext. When the user flagged that secrets must not be stored in plaintext, the assistant corrected by placing the token in a separate restricted file and loading it at runtime via ExecStartPre in the systemd service [14].

The first major obstacle was "dirty migration" states in the YugabyteDB CQL keyspaces. Previous test runs had left migration flags set to dirty = true in the schema_migrations table, preventing the kuri daemons from starting. The assistant manually corrected these flags to false, enabling both storage nodes to launch successfully [14].

The second obstacle was cross-node S3 reads. Each Kuri node could only serve data from its local blockstore, not from the peer node. The assistant resolved this by deploying the s3-proxy frontend on the head node, configuring it with all backend node addresses, and routing all S3 traffic through the proxy, which correctly dispatched requests to the owning node based on stored object metadata [14].

Debugging the Deal Pipeline

With the QA cluster operational, the team turned to the deal-making pipeline—the process by which the Filecoin Gateway creates storage deals with Filecoin storage providers. The first issue was CIDgravity API timeouts preventing new deals [15]. The assistant diagnosed that the get-on-chain-deals call was taking 110–160 seconds, exceeding the 30-second HTTP client timeout.

Simultaneously, the assistant cleaned up legacy Lassie/Graphsync retrieval code, removing the Lassie dependency from go.mod and all source files, and rewriting deal_repair.go to implement HTTP-only group retrieval from storage providers with PieceCID verification. Repair workers were enabled in the startup path with Ansible configuration support [15].

The Lotus API endpoint was migrated from api.chain.love (which was rate-limiting) to pac-l-gw.devtty.eu. The repair staging path was fixed to resolve relative to RIBS_DATA instead of defaulting to a read-only partition. After redeployment, repair workers launched successfully, but the CIDgravity deal check failed again because the new gateway endpoint was unreachable [15].

Once the gateway became operational, the assistant discovered that no deals were being made despite a group being ready with ~30GB of data. Through progressive debug logging, the root cause was traced to the CIDgravity GBAP (Get Best Available Providers) call returning NO_PROVIDERS_AVAILABLE because the request lacked the required removeUnsealedCopy field [16].

The Fallback Provider and 164 Tests

The CIDgravity no-providers issue was resolved by implementing a configurable fallback provider mechanism (RIBS_DEAL_FALLBACK_PROVIDERS). When GBAP returns empty, the system now uses a comma-separated list of storage providers. Three of four fallback providers immediately accepted deals for Group 1 [17].

Following this fix, the assistant executed a systematic test coverage initiative. Using multiple subagents working sequentially, the assistant created a detailed testing plan and implemented 164 new unit tests across eight new test files, covering configuration loading, the CIDgravity client, deal repair, fallback provider parsing, S3 authentication (AWS SigV4), S3 request handlers, wallet key management, and robust HTTP retrieval. Several bugs were discovered and fixed during testing, including duplicate Prometheus metrics registration and a CQL migration issue. All tests passed and were committed as 5344f33 [17].

Phase 5: Closing Gaps and Optimization (Segments 15–18)

The Unlink Method and L1-to-L2 Promotion

With the deal pipeline working and tests in place, the assistant addressed critical implementation gaps [18]. The long-stalled Unlink method—left as panic("implement me") in both rbstor/rbs.go and rbstor/group.go—was finally implemented. The implementation removed multihash entries from the CQL index via DropGroup, updated group metadata counters for dead blocks and bytes, and handled the offloaded group case. A dedicated test file verified the full put-unlink-view cycle.

The L1-to-L2 cache promotion callback was also wired up. The assistant modified rbcache/arc.go to add an evictionCallback field and SetEvictionCallback() method, then updated the eviction paths in evictFromT1() and evictFromT2() to call the callback. The callback was wired in retr_provider.go to promote evicted items from the L1 ARC cache to the L2 SSD cache [18].

The Prefetcher's Fetch() method—previously a stub returning an error—was implemented with a proper implementation that leverages the existing cache hierarchy (L1 → L2 → HTTP), using FindHashes via the Storage() interface and getAddrInfoCached for provider URLs [18].

The Documentation Gap

The user asked whether the project README explained how to use Ansible for deployment. The assistant investigated and found that the README lacked any Ansible documentation—it only described manual deployment steps. A comprehensive "Ansible Deployment" section was added, covering inventory configuration, variable customization, playbook targeting, and troubleshooting tips [18][19].

This documentation gap was significant because the Ansible deployment pipeline had been iteratively debugged over multiple sessions, but the knowledge of how to use it existed only in the conversation history and the playbook files themselves. New team members or operators would have had no way to discover the deployment procedure without reading the source code. Closing this gap made the system operationally self-sufficient [19].

The Unseen Cost of a Single Print

The next phase began with a performance investigation. The assistant inspected pprof profiles on a live node and found low CPU usage (6.2%) with no significant contention. However, the logs showed a recurring "syncing group 201" message printed on every group sync. Tracing this to a debug fmt.Println statement in group.go, the assistant removed it, believing this was the performance bottleneck [20].

The user's response was corrective: "Print doesn't matter for perf, the sync however should be parallelised." The real bottleneck was that Group.Sync() was called for every batch and performed an expensive jb.Commit() (with fsync) while serializing on dataLk. Under concurrent workloads, this created a serialization point where writers queued up behind disk synchronization [21].

The Generation-Based Sync Coalescing Design

The assistant embarked on a systematic code-reading exercise to map the entire write path. The investigation revealed that every batch flush called Group.Sync(), which acquired the exclusive dataLk mutex and performed jb.Commit() with an fsync system call. The legacy flush path iterated over groups sequentially, calling g.Sync(ctx) for each one—the heart of the serialization problem [21].

The assistant proposed a coalesced sync design: when a goroutine calls Group.Sync() and a sync is already in progress, the caller should wait for that sync to complete rather than starting a new one. The design added fields to the Group struct for tracking sync state and waiters.

But the user identified a critical flaw: "This coalesce is not safe tho no? will not account for writes since inital sync started?" The insight was that a goroutine waiting for an in-flight sync would return believing its data was on disk, when in fact its writes occurred after the sync began and were never included [21].

The fix was generation-based tracking. Each Put() increments a writeGen counter. Each completed sync records the generation it synced up to in syncedGen. When a goroutine calls Sync(), it captures the current writeGen as its neededGen—the minimum generation that must be synced before it can safely return. Waiters only return once syncedGen >= neededGen. This transforms the predicate from "is a sync running?" to "has my generation been synced?"—precisely the distinction the user identified [21].

The assistant performed a detailed lock ordering analysis to validate the design. Both Put() and Sync() acquire m.dataLk, making them mutually exclusive at the Group level. This means no new writes can happen during a sync, providing a safety guarantee that the generation counter alone cannot. The two mechanisms work together to ensure correctness [21].

Themes Across the Journey

The Verification Culture

Throughout the entire session, the assistant never assumed completion based on file creation alone. Every component was verified: git log confirmed commits, file listings confirmed existence, grep confirmed configuration, test runs confirmed functionality. This layered verification approach is what separates professional infrastructure work from ad-hoc development.

The Architecture Document as Source of Truth

The scalable-roadmap.md document served as the architectural north star throughout the session. When the test cluster configuration drifted from the roadmap, the user's act of referencing the document forced a confrontation between the documented design and the actual implementation. The architecture document was not a static artifact—it was a living contract between design intent and implementation reality.

Systematic Debugging Methodology

Whether debugging a missing database column, a missing API field, or a write-path bottleneck, the assistant followed the same pattern: observe the symptom, trace the call chain, identify the root cause, apply the fix, and verify the result. This methodology was applied consistently across infrastructure bugs (Docker Compose container detection), API integration issues (CIDgravity GBAP parameters), and performance bottlenecks (fsync serialization).

The Human-AI Collaboration Model

The session reveals a sophisticated collaboration model where the AI handles the heavy lifting of code generation, exploration, and implementation, while the human provides the architectural vision, domain expertise, and corrective feedback that keeps the implementation aligned with the design intent. The user's brief messages—"Continue if you have next steps," "Execute recommendations," "This coalesce is not safe tho no?"—carried immense weight because they were built on a foundation of shared context established through detailed planning documents and methodical execution.

Production Readiness Is Multifaceted

The journey from architecture blueprint to production readiness touched every dimension of system engineering: architecture design, infrastructure automation, observability, performance optimization, testing, documentation, and operational procedures. Each phase revealed that production readiness is not a single activity but a layered practice: create, verify, document, automate, test, and verify again.

Conclusion

The Filecoin Gateway journey documented across these nineteen segments represents a microcosm of distributed systems engineering in the modern era. It began with an architectural blueprint for a horizontally scalable S3-compatible storage system and ended with a generation-based sync coalescing design that would eliminate the last major write-path bottleneck. In between, the team built a test cluster, corrected a fundamental architectural error, implemented keyspace segregation, created a monitoring dashboard, optimized performance through batching and pre-allocated buffers, automated deployment with Ansible, built enterprise-grade observability and AI support, deployed on physical hardware, debugged the deal pipeline through multiple layers of failure, wrote 164 unit tests, closed critical implementation gaps, documented the deployment process, and systematically profiled and redesigned the write path.

What makes this journey remarkable is not the volume of work but the consistency of the engineering approach. The same systematic methodology that resolved a missing database column also uncovered a missing API field. The same iterative refinement that produced a working Ansible pipeline also produced a correct-by-construction concurrency design. The same commitment to verification that validated the enterprise dashboards also drove the creation of comprehensive test coverage.

The system that emerged from this journey is fundamentally different from the one that entered it. It is more observable, with dashboards and metrics that reveal its internal state. It is more deployable, with Ansible automation that can provision a multi-node cluster in minutes. It is more robust, with 164 new unit tests and a fallback provider mechanism that ensures deal flow continues even when external APIs fail. It is more performant, with a CQL batcher, pre-allocated buffers, and a generation-based sync coalescing design that eliminates redundant fsyncs. And it is more documented, with runbooks, deployment guides, and a README that makes the system accessible to new operators.

In the end, the Filecoin Gateway is not just a distributed storage system—it is a testament to what disciplined engineering can achieve when architecture, infrastructure, observability, and operations are treated as integrated concerns rather than sequential phases.## References

[1] Segment 0 — The initial architecture specification and implementation of the horizontally scalable S3 architecture, including the scalable-roadmap.md, Phase 1-3 implementation, and the monitoring dashboard design.

[2] Segment 0 — The test cluster infrastructure construction: Docker Compose configuration, shell scripts, database initialization, and the cascade of debugging fixes.

[3] Segment 0 — The architecture correction triggered by the user identifying that Kuri nodes were configured as direct S3 endpoints rather than internal storage nodes behind a stateless proxy layer.

[4] Segment 1 — Keyspace segregation and dual CQL connections: the discovery of database deadlocks when both Kuri nodes shared the same keyspace, and the architectural solution of per-node keyspaces with a shared S3 metadata keyspace.

[5] Segment 2 — Debugging the monitoring dashboard: HTTP route conflicts, Nginx reverse proxy setup, missing CQL columns, real-time metrics implementation, and the SLA-to-SLO rename.

[6] Segment 3 — Performance benchmarking of the data generator, revealing three performance tiers and the optimization to pre-allocated buffers eliminating MD5 and allocation bottlenecks.

[7] Segment 4 — False corruption investigation and CQL batcher implementation: distinguishing timeouts from actual corruption, and the design of a batched write path with worker pool and exponential backoff.

[8] Segment 5 — Test cluster stabilization after host networking port conflicts, bridge network revert, configuration fixes, and the loadtest header fix.

[9] Segment 6 — Ansible deployment automation: 7 roles, 5 playbooks, Docker test harness, and the first round of debugging fixes for health checks, missing packages, and task ordering.

[10] Segment 7 — The second round of Ansible debugging: environment file syntax, log level format, wallet dotfiles, duplicate table creation, and the final successful deployment pipeline.

[11] Segment 8 — The transition from validated infrastructure to research-driven planning for Milestones 02, 03, and 04, with emphasis on efficiency, configurability, and incremental testing.

[12] Segment 9 — Implementation of Milestones 03 and 04: L2 SSD cache with SLRU eviction, access tracker, DAG-aware prefetch engine, passive garbage collection with reverse indices and reference counting.

[13] Segment 10 — Enterprise-grade Milestone 02 completion: Ansible backup roles, five Grafana dashboards, six operational runbooks, AI support system with LangGraph/Ollama, and comprehensive test coverage.

[14] Segment 11 — QA cluster deployment on three physical nodes: YugabyteDB installation, binary deployment, credential vaulting, dirty migration state resolution, and S3 proxy frontend configuration for cross-node reads.

[15] Segment 12 — Deal pipeline debugging: CIDgravity API timeouts, Lassie code removal, HTTP-only repair workers, Lotus endpoint migration, and repair staging path fixes.

[16] Segment 13 — Stalled deal flow investigation: progressive debug logging revealing the missing removeUnsealedCopy field in the CIDgravity GBAP request.

[17] Segment 14 — Fallback provider mechanism implementation and 164 new unit tests across eight test files, with bug fixes discovered during testing.

[18] Segment 15 — Critical implementation gap closure: Unlink method implementation, L1-to-L2 cache promotion callback wiring, Prefetcher Fetch method implementation, and README documentation update.

[19] Segment 16 — Documentation gap confirmation: verification that the README now comprehensively covers Ansible deployment with step-by-step instructions.

[20] Segment 17 — Debug print removal: pprof profiling revealing the fmt.Println statement in the group sync write path, and its removal to eliminate unnecessary I/O.

[21] Segment 18 — Production readiness improvements and write-path optimization: seven production enhancements, systematic code reading of the write path, coalesced sync design, user critique of safety flaw, and generation-based sync coalescing design with lock ordering analysis.