Production Readiness and Write-Path Optimization: A Systematic Journey Through the Filecoin Gateway

Introduction

The path from a working prototype to a production-ready distributed storage system is rarely a straight line. It involves documentation, observability, configuration tuning, infrastructure simplification, and—when the system is under real workload—the discovery that the performance bottleneck is not where you thought it was. This article chronicles a concentrated burst of work on the Filecoin Gateway (FGW), a horizontally scalable S3-compatible storage system built on the Filecoin network, during which the team executed seven distinct production readiness improvements before pivoting to a deep investigation of write-path performance that culminated in a generation-based sync coalescing design.

The session unfolded in two acts. Act One was a rapid-fire sequence of production enhancements: adding Ansible deployment instructions to the README, integrating CIDGravity connection status and L1/L2 cache metrics into the WebUI, simplifying the Ansible roles by removing unused components, making SQL connection pool limits configurable, enabling parallel writes in the QA environment, and fixing a noisy debug print statement. Act Two began when the user identified that the debug print was a red herring—the real bottleneck was architectural, not cosmetic. This led to a systematic code-reading exercise, a flawed first design for sync coalescing, a sharp-eyed user critique, and ultimately a correct-by-construction generation-based synchronization strategy [1].

This article examines both acts in detail, drawing lessons about production engineering, performance debugging, and concurrent systems design.

Act One: The Seven Production Readiness Improvements

1. Ansible Deployment Documentation

The first accomplishment was adding comprehensive Ansible deployment instructions to the README. The new "Option 3 — Ansible (Multi-Node Clusters)" section provided a quick-start guide, inventory structure, example hosts.yml, a playbook table, and operations commands. This was a significant documentation milestone because it codified the deployment procedure that had been iteratively debugged in earlier sessions. The README now covers three deployment paths: single-node manual setup, multi-node manual setup, and multi-node Ansible deployment—giving operators flexibility based on their infrastructure maturity [1].

2. CIDGravity Connection Status in the WebUI

The team added real-time visibility into the gateway's connection to CIDGravity, the deal-making service that connects the Filecoin Gateway to storage providers. A new CheckStatus() method in cidgravity/status.go validates the API token by calling the get-on-chain-deals endpoint. This status was wired through the RIBSDiag interface, implemented in rbdeal/deal_diag.go, exposed via an RPC endpoint in integrations/web/rpc.go, and rendered in the React frontend as a CIDGravityStatusTile component showing connection status, token validity, and response time [1].

This is a classic observability improvement: operators need to know whether the gateway's connection to its deal-making partner is healthy, and if not, whether the problem is a bad token, a network issue, or a CIDGravity outage. Before this change, diagnosing a CIDGravity connectivity problem required digging through logs or manually curling the API. After this change, it's visible on the status page at a glance.

3. L1/L2 Cache Metrics in the WebUI

The Filecoin Gateway uses a two-tier cache for retrieval from the Filecoin network: L1 is an ARC (Adaptive Replacement Cache) in memory, and L2 is an SSD-based cache. The team added a CacheStats struct to the RIBSDiag interface with fields for both tiers, implemented a CacheStats() method in rbdeal/retr_provider.go that collects statistics from both cache tiers, and rendered the results in a CacheStatsTile component showing hit rate, L1/L2 sizes, item counts, and other metrics [1].

This is another observability win. Cache hit rates are a critical metric for any storage system—they directly affect retrieval latency and the load on the Filecoin network. Having these metrics visible in the UI allows operators to tune cache sizes and understand whether the cache is effective for their workload.

4. Ansible Role Simplification

The team removed several unused Ansible roles that had been part of an earlier deployment strategy but were no longer needed. The deleted roles were:

5. Configurable SQL Connection Pool Limits

The team made SQL connection pool limits configurable through environment variables, adding four new settings to YugabyteSqlConfig:

6. Parallel Writes in QA

The team enabled parallel writes in the QA environment by adding ribs_enable_parallel_writes: "true" and ribs_max_parallel_groups: 2 to the QA group variables, and adding the corresponding environment variable templates to the Kuri role. This was deployed to the two QA nodes (10.1.232.83 and 10.1.232.84) with two parallel groups per node [1].

Parallel writes had been implemented in earlier sessions but had not been enabled in the QA environment. This change allowed the team to test concurrent write workloads in a realistic setting—which would prove crucial for the performance investigation that followed.

7. The Debug Print Fix

The seventh accomplishment was removing a fmt.Println("syncing group", m.id) statement from rbstor/group.go:264. This debug print was writing to stdout on every group sync operation, creating noisy logs and—the team initially believed—causing measurable overhead in the write path [1].

The fix was trivial: a one-line deletion. But as we'll see in Act Two, this was not the real problem.

Act Two: The Performance Investigation

The User's Pivot

The user's response to the debug print fix was brief but transformative: "Print doesn't matter for perf, the sync however should be parallelised - make sure there are parallel write goroutines up to the ribsbs level, and that multiple parallel writes are coalesced into one sync" [5].

This message contained two directives:

  1. Ensure that parallel write goroutines extend all the way to the ribsbs (blockstore) level, not just the rbstor level.
  2. Implement sync coalescing so that multiple concurrent callers to Group.Sync() share a single expensive fsync operation. The user had identified that the real bottleneck was architectural: every batch flush called Group.Sync(), which acquired the exclusive dataLk mutex and performed a jb.Commit() that included an fsync system call. Under concurrent workloads, this created a serialization point where writers queued up behind disk synchronization [5].

Systematic Code Reading

Rather than jumping into implementation, the assistant embarked on a systematic code-reading exercise to map the entire write path. This is a textbook example of how to approach performance debugging: understand the architecture before proposing changes.

The assistant read rbstor/group.go to understand the Sync() method, which acquires dataLk and calls m.sync()—the internal method that performs jb.Commit(). Then it read rbstor/rbs.go to understand the flush logic, discovering that Flush() branches between flushParallel() and flushLegacy() based on configuration. The legacy flush acquires a global mutex (r.r.lk) and iterates over groups sequentially, calling g.Sync(ctx) for each one—the heart of the serialization problem [8].

The assistant then ran targeted grep searches to map the entire sync call graph. Searching for Sync\(ctx found 14 matches across the codebase, revealing sync calls at the Group level, the RBS level, the index sync coordinator, and metered wrappers. A search for batch put functions found only a single match in a test file, confirming that no existing batching mechanism could be reused. The assistant also traced the S3 blockstore layer, searching for .Batch(, .Flush(, and Session( to understand how the S3 frontend connects to the storage engine [7][10].

This systematic tracing produced a complete map of the write path: S3 PUT requests flow through a blockstore that creates batches via ribBatch, which distributes writes across groups using a load balancer, and each group flush eventually calls Group.Sync() with its expensive jb.Commit(). The bottleneck was now fully understood.

The First Design: Coalesced Sync

Armed with this understanding, the assistant proposed a detailed design for coalesced sync. The core idea was straightforward: when a goroutine calls Group.Sync() and a sync is already in progress, the caller should wait for that sync to complete rather than starting a new one. The design added three fields to the Group struct: syncMu (a mutex protecting sync state), syncInFlight (a boolean flag), and syncWaiters (a slice of channels for waiting goroutines) [26].

The algorithm worked as follows: the first caller becomes the "leader," sets syncInFlight = true, acquires dataLk, calls jb.Commit(), releases dataLk, then wakes all waiters by sending the error on each channel. Subsequent callers see syncInFlight == true, create a waiter channel, append it to the list, and block on it with context cancellation support [26].

The assistant also produced a comprehensive test plan with eight test cases covering basic coalescing, concurrent Put+Sync, context cancellation, error propagation, serial sync after coalesced sync, race condition stress testing, integration with ribBatch.Flush(), and mixed parallel/sequential access patterns [26].

The design looked thorough. It looked correct. But it wasn't.

The User's Critique: "This coalesce is not safe tho no?"

In a three-line message, the user punctured the entire design: "This coalesce is not safe tho no? will not account for writes since inital sync started?" [27].

The insight is subtle but devastating. Consider this timeline:

Goroutine A: Put(data1) → Sync() starts ──────────→ Sync() completes
Goroutine B:              Put(data2) → Sync() [waits for A's sync]

Goroutine B's Put(data2) happens after A's sync has started. The sync that A is performing will never include data2 because it was written to the buffer after the sync began. If B simply waits for A's sync to complete and returns success, B will incorrectly believe that data2 has been fsynced to disk when it hasn't [27][28].

This is a classic concurrency pitfall. The assistant's design conflated "a sync is running" with "my writes are included in that sync." These are different predicates, and the design only satisfied the former. The user's ability to spot this instantly came from a deep understanding of the system's semantics: Sync() means "my data is on disk," not "some sync happened" [27].

The Generation-Based Fix

The assistant's response to the user's critique was a detailed walkthrough of the problem, drawing timelines, analyzing lock ordering, and considering edge cases [28]. The fix was to introduce generation-based tracking.

The core idea is to track which writes have been synced to disk using monotonically increasing counters. Each Put() increments a writeGen counter (while holding dataLk). Each completed sync records the generation it synced up to in syncedGen. When a goroutine calls Sync(), it captures the current writeGen as its neededGen—the minimum generation that must be synced before it can safely return [28].

The revised algorithm:

  1. Capture neededGen = atomic.LoadUint64(&m.writeGen) — the generation we need synced.
  2. If syncedGen >= neededGen, return immediately (data is already on disk).
  3. If a sync is in flight, join as a waiter with minGen = neededGen.
  4. If no sync is in flight, become the leader: perform the sync, capture syncedUpTo = atomic.LoadUint64(&m.writeGen), update syncedGen, then wake waiters.
  5. When waking waiters, check each waiter's minGen against syncedGen. If syncedGen >= minGen, the waiter's data is safe. If not, re-queue the waiter for another sync [28]. This transforms the predicate from "is a sync running?" to "has my generation been synced?"—precisely the distinction the user identified.

Lock Ordering and Safety Analysis

The assistant then performed a detailed lock ordering analysis to validate the design. Both Put() and Sync() acquire m.dataLk, making them mutually exclusive at the Group level. This means no new writes can happen during a sync. When the sync completes, syncedUpTo == writeGen at that moment includes all data that was written before the sync acquired dataLk [28].

The assistant considered a subtle race: what if a waiter captures neededGen = N before its own Put completes, but the Put increments writeGen to N+1? The waiter would only require generation N to be synced, but its own write was generation N+1. However, this scenario is impossible because Put holds dataLk and Sync also needs dataLk—they cannot interleave. The Put must complete (releasing dataLk) before Sync can proceed, so by the time Sync captures neededGen, the Put is fully done [28].

This is a crucial insight: the mutual exclusion of dataLk provides a safety guarantee that the generation counter alone cannot. The two mechanisms work together to ensure correctness.

The Test Plan

The assistant's test plan, already comprehensive in the initial design, was refined to cover the generation-based approach. Eight test scenarios were specified:

  1. Basic coalescing: N goroutines call Sync() simultaneously; verify all return and sync was called only once.
  2. Concurrent Put + Sync: Multiple goroutines doing Put → Sync in a loop; verify data integrity and coalescing effectiveness.
  3. Context cancellation: Waiters with short context timeout; verify they return context error while the actual sync completes.
  4. Error propagation: Mock sync returns error; all waiters receive the same error.
  5. Serial sync after coalesced sync: New sync request after coalesced sync starts a fresh sync.
  6. Race condition stress test: 100 goroutines hammering Sync(); verify no deadlocks, races, or data corruption (run with -race flag).
  7. Integration with ribBatch.Flush(): Multiple batches flushing to same group; verify coalescing at Group level.
  8. Mixed parallel/sequential access: Some goroutines doing Put only, some doing Sync only, some doing both [26][28]. The plan also included metrics to track coalescing effectiveness: sync_calls_total, sync_actual_total, and sync_waiters_histogram [26].

Lessons in Production Engineering

This session offers several enduring lessons for distributed systems engineering.

Production readiness is multifaceted. It encompasses documentation, observability, configuration flexibility, and infrastructure simplification—not just feature work. The seven accomplishments in Act One were unglamorous but essential. Each one addressed a specific production concern: "Can operators deploy this system without reading the source code?" (Ansible docs), "Can operators tell if the gateway is healthy?" (CIDGravity status, cache metrics), "Can operators tune database behavior without recompiling?" (configurable pool limits), "Is the deployment pipeline as simple as possible?" (role simplification) [1].

The most visible symptom is not always the root cause. The debug print was a red herring that distracted from the real bottleneck. It was visible, it was noisy, and it was easy to fix. But removing it would have had negligible impact on write performance because the real problem was architectural: the lack of sync coalescing under concurrent workloads [4][5]. The lesson is to resist the temptation to fix the most visible symptom and instead invest in understanding the system's behavior under load.

Systematic code reading is a powerful debugging tool. The assistant's methodical tracing of the write path—from the S3 layer through the blockstore, ribBatch, and Group—produced the understanding needed to design a correct solution. The grep commands, the file reads, the lock ordering analysis—these are the tools of disciplined systems engineering [6][7][8][10].

Concurrency design requires explicit reasoning about invariants. The user's three-line critique exposed a fundamental flaw in the initial coalescing design [27]. The generation-based fix made the relationship between writes and syncs explicit, using counters to track what each sync actually covered [28]. The lock ordering analysis validated that the design was correct by construction.

The best designs emerge from iteration and critique. The assistant's first attempt was plausible but wrong. The user's feedback, the assistant's self-correction, and the detailed lock ordering analysis together produced a design that is correct by construction. The generation-based sync coalescing strategy that emerged from this session is a robust solution to a classic performance problem—one that will serve the Filecoin Gateway well under production workloads.

Conclusion

The Filecoin Gateway session documented here represents a microcosm of production engineering in distributed systems. It began with a burst of production readiness improvements—documentation, observability, configuration, simplification—that made the system more deployable and more observable. It then pivoted to a performance investigation that started with a red herring (the debug print) and ended with a fundamental redesign of the write-path synchronization mechanism.

The generation-based sync coalescing design is a textbook solution to a classic problem: how to amortize the cost of expensive I/O operations (fsync) across multiple concurrent callers while maintaining correctness guarantees. The key insight—tracking which writes have been synced using monotonically increasing generations—is simple in retrospect but required a sharp-eyed critique to uncover.

For the Filecoin Gateway, this session marked a significant step toward production readiness. The system is now more observable, more configurable, and better understood. The write path has been analyzed in detail, the bottleneck has been identified, and a correct design has been proposed. The next step is implementation and testing—but the hardest part, understanding the problem and designing the solution, is done.## References

[1] Segment 18 Analyzer Summary — The comprehensive session summary that documented seven production readiness accomplishments and set the stage for the performance investigation.

[4] Message 2882 — The user's initial report of "slow and bursty writes" and the assistant's investigation including goroutine counts, CPU profiles, and the discovery of the debug print.

[5] Message 2882 — The user's pivotal message: "Print doesn't matter for perf, the sync however should be parallelised."

[6] Message 2883 — The assistant's systematic code-reading approach: "Let me understand the current architecture to plan how to parallelize syncs properly."

[7] Message 2884 — The grep search for Sync\(ctx that found 14 matches across the codebase, mapping the sync call graph.

[8] Message 2885 — The reading of rbs.go that revealed the flush branching logic between flushParallel() and flushLegacy().

[10] Message 2888 — The grep search for .Batch(, .Flush(, and Session( to trace the S3-to-storage connection.

[26] Message 2904 — The initial coalesced sync design proposal with its comprehensive test plan.

[27] Message 2905 — The user's three-line critique: "This coalesce is not safe tho no? will not account for writes since inital sync started?"

[28] Message 2906 — The generation-based fix and the detailed lock ordering analysis that validated the design.