Building Production-Grade Distributed Systems: Multi-Tier Caching, Passive Garbage Collection, and the Discipline of Verification

Introduction

This segment of the opencode coding session captures one of the most intense and productive implementation sprints in the entire FGW (Filecoin Gateway) project. Over the course of approximately 100 messages spanning messages 1717 through 1824, the assistant and user completed two major milestones—Persistent Retrieval Caches (Milestone 03) and Data Lifecycle Management (Milestone 04)—then pivoted into a rigorous verification checkpoint against the project's authoritative execution plan before beginning the remaining infrastructure work for Milestone 02 (Enterprise Grade).

What makes this segment particularly noteworthy is not just the algorithmic sophistication of the components built—an L2 SSD cache with SLRU eviction, an access tracker with decaying popularity counters, a DAG-aware prefetch engine, and a complete passive garbage collection system—but the engineering discipline that governed the process. The assistant did not simply write code and move on. It paused for verification, compared its implementation against a 1,004-line specification document, produced a structured gap analysis, and only then proceeded to the next phase of work. This discipline is the hallmark of production-grade software development, and this article examines every facet of how it unfolded.

Milestone 03: The Multi-Tier Retrieval Cache System

The session opens with the assistant receiving a detailed prompt summarizing the state of the project (message 1717). Milestone 03—Persistent Retrieval Caches—was already partially complete: the L1 ARC (Adaptive Replacement Cache) had been built and tested, along with the metrics infrastructure, logging enhancements, and tracing system that comprised the completed portions of Milestone 02. What remained was the L2 SSD cache, the access tracker, the prefetch engine, and the integration work to wire everything together into a coherent multi-tier cache hierarchy.

The L2 SSD Cache with SLRU Eviction

The assistant began by reviewing the existing ARC cache implementation in rbcache/arc.go to understand the interface conventions, then created the L2 SSD cache in rbcache/ssd.go (message 1721). The SSD cache was designed around several key architectural decisions:

SLRU (Segmented LRU) Eviction: Unlike a standard LRU cache, SLRU divides the cache into two segments—a probationary segment occupying 80% of the capacity and a protected segment occupying the remaining 20%. Items start in the probationary segment and are promoted to the protected segment on a second access. This design prevents cache pollution from one-hit-wonders while ensuring that frequently accessed items remain in the cache longer. The probationary segment acts as a filter, admitting only items that demonstrate repeated utility.

Adaptive Admission Policy: The SSD cache does not admit every item that is evicted from the L1 cache. Instead, it implements an admission policy that only admits items with two or more accesses in the L1 cache and a read-to-write ratio that favors reads. This policy ensures that the SSD cache, which has higher latency and limited write endurance compared to DRAM, is reserved for items that are genuinely popular rather than transient noise.

Write Buffering: SSD writes are batched into sequential operations to minimize write amplification and maximize throughput. The cache accumulates write operations in an in-memory buffer and flushes them in sequential batches, aligning with SSD performance characteristics that favor large sequential writes over small random ones.

On-Disk Persistence: The cache maintains an in-memory index for fast lookups while storing data on disk. CRC32 checksums provide data integrity verification, and the index can be saved and reloaded on restart to avoid cold-start penalties. Compaction logic reclaims fragmented space from evicted entries.

The assistant wrote tests for the SSD cache in rbcache/ssd_test.go (message 1722) and verified that all tests passed (message 1723), including tests for basic operations, the admission policy with various access patterns, and the write buffering behavior.

The Access Tracker with Decaying Popularity Counters

With the SSD cache complete, the assistant moved to the access tracker (message 1727). The access tracker, implemented in rbstor/access_tracker.go, provides the observability layer that feeds both the prefetch engine and the cache admission policy. Its design centers on three mechanisms:

Decaying Popularity Counters: Rather than maintaining absolute access counts that grow monotonically, the access tracker uses decaying counters that apply a decay factor (default 0.99) at regular intervals. This design ensures that popularity reflects recent access patterns rather than cumulative historical totals. A group that was popular last week but has not been accessed today will have a lower score than a group that has been accessed repeatedly in the last hour.

Sequential Access Detection: The tracker maintains a ring buffer of recent access events and analyzes the buffer for sequential patterns. When it detects that a client is accessing blocks in a predictable sequence (e.g., block 1, block 2, block 3), it emits a sequential pattern event that the prefetch engine can use to predict and pre-load subsequent blocks.

Hourly Access Patterns: The tracker maintains separate decaying counters for each hour of the day, enabling temporal analysis of access patterns. This data can be used to predict diurnal patterns—for example, pre-loading popular content before peak usage hours.

The initial implementation encountered a metrics registration conflict (message 1730) because Prometheus metrics were being registered with the same names across multiple test invocations. The assistant fixed this by refactoring the metrics to use a singleton registration pattern (messages 1731-1733), then verified that all tests passed (message 1735).

The DAG-Aware Prefetch Engine

The final component of Milestone 03 was the prefetch engine, implemented in rbcache/prefetcher.go (message 1736). The prefetch engine is the most algorithmically sophisticated component of the cache system, combining multiple strategies for predicting future access patterns:

DAG-Aware Prefetching: Content-addressable storage systems like IPFS and Filecoin organize data into Directed Acyclic Graphs (DAGs), where each block contains links to its children. When a block is accessed, the prefetch engine follows these links to schedule prefetch of child blocks, with priority decreasing by depth and link position. This means that the first child of a root block gets the highest prefetch priority, while deeper descendants receive lower priority.

Sequential Pattern Prefetching: When the access tracker detects a sequential access pattern, the prefetch engine schedules prefetch of the predicted next items. The priority of these prefetches decreases with distance from the current position—the immediately next item gets the highest priority, while items further ahead receive lower priority.

Priority Queue Scheduling: Prefetch jobs are managed in a priority queue implemented as a heap. The worker goroutines dequeue jobs in priority order, ensuring that the most important prefetches are executed first. Workers skip items that are already present in either the L1 or L2 cache, avoiding redundant work.

Configurable Parameters: The prefetch engine supports configuration of depth limits (how deep to traverse the DAG), queue size (how many pending prefetches to allow), and worker count (how many concurrent prefetch operations to run).

The initial implementation had a subtle bug in the worker shutdown logic (message 1739): the worker goroutine could block indefinitely on the queue channel even after the stop channel was closed. The assistant fixed this by restructuring the worker loop to check the stop channel more frequently during job processing (message 1741), and all tests passed on the next run (message 1742).

Integration into the Retrieval Provider

With all three components built and tested, the assistant integrated them into rbdeal/retr_provider.go (messages 1750-1763). The integration replaced the existing simple LRU cache with a multi-tier hierarchy:

  1. L1 ARC Cache: The first tier, in DRAM, uses the Adaptive Replacement Cache for scan-resistant caching of frequently accessed blocks.
  2. L2 SSD Cache: The second tier, on SSD, uses SLRU eviction with admission policy for persistent caching of blocks that survive L1 eviction.
  3. Network (Origin): The third tier is the network—Filecoin retrieval or external storage. The cache hierarchy operates with a promotion strategy: an L2 hit promotes the block to L1, ensuring that frequently accessed blocks are served from the fastest tier. The integration also added configuration options to the configuration/config.go file, including cache sizes, policies, paths, and prefetch settings. The assistant committed Milestone 03 in message 1770 with a detailed commit message cataloguing all 10 files changed, 4,488 insertions, and the complete set of new configuration environment variables.

Milestone 04: Passive Garbage Collection and Data Lifecycle Management

With Milestone 03 committed, the assistant immediately pivoted to Milestone 04: Data Lifecycle Management (message 1771). The design philosophy for this milestone was captured in the execution plan's key insight: "Removed/Retired sectors are simply not renewed." The garbage collection strategy is passive—rather than actively deleting data, the system simply stops extending storage claims for groups that have no live references.

Schema Migrations for Reverse Indices

The first task was to create the database schema migrations that would enable efficient garbage collection. The assistant used an agent to explore the existing migration structure (message 1772), discovering the CQL migrations directory at database/cqldb/migrations/ and the SQL migrations directory at database/sqldb/migrations/.

The CQL migration (database/cqldb/migrations/1769890615_gc_index.up.cql, message 1775) created three tables:

Reference Counting Implementation

The reference counter (rbstor/refcount.go, message 1776) provides the mechanism for tracking live block references. Its design centers on batched operations for efficiency:

IncrementRefs: Called when an S3 object is created. The method walks the DAG of the object's root CID and increments the reference count for every reachable block. Rather than issuing individual database updates, it accumulates increments in a batch buffer.

DecrementRefs: Called when an S3 object is deleted. The method walks the DAG and decrements reference counts, again using batched updates.

Background Flush: A periodic ticker flushes accumulated batches to the database, ensuring that reference counts are eventually consistent while avoiding per-block database overhead.

The batched design is critical for performance because S3 objects can reference thousands of blocks, and issuing individual database operations for each block would be prohibitively expensive.

The Passive Garbage Collector

The garbage collector (rbdeal/gc.go, message 1777) implements the passive GC algorithm. Its design reflects the principle that GC should be safe, conservative, and non-destructive:

Two-Phase Marking: Groups progress through GC states in two phases. First, they are marked as "candidates" when they have zero live blocks and meet the minimum age requirement. After a configurable grace period (default 24 hours), they are promoted to "confirmed" status. This two-phase approach prevents race conditions with ongoing uploads and provides a safety window for recovery.

Configurable Thresholds: The GC supports configuration of scan interval (how often to check for dead groups), grace period (time before confirmation), and minimum group age (preventing GC of recently created groups).

Claim Extender Integration: The claim extender (rbdeal/claim_extender.go, message 1788) was modified to query groups marked for GC and skip claim extension for those groups. This is the core of the passive GC strategy—rather than actively deleting data, the system simply stops paying for its storage on Filecoin. The data remains accessible until existing claims expire, after which it naturally becomes unavailable.

The initial implementation of the GC had several type errors (messages 1777-1784) because the assistant initially used incorrect type names for the database interfaces. These were resolved by examining the actual type definitions in rbdeal/deal_db.go and database/sqldb/db.go and adjusting the code to use the correct method signatures.

Repair Worker Configuration

The final component of Milestone 04 was enabling the repair worker infrastructure. The existing repair system in rbdeal/deal_repair.go was fully implemented but disabled due to missing staging area configuration. The assistant added configuration options for repair workers (enabled flag, worker count, staging path) to the configuration file (message 1794) and ensured that the repair worker startup code in rbdeal/ribs.go would be triggered when enabled.

The assistant committed Milestone 04 in message 1799 with a detailed commit message cataloguing 6 files changed, 889 insertions, and the complete set of GC and repair configuration options.

The Verification Checkpoint

With both milestones committed, the assistant produced a comprehensive summary in message 1802—a structured retrospective cataloguing every component, file, and configuration option. This summary served as both a status update and a cognitive closure mechanism, signaling that the intense feature-development phase was complete.

But the user was not content to simply accept the summary. In message 1803, the user issued a directive that would reshape the remainder of the session: "@milestone-execution.md Verify work done so far, plan out any remaining work." Attached to this brief command was the full content of the 1,004-line execution plan document.

This message is a masterclass in engineering discipline. The user was not asking for a vague status update; they were demanding accountability against a concrete, pre-existing plan. The milestone document specified not just high-level features but concrete file paths, function signatures, configuration parameters, and even SQL/CQL schema definitions. The verification request forced the assistant to compare its implementation against this authoritative reference, identifying any gaps or deviations.

The assistant's response in message 1804 was a meticulously structured "Milestone Verification & Remaining Work Analysis." It presented Milestones 03 and 04 as fully complete with tables listing every component, its status, and its file location. Then it pivoted to Milestone 02 (Enterprise Grade), which was marked as "PARTIALLY COMPLETE." The analysis enumerated seven already-implemented components—deal pipeline metrics, financial metrics, database metrics, S3 frontend metrics, JSON logging configuration, correlation IDs/tracing, and a backup config struct—and then listed everything that remained: Loki and Promtail Ansible roles, wallet backup and YugabyteDB backup roles, backup playbooks and scripts, five Grafana dashboards, Prometheus recording rules, operational runbooks, API documentation, and an entire AI Support System with knowledge base, vector database, and LLM integration.

The analysis concluded with five specific questions about priority order, Ansible scope, dashboard depth, runbook format, and commit strategy. Each question offered concrete options, framing the decision space for the user.

The Directive That Changed the Cadence

The user's response in message 1805 was succinct and unambiguous: "Complete everything in order." Four words. No elaboration. No answers to the five detailed questions. Yet this brief utterance carried more decision-making weight than many paragraphs-long specifications.

This response operated on multiple levels simultaneously. At its most literal, "in order" invoked the existing execution plan, which already encoded priorities and sequencing. By not engaging with the individual questions, the user was making a powerful statement about trust and delegation—signaling that the assistant's judgment about implementation depth, tooling choices, and organizational strategy was sufficient. The user was choosing to operate at a higher level of abstraction, leaving tactical decisions to the agent while retaining strategic direction.

The assistant's response in message 1806 was the operational translation of that directive. It deployed a todowrite command to instantiate a structured task list with the first item—"M02: Loki/Promtail Ansible roles"—set to "in_progress" and the rest to "pending." This message marked the pivot point between planning and execution, between analysis and action.

The Transition to Infrastructure

What followed was the beginning of a methodically executed infrastructure buildout. The assistant did not dive directly into file creation. Instead, it began with reconnaissance—surveying the existing Ansible directory structure to understand the conventions already established in the project. Messages 1807 and 1808 contain nothing more than ls -la commands and their output. But these directory listings represent a deliberate, disciplined approach to infrastructure development. The assistant needed to know what roles already existed, what directory structure they followed, and what naming conventions were used. This reconnaissance prevented creating duplicate roles, violating established conventions, or building on incorrect assumptions.

The assistant then began creating the Loki and Promtail Ansible roles, following the pattern established by the existing roles. The ordering—defaults, tasks, templates, handlers—reflects a logical dependency chain that mirrors the dependencies of the deployed services themselves.

What This Segment Reveals About Engineering Practice

This segment of the session reveals several profound truths about production-grade software development.

First, algorithmic sophistication must be paired with operational discipline. The L2 SSD cache with SLRU eviction and the passive garbage collector are algorithmically sophisticated components that directly impact system performance and correctness. But the assistant did not stop at building them—it paused for verification against the execution plan, identified remaining gaps, and only then proceeded to the next phase. This verification checkpoint prevented the common pitfall of assuming that "code compiles" means "milestone is done."

Second, the most important infrastructure is often the quietest. The cache algorithms and garbage collection strategies are the headline features of this segment. But the verification checkpoint—the pause to compare against the execution plan—is arguably the most important engineering practice demonstrated. Without it, the assistant might have moved on to new feature work while critical operational gaps remained unfilled.

Third, trust is earned through methodical execution. The user's directive to "Complete everything in order" was a vote of confidence in the assistant's judgment. But that trust was built through the assistant's consistent demonstration of thoroughness—the reconnaissance before writing, the dependency-aware ordering of components, the adherence to established conventions, and the clear communication of progress through structured todo updates.

Fourth, the boundary between "feature work" and "infrastructure work" is porous. The same assistant that wrote the ARC cache algorithm and the garbage collector also wrote the CQL schema migrations and the systemd service templates. In a production-grade system, there is no sharp division between "interesting" and "mundane" work. Every component—from the cache eviction policy to the database migration—must be built with the same attention to detail and operational awareness.

Conclusion

This segment of the coding session captures a complete arc of engineering work: the completion of two sophisticated feature milestones, a deliberate verification pause, a strategic reorientation based on user direction, and the beginning of a methodical infrastructure buildout. The assistant navigated from cache algorithms to garbage collection, from schema migrations to Ansible roles, with the same disciplined approach throughout.

The arc demonstrates that effective AI-assisted software development is not just about generating code faster. It is about understanding when to build novel features and when to pause for verification, when to follow established patterns and when to establish new ones. The quiet moments—the ls -la commands, the todowrite updates, the verification analysis—are as essential to the final result as the sophisticated algorithms that preceded them.

In the end, what makes a system production-ready is not just the cleverness of its cache policies or the elegance of its garbage collection strategy. It is the sum of all the small, correct decisions about how the system will be built, verified, deployed, and maintained. This segment of the session is a case study in making those decisions well.