The Art of Letting Go: Building a Passive Garbage Collection System for Filecoin Storage

Introduction

On a terminal somewhere in the development of a horizontally scalable S3-compatible storage system built on Filecoin, a single git commit command sealed the completion of Milestone 04 — a passive garbage collection system for data lifecycle management. The message, at first glance, appears to be a routine commit: a developer running git commit -m with a lengthy description of what was built. But this message is far more than a version control entry. It is the culmination of a carefully reasoned design process, a declaration of architectural philosophy, and the final piece of a two-milestone sprint that transformed the project's approach to data retention, caching, and lifecycle management. To understand this message fully, we must unpack not only what it says but the context in which it was written, the decisions it embodies, and the thinking that led to each line of its detailed commit description.

The Message Itself

The commit was executed as follows:

[assistant] [bash] cd /home/theuser/gw && git commit -m "feat: implement passive garbage collection system (Milestone 04)

Implement data lifecycle management with passive GC strategy:

Schema Migrations:
- database/sqldb/migrations/1769890615_gc_state.up.sql:
  - Add gc_state column to groups (0=active, 1=candidate, 2=confirmed, 3=complete)
  - Add live_blocks/dead_blocks counters for GC decisions
  - Add gc_marked_at timestamp for grace period tracking
  - Add indexes for efficient GC queries

- database/cqldb/migrations/1769890615_gc_index.up.cql:
  - GroupToMultihash: reverse index for O(n) block enumeration per group
  - MultihashRefCount: counter table for reference tracking
  - GCQueue: queue for batched orphan processing

Reference Counting (rbstor/refcount.go):
- Tracks references from S3 objects to blocks
- Batched increments/decrements for efficiency
- Background flush loop with configurable interval
- Prometheus metrics for monitoring

Garbage Collector (rbdeal/gc.go):
- Passive GC: don't extend claims for groups with no live references
- Configurable scan interval, grace period, and min group age
- Two-phase marking: candidate -> confirmed after grace period
- Prometheus metrics for GC operations
- Helper methods for claim extender integration

Claim Extender Integration (rbdeal/claim_extender.go):
- Query groups marked for GC at cycle start
- Build provider+piece -> group mapping
- Skip claims for GC candidate/confirmed groups
- Log skipped claims for visibility

Configuration (configuration/config.go):
- RIBS_GC_ENABLED: enable passive GC (default: false)
- RIBS_GC_SCAN_INTERVAL: scan frequency (default: 1h)
- RIBS_GC_GRACE_PERIOD: time before confirmation (default: 24h)
- RIBS_GC_MIN_GROUP_AGE: minimum group age (default: 168h/1 week)
- RIBS_REPAIR_ENABLED: enable repair workers (default: false)
- RIBS_REPAIR_WORKERS: number of repair workers (default: 4)
- RIBS_REPAIR_STAGING_PATH: staging directory for repairs

Key design decisions:
- Passive GC only: data naturally expires when claims aren't extended
- No active deletion: simpler, safer, no external storage cleanup needed
- Grace period prevents race conditions with ongoing uploads
- Min group age prevents GC of recently created groups"
[pgf-port 39436d5] feat: implement passive garbage collection system (Milestone 04)
 6 files changed, 889 insertions(+), 1 deletion(-)
 create mode 100644 database/cqldb/migrations/1769890615_gc_index.up.cql
 create mode 100644 database/sqldb/migrations/1769890615_gc_state.up.sql
 create mode 100644 rbdeal/gc.go
 create mode 100644 rbstor/refcount.go

Why This Message Was Written: The Motivation and Context

This message did not emerge from a vacuum. It was written as the capstone of a carefully sequenced implementation plan that had been unfolding over multiple sessions. Just moments earlier, the assistant had committed Milestone 03 — a sophisticated multi-tier retrieval cache system encompassing an L1 ARC cache, an L2 SSD cache with SLRU eviction, an access tracker with decaying popularity counters, and a DAG-aware prefetch engine. That milestone addressed the problem of how to serve data faster. Milestone 04, by contrast, addressed the complementary problem of what to stop serving.

The motivation for this message was the recognition that a storage system without data lifecycle management is a ticking time bomb. In a Filecoin-based storage architecture, data is stored in "groups" — collections of blocks that are the subject of storage deals with Filecoin miners. These deals have finite durations and must be periodically extended (via the claim extender) to keep data alive on the network. Without a garbage collection mechanism, every piece of data ever uploaded would be perpetually renewed, consuming storage deals, costing money, and cluttering the system with orphaned blocks that no S3 object references anymore.

The assistant had just completed the caching infrastructure (Milestone 03) and immediately pivoted to lifecycle management (Milestone 04), as indicated by the message at index 1771: "Milestone 03 committed. Now moving to Milestone 04: Data Lifecycle Management. Let me start with GC schema migrations." This rapid transition shows that the two milestones were conceived as a pair — one optimizing the read path, the other cleaning up the write path's long-term consequences.## The Reasoning Behind "Passive GC"

The most striking design decision announced in this commit message is the choice of passive garbage collection. The commit message explicitly states: "Passive GC only: data naturally expires when claims aren't extended." This is a deliberate architectural philosophy, not a technical limitation. Let us examine why.

In a conventional garbage collection system, a collector actively scans memory or storage, identifies unreferenced objects, and reclaims their space. The assistant considered this approach and consciously rejected it. The reasoning, embedded in the commit message's "Key design decisions" section, is: "No active deletion: simpler, safer, no external storage cleanup needed." This is a profound trade-off. Active deletion would require the system to not only identify dead blocks but also to remove them from the underlying storage — which in this architecture includes both local disk caches and potentially Filecoin network storage. Removing data from Filecoin deals is not a simple operation; it involves on-chain transactions, miner coordination, and potential penalty risks. By choosing passive GC, the assistant sidestepped all of that complexity.

The passive approach works by simply not extending the storage deal for groups that have no live references. When a Filecoin storage deal expires naturally, the data is no longer guaranteed to be retrievable from the network. The system does not actively delete anything; it merely allows entropy to take its course. This is elegant in its simplicity but carries an implicit assumption: that the cost of keeping expired data around (in terms of disk space on local caches) is acceptable until the deal naturally expires. The assistant judged this to be a safe assumption, and the commit message reflects that judgment.

The Two-Phase Marking Protocol

Another design decision visible in the commit message is the two-phase marking protocol: candidate → confirmed → complete. The assistant implemented a grace period (default 24 hours) between marking a group as a candidate and confirming it for GC. This is a defense against race conditions. Consider a scenario where an S3 upload is in progress, creating references to blocks in a group. If the GC scanner runs during the upload, it might see zero references and mark the group for collection. The grace period gives the upload time to complete and register its references. The minimum group age (default 168 hours, or one week) adds another layer of protection, preventing the GC from touching groups that were recently created and might still be accumulating references.

This two-phase approach reveals a conservative engineering mindset. The assistant could have implemented a simpler single-pass system — scan, find groups with zero references, mark for deletion. Instead, it chose a design that prioritizes safety over immediacy. The defaults (24-hour grace period, 7-day minimum age) are generous, suggesting that the assistant expected operators to tune these downward after gaining confidence in the system's correctness.

Input Knowledge Required to Understand This Message

To fully grasp what this commit message describes, a reader needs substantial domain knowledge spanning several layers:

First, one must understand the Filecoin storage model: that data is stored in deals with miners, that deals have durations, and that claims must be periodically extended to keep data alive. The commit message references "claim extender," "groups," "deals," and "providers" — all Filecoin-specific concepts.

Second, one must understand the project's database architecture: that it uses both SQL (PostgreSQL-compatible, via sqldb) for relational state and CQL (Cassandra-compatible, via cqldb) for high-throughput index data. The commit message references both migration directories, and understanding why some state lives in SQL and some in CQL requires knowing the project's hybrid storage strategy.

Third, one must understand the concept of "reverse indices" and "reference counting" in the context of content-addressed storage. The CQL migration creates a GroupToMultihash table, which is a reverse index — it maps from groups to the multihashes (content identifiers) they contain. Normally, the system looks up which group a block belongs to; the reverse index allows enumerating all blocks in a group efficiently (O(n) rather than O(n log n)). The MultihashRefCount table tracks how many S3 objects reference each block, enabling the GC to determine which blocks are truly orphaned.

Output Knowledge Created by This Message

The commit message itself creates a permanent record of architectural decisions. Future developers reading the git log will see not just what changed but why it was designed that way. The "Key design decisions" section is particularly valuable — it encodes the reasoning that might otherwise be lost to time. The message also serves as documentation of the configuration surface area: six new environment variables with their defaults and purposes.

Beyond documentation, the commit message creates a boundary between milestones. By tagging this as "Milestone 04" and the previous as "Milestone 03," the assistant established a clear separation of concerns. The caching system (Milestone 03) is about performance; the GC system (Milestone 04) is about data hygiene. This separation helps with testing, debugging, and reasoning about the system.

The Thinking Process Visible in the Reasoning

The assistant's thinking process, visible across the messages leading up to this commit, reveals a methodical approach to problem-solving. Before writing any code, the assistant explored the existing codebase extensively. It read the database migration files to understand the existing schema. It read the claim extender to understand how deals are currently renewed. It identified a critical architectural insight: "The claim extender does NOT work at the group level directly. Instead, it works at the provider/deal level." This discovery shaped the entire GC design. Because the claim extender operates at the deal level, the GC could not simply say "skip this group" — it had to build a mapping from provider+piece to group, then filter at the claim level.

The iterative debugging visible in messages 1777-1784 shows the assistant grappling with real implementation challenges. The first draft of gc.go referenced RibsDB, which didn't exist — the actual type was ribsDB. The assistant fixed this by checking the actual type definition in deal_db.go. Then it encountered errors about QueryRowContext not existing on the sqldb.Database interface. Rather than trying to add the method to the interface (which would require changing a shared abstraction), the assistant adapted the GC code to use the available QueryRow method, accepting that the context parameter would be unused. This is a pragmatic trade-off: the code is slightly less ideal (no context propagation for cancellation) but compiles against the existing interface without modification.

Assumptions and Potential Mistakes

The commit message and the surrounding code reveal several assumptions that could prove incorrect in production:

The assumption that "no active deletion" is safe depends on the cost of storage. If local SSD cache space is scarce, allowing dead blocks to linger until their deals expire could cause capacity issues. The assistant mitigated this with configurable parameters, but the default behavior favors safety over efficiency.

The assumption that reference counting via batched increments/decrements will keep pace with S3 write throughput is unproven. The reference counter has a "background flush loop with configurable interval," which means there is a window between when a reference is created and when it is persisted. If the GC scanner runs during this window, it might see stale reference counts and incorrectly mark a group for GC. The grace period is designed to catch this, but the interaction between flush interval and grace period timing could be subtle.

The assumption that the claim extender can be modified to skip GC groups without breaking the deal renewal logic is another risk. The assistant noted that "the current claim extension logic doesn't easily map back to groups because it works at the claim level from the chain." The solution — querying deals by provider and piece CID to find the group — adds a database join to a critical path. If this query is slow or misses edge cases, the claim extender might skip deals that should be renewed, potentially causing data loss.

Conclusion

The commit message at index 1799 is far more than a version control annotation. It is a design document, a milestone marker, and a declaration of architectural philosophy. It captures the moment when a developer, having just completed a sophisticated caching system, turned to the equally important problem of knowing when to let go. The passive GC system it describes is conservative by design — prioritizing safety over immediacy, simplicity over completeness, and configurability over hard-coded assumptions. The message encodes not just what was built, but the reasoning behind every major decision: why passive over active, why two-phase marking, why generous grace periods, why reference counting with batched persistence.

For a reader unfamiliar with the project, this message opens a window into the challenges of building production-grade storage infrastructure on top of a decentralized network. For a future maintainer, it provides a roadmap of intent — a record of the trade-offs that were consciously accepted. And for the system itself, it marks the point at which the codebase grew from a cache-and-serve architecture into one that could sustainably manage its own lifecycle. That is the weight carried by a single git commit.