The Pivotal Read: Wiring L1→L2 Cache Promotion in a Distributed Storage System

Introduction

In the intricate dance of building a horizontally scalable S3-compatible storage system, few architectural decisions carry as much weight as the caching hierarchy. The Filecoin Gateway (FGW) project, a distributed storage platform built on YugabyteDB and IPFS technology, employs a multi-tier caching strategy: an in-memory L1 ARC (Adaptive Replacement Cache) for hot blocks, and an SSD-backed L2 cache for warm blocks that don't fit in memory. The message under examination — message 2583 in the conversation — captures a seemingly small but architecturally significant moment: the assistant reading the exact location in the codebase where an eviction callback must be inserted to bridge these two cache tiers. This message is the calm before the edit, the moment of reconnaissance before a critical wiring change that would complete the cache promotion pipeline.

The Message

The subject message reads:

Agent Reasoning I see where the L1 cache is created. I need to add the eviction callback after creating the ARC cache. Let me add the callback that promotes evicted items to L2. [read] /home/theuser/gw/rbdeal/retr_provider.go <file> 00159| // Use new ARC cache 00160| rp.l1Cache = rbcache.NewARCCache[mhStr, []byte]( 00161| l1SizeBytes, 00162| func(data []byte) int64 { return int64(len(data)) },\ 00163| "l1_block", 00164| ) 00165| log.Infow("initialized L1 ARC cache", "size_mib", cacheCfg.L1SizeMiB) 00166| } else { 00167| // Fallback to legacy LRU cache 00168| rp.blockCache = must.One(lru.New[mhStr, []byte](BlockCacheSize)) 00169| log.Infow("initialized legacy LRU cache", "size_entries", BlockCac...

On its surface, this is a simple read operation: the assistant opens a file to see where the L1 cache is initialized. But the context transforms this into a decisive moment — the culmination of a multi-step refactoring effort that began with modifying the ARC cache library itself.

The Reasoning and Motivation

To understand why this message was written, we must trace the chain of reasoning that led here. The assistant was working through a prioritized list of "critical implementation gaps" identified by an earlier subagent analysis. Three items stood out: implementing the Prefetcher Fetch() method, adding an L1→L2 cache promotion callback, and wiring the RefCounter integration. The Prefetcher had just been completed (message 2572 marked it as done), and the assistant had moved on to the cache promotion callback.

The reasoning is explicit in the message's opening sentence: "I see where the L1 cache is created." This is not a discovery — the assistant had already read the ARC cache implementation, added an evictionCallback field to the ARCCache struct, created a SetEvictionCallback() method, and modified both evictFromT1() and evictFromT2() to invoke the callback when items are evicted. What remained was the final wiring step: connecting the L1 cache's eviction events to the L2 cache's Put method so that evicted blocks would be promoted to SSD storage rather than silently discarded.

The motivation is pragmatic and performance-driven. In a distributed storage gateway serving IPFS content, blocks are fetched from remote providers and cached locally. The hottest blocks — those accessed most frequently — live in the L1 ARC cache, which uses the Adaptive Replacement Cache algorithm to balance recency and frequency of access. When the L1 cache reaches capacity, it must evict entries. Without a promotion callback, those evicted blocks would be lost, requiring a costly re-fetch from the network if accessed again. By promoting evicted blocks to the L2 SSD cache, the system creates a graceful degradation path: blocks that are no longer hot enough for memory remain available on disk, trading access speed for persistence.

The Assumptions and Decisions

This message embodies several assumptions and decisions, both explicit and implicit. The most important assumption is that the L2 cache already exists and is accessible at the point where the callback is wired. The assistant's reasoning — "I need to add the eviction callback after creating the ARC cache" — implies that the callback registration must happen after both caches are initialized. This is a correct assumption: the callback lambda needs to reference rp.l2Cache.Put(), which requires the L2 cache to be fully constructed. The assistant would later discover (in message 2585) that Put returns a value, causing a compilation error, but the ordering assumption itself was sound.

Another assumption is that the ARC cache's eviction path is the correct place to intercept blocks for promotion. This is a design decision with trade-offs: it means only blocks evicted from L1 (not blocks explicitly removed or invalidated) are promoted. The assistant implicitly chose this granularity, which aligns with the typical use case where cache pressure is the primary reason for demotion.

A third assumption is that the callback should be set only for the ARC cache variant, not for the legacy LRU fallback. The code shows an if/else branch: if the L1 policy is "arc" (or default), the new ARC cache is used; otherwise, a legacy LRU cache is used. The assistant planned to add the callback inside the ARC branch, leaving the legacy path unchanged. This is a pragmatic decision that avoids modifying code paths that may be deprecated.

Input Knowledge Required

To fully understand this message, one needs knowledge of several layers of the system. First, the caching architecture: the system has two tiers — L1 (memory, ARC algorithm) and L2 (SSD, SLRU algorithm). The rbcache package provides both implementations, with ARCCache in arc.go and SSDCache in ssd.go. Second, the retrieval provider (retr_provider.go) is the component that orchestrates block fetching, using the cache hierarchy to serve data with minimal latency. Third, the ARC cache's internal structure includes four lists (T1, T2, B1, B2) that implement the adaptive replacement algorithm, and eviction happens in the evictFromT1() and evictFromT2() methods.

The reader also needs to understand the concept of an eviction callback: a function that is invoked when an entry is removed from the cache due to capacity constraints. This is a common pattern in cache libraries (e.g., Guava Cache's RemovalListener, Caffeine's eviction listeners) that allows the application to react to evictions — in this case, by writing the evicted data to a slower but larger storage tier.

Output Knowledge Created

This message, by itself, does not produce a code change. It is a reconnaissance action — the assistant reads the file to identify the exact insertion point. The output knowledge is the location itself: lines 159-165 of retr_provider.go, immediately after the rp.l1Cache = rbcache.NewARCCache(...) call and before the log statement. This knowledge is then used in the subsequent message (2584) where the assistant actually makes the edit.

However, the message also creates conceptual knowledge: it establishes the pattern for how the callback should be wired. The assistant's reasoning — "add the callback that promotes evicted items to L2" — defines the contract. The callback should take the evicted key and value, and call l2Cache.Put(key, value) to persist the block to SSD. This pattern is significant because it completes the cache lifecycle: blocks enter the system via HTTP retrieval, get cached in L1 for fast access, and upon eviction from L1, get promoted to L2 for slower-but-persistent storage.

The Broader Significance

The L1→L2 cache promotion callback is more than a minor optimization — it is a fundamental architectural component of a multi-tier storage system. Without it, the L1 cache operates as a standalone memory buffer with no connection to the persistent cache layer. Blocks that are evicted from L1 are simply discarded, and if accessed again, must be re-fetched from the network. This defeats the purpose of having an L2 cache, which is meant to serve as a warm storage tier that reduces network round-trips.

The implementation also reflects a design philosophy visible throughout the conversation: pragmatic gap-filling without over-engineering. The assistant chose a simple callback mechanism rather than a more complex eviction listener framework. The callback is a lambda that directly calls l2Cache.Put, without additional abstractions like eviction notifications, batching, or asynchronous write-behind. This simplicity is appropriate for the current stage of the project, where the goal is to unblock the data lifecycle rather than build a production-ready caching infrastructure.

Conclusion

Message 2583 captures a moment of focused intent in a complex software engineering session. The assistant, having already modified the ARC cache library to support eviction callbacks, reads the retrieval provider code to find the exact location for the final wiring. The reasoning is clear and purposeful: "I need to add the eviction callback after creating the ARC cache." This message is the bridge between library modification and system integration — the moment when a generic capability (eviction callbacks in ARC) becomes a specific feature (L1→L2 cache promotion in the retrieval provider). It is a testament to the methodical, step-by-step approach required to build robust distributed systems, where each small edit is preceded by careful reconnaissance of the existing code.