The Architecture of a Multi-Tier Cache: Building the L2 SSD Layer for Persistent Retrieval

In the span of a single, deceptively brief message, an AI coding assistant transitions from comprehension to creation. The message — a mere four lines of declarative intent followed by a file write operation — marks the moment when the abstract design for a persistent retrieval cache crystallizes into concrete implementation. The assistant writes:

Now I have a good understanding of the ARC cache interface. Let me create the L2 SSD Cache with: - SLRU eviction policy (probationary + protected segments) - Admission policy (only admit items evicted from L1 with 2+ accesses) - Write buffering for sequential SSD writes - In-memory index with on-disk data storage [write] /home/theuser/gw/rbcache/ssd.go Wrote file successfully.

This message is the hinge point in a larger narrative — the implementation of Milestone 03 (Persistent Retrieval Caches) for the Filecoin Gateway (FGW), a distributed storage system that provides an S3-compatible API on top of the Filecoin network. To understand why this message matters, one must understand the problem it solves and the architectural context that demanded it.

The Problem: Why a Multi-Tier Cache Exists

The Filecoin Gateway is not a simple file server. It retrieves data from a decentralized storage network, where blocks of content-addressed data (identified by CIDs — Content Identifiers) are spread across storage providers. Retrieval is expensive: it involves network requests to Filecoin miners, verification of proofs, and reassembly of data from DAG structures. Every retrieval miss carries a latency cost measured in seconds, not milliseconds.

The existing system already had a basic LRU (Least Recently Used) cache in the retrieval path. But LRU has well-known pathologies: a sequential scan of a large dataset can evict the entire working set, causing catastrophic cache pollution. The milestone document called for something more sophisticated — a multi-tier cache hierarchy that could survive sequential scans, adapt to changing access patterns, and leverage SSD storage for capacity.

The assistant had already completed the L1 cache: an Adaptive Replacement Cache (ARC) implementation in /home/theuser/gw/rbcache/arc.go. ARC dynamically balances between recency and frequency using two competing lists (T1 for recent, T2 for frequent) and two ghost lists (B1, B2) that track recently evicted items to adapt the balance parameter p. It is scan-resistant by design — a sequential scan fills T1 but the working set survives in T2 and the ghost lists prevent the scan from permanently polluting the cache.

But ARC operates in memory. For a cache that needs to hold hundreds of gigabytes (the configuration default was 256 GB), an in-memory-only approach is impractical. The L2 SSD Cache was the necessary complement: a larger, slower tier that could persist data to disk while still providing fast access for frequently-requested blocks.

The Design Decisions Embedded in Four Bullet Points

The assistant's message lists four design parameters. Each represents a deliberate architectural choice, and each carries assumptions about the underlying hardware and workload.

SLRU (Segmented LRU) Eviction Policy. The choice of SLRU over alternatives like LFU (Least Frequently Used), 2Q (Two Queue), or ARC itself for the L2 tier is telling. SLRU divides the cache into two segments: a probationary segment for newly admitted items and a protected segment for items that have been accessed multiple times. Items in the probationary segment are evicted quickly if not re-accessed; items promoted to the protected segment are given a longer lifetime. This is a natural fit for an SSD-backed cache because it concentrates repeated accesses in a smaller, hotter region, reducing write amplification. The assistant's explicit mention of "probationary + protected segments" shows awareness that the eviction policy must be sympathetic to the underlying storage medium.

Admission Policy: Only Items Evicted from L1 with 2+ Accesses. This is perhaps the most consequential design decision. The L1 ARC cache already filters the access stream. By admitting only items that were evicted from L1 and had two or more accesses, the L2 cache acts as a safety net for items that are popular enough to survive L1 for a while but not popular enough to stay indefinitely. The threshold of "2+ accesses" is a heuristic: one access could be a one-off; two accesses suggests a pattern. This policy also prevents the L2 cache from being polluted by the same sequential scans that ARC already resists — items that pass through L1 once and are evicted never reach L2. The assumption is that the L1 cache is large enough to absorb transient traffic, and only the "tail" of moderately popular items needs SSD persistence.

Write Buffering for Sequential SSD Writes. SSDs have asymmetric performance: reads are fast, but random writes cause write amplification and wear. By buffering writes and flushing them sequentially, the cache can coalesce small random writes into larger sequential ones. This is a well-known optimization in storage engines (LevelDB, RocksDB, WAL files). The assistant's inclusion of this detail signals an understanding that the cache is not just an algorithm — it is a system that must respect the physics of the hardware.

In-Memory Index with On-Disk Data Storage. This separates the metadata path from the data path. The index (mapping keys to disk locations, sizes, and timestamps) lives in memory for fast lookups. The actual block data lives on SSD. This is the standard architecture for high-performance key-value caches (think: Memcached with external storage, or Facebook's McDipper). It assumes that the index is small relative to the data — a reasonable assumption when each entry is a few dozen bytes of metadata for potentially megabytes of block data.

The Reasoning Process Visible in the Message

The message begins with a critical phrase: "Now I have a good understanding of the ARC cache interface." This is not a throwaway line. It reveals the assistant's working method: before building the L2 cache, it read the existing ARC implementation to understand the interface contract it must satisfy. The two caches must interoperate — the L1 ARC cache feeds evictions into the L2 admission policy. The assistant needed to know what the ARC cache exposes: its eviction callback, its size tracking, its metrics. Only after that review did the assistant feel confident enough to proceed.

The four bullet points are not random. They follow a logical progression: first, how the cache decides what to keep (SLRU eviction); second, what qualifies for admission (the policy); third, how data is written efficiently (write buffering); fourth, the storage architecture (index + data). This is a top-down decomposition from policy to mechanism to physical storage.

Input Knowledge Required

To understand this message, one needs familiarity with:

Output Knowledge Created

The message produces a single file: /home/theuser/gw/rbcache/ssd.go. But the output knowledge is richer than a file path. It establishes:

Assumptions and Potential Pitfalls

The design makes several assumptions worth examining. The admission policy assumes that L1 evictions with 2+ accesses are a reliable signal of future value. This could fail if access patterns shift suddenly — an item that was popular for three accesses and then becomes dead weight will occupy SSD space until evicted by SLRU. The write buffering assumes that the system can tolerate a small window of data loss on crash (buffered but unflushed writes). The in-memory index assumes that the index fits in RAM — for a 256 GB cache with, say, 4 KB blocks and 64-byte index entries, that's about 4 million entries and 256 MB of index, which is manageable but grows with smaller block sizes.

Conclusion

This message is a microcosm of systems engineering: a moment where understanding crystallizes into action, where abstract policies become concrete code, and where the careful selection of four design parameters determines the performance characteristics of a distributed storage system. The assistant's brief declaration — "Now I have a good understanding" — masks the depth of reasoning required to make those four choices correctly. The L2 SSD Cache is not just a file; it is a set of assumptions about hardware, workload, and access patterns, encoded in Go and destined for a production retrieval path.