The Bridge Between Eviction and Promotion: Wiring L1→L2 Cache Promotion in a Distributed Storage System

Introduction

In the lifecycle of any caching system, eviction is not the end—it is a transition. When a hot cache reaches capacity and must discard entries, those entries often still hold value; they simply are not hot enough to justify their current position. A well-designed system recognizes this and, rather than discarding them entirely, promotes them to a slower but more capacious tier. This article examines a single message from an AI-assisted coding session that sits at the exact midpoint of implementing exactly such a promotion path: wiring an eviction callback in an Adaptive Replacement Cache (ARC) so that evicted blocks from an L1 memory cache are automatically promoted to an L2 SSD cache.

The message in question is message index 2578 from a lengthy coding session focused on closing critical implementation gaps in the Filecoin Gateway (FGW) project—a horizontally scalable, distributed S3-compatible storage system built on YugabyteDB and the Kuri storage engine. At this point in the session, the assistant had already fixed the Prefetcher Fetch() method (replacing a stub that returned "prefetch not fully implemented" with a real implementation) and was now turning its attention to the L1→L2 cache promotion pathway, which had been identified as a missing critical piece.

The Message

The subject message reads as follows:

[assistant] ## Agent Reasoning
Let me add a method to set the eviction callback after the NewARCCache function.
[read] /home/theuser/gw/rbcache/arc.go
<file>
00161| 		t2:       list.New(),
00162| 		t2Keys:   make(map[K]*list.Element),
00163| 		b1:       list.New(),
00164| 		b1Keys:   make(map[K]*list.Element),
00165| 		b2:       list.New(),
00166| 		b2Keys:   make(map[K]*list.Element),
00167| 		p:        0,
00168| 		sizeFunc: sizeFunc,
00169| 		metrics:  newARCMetrics(metricsName),
00170| 	}
00171| }
00172| 
00173| // Get retrieves a value from the cache.
00174| // Returns the value and true if found, zero value and false otherwise.
00175| fun...

On its surface, this appears to be a simple read operation—the assistant is examining the arc.go file to find the right insertion point for a new method. But this message is far more interesting than a mere file read. It represents the precise moment when the assistant transitions from understanding the existing structure to making the structural modification. It is the pivot point in a multi-step implementation sequence.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the chain of reasoning that led to it. Earlier in the session, the user had directed the assistant to fix the most critical implementation gaps identified by a comprehensive subagent analysis. The assistant maintained a prioritized todo list:

  1. Implement RefCounter integration with S3 object operations (pending)
  2. Fix Prefetcher Fetch() implementation (completed just before this message)
  3. Add L1→L2 cache promotion callback (in progress)
  4. Wire AccessTracker to retrieval flow (pending)
  5. Metrics integration (pending) The assistant had just completed item 2 (the Prefetcher Fetch() method) and marked it as done. It then turned to item 3: adding an L1→L2 cache promotion callback. The reasoning was straightforward: the system had a two-tier cache hierarchy—an L1 ARC cache in memory and an L2 SSD-based persistent cache—but there was no mechanism to promote entries from L1 to L2 when they were evicted. This meant that when the memory cache filled up and discarded entries, those entries were simply lost, even though they might still be useful enough to warrant storage on the slower SSD tier. The assistant's approach was methodical. It first read the ARC cache implementation (rbcache/arc.go) to understand the eviction mechanics (message 2573–2575). It discovered that eviction happened in two methods: evictFromT1() and evictFromT2(), which removed entries from the T1 (recency) and T2 (frequency) lists respectively and moved them to ghost lists (B1 and B2) for adaptation tracking. The assistant then made an initial edit (message 2576) to add an evictionCallback field to the ARCCache struct. But adding the field was only half the work. The assistant needed to know where to add the SetEvictionCallback() method that would allow external code to register a callback. Message 2578 is the assistant reading the file to find that exact insertion point—after the NewARCCache constructor function and before the Get method.

The Thinking Process: What the Assistant Was Reasoning

The assistant's reasoning, visible in the ## Agent Reasoning block, reveals a clear and deliberate thought process. The key phrase is: "Let me add a method to set the eviction callback after the NewARCCache function." This tells us several things:

  1. The assistant already knew the structure of the file. It had read arc.go multiple times in previous messages and understood the layout: the NewARCCache constructor (ending at line 171), followed by the Get method (starting at line 173). The assistant was not exploring blindly; it was confirming the exact line numbers before making an edit.
  2. The assistant was thinking about API design. By placing the SetEvictionCallback method after the constructor, the assistant was following a common Go pattern: the constructor creates the object, and optional configuration methods follow. This allows the callback to be set after construction but before the cache is actively used, without requiring it to be a constructor parameter.
  3. The assistant was being precise about placement. The [read] command was not a casual glance—it was a deliberate check of the exact code at the boundary between the constructor and the first public method. The assistant needed to see the closing brace of NewARCCache (line 171) and the comment for Get (line 173) to know that lines 171–173 were the safe insertion zone. The read output shows lines 161–175 of the file, which is the tail end of the NewARCCache function (initializing the ARC cache's internal lists: t2, t2Keys, b1, b1Keys, b2, b2Keys, p, sizeFunc, metrics) and the beginning of the Get method. The assistant was looking at exactly the right place.

Input Knowledge Required

To understand and act on this message, several pieces of prior knowledge were necessary:

  1. The ARC cache algorithm: The assistant needed to understand Adaptive Replacement Cache mechanics—the four lists (T1, T2, B1, B2), the concept of ghost entries, and how eviction decisions are made. This knowledge was gathered from reading the file's comments and code in earlier messages.
  2. The existing cache hierarchy: The assistant knew that the retrievalProvider in retr_provider.go creates both an L1 ARC cache and an L2 SSD cache (SSDCache), and that these are separate objects with different storage backends (memory vs. disk).
  3. Go language patterns: The assistant needed to know idiomatic Go patterns for optional configuration—specifically, the pattern of providing a Set* method rather than requiring the callback in the constructor, which would have required breaking changes to the NewARCCache signature.
  4. The project's directory structure: The assistant knew that rbcache/arc.go contained the ARC implementation and that rbdeal/retr_provider.go contained the retrieval provider where the callback would be wired up.
  5. The todo list priority: The assistant was working from a prioritized list of critical gaps, and understood that this item was blocking the data lifecycle—without L1→L2 promotion, blocks evicted from memory were simply lost, reducing cache effectiveness.

Output Knowledge Created

This message itself did not create output knowledge—it was a read operation. However, it was the immediate precursor to the output knowledge created in the subsequent messages:

  1. Message 2579: The assistant added the SetEvictionCallback method to arc.go, creating the public API for registering eviction callbacks.
  2. Messages 2580–2581: The assistant modified evictFromT1() and evictFromT2() to invoke the callback when entries are evicted, passing the key and value to the registered function.
  3. Messages 2582–2588: The assistant wired the callback in retr_provider.go, creating a lambda that calls l2Cache.Put(key, value) to promote evicted L1 entries to the SSD cache. The chain of output knowledge ultimately produced a working L1→L2 promotion mechanism, verified by a successful build (message 2590). The callback architecture meant that the ARC cache remained generic—it didn't need to know about SSDCache or any particular promotion target—while the retrieval provider supplied the specific promotion logic.

Assumptions Made

The assistant made several assumptions during this work:

  1. That the callback should be optional. The assistant added the callback as an optional field (nil by default) rather than requiring it in the constructor. This was a reasonable assumption—the ARC cache is used in other contexts where promotion might not be desired—but it meant that the code had to check for nil before calling the callback in every eviction path.
  2. That the callback should receive both key and value. The assistant designed the callback signature to pass (key K, value V) to the callback. This assumed that the promotion target (L2 cache) needed both the key and the data. This turned out to be correct, as SSDCache.Put(key, data) requires both.
  3. That the eviction callback should be called synchronously. The assistant did not add any goroutine or async mechanism—the callback is called inline during eviction. This assumed that the promotion operation (writing to SSD) would be fast enough not to block the cache operation significantly. This was a pragmatic choice but could become a performance concern under heavy eviction pressure.
  4. That the ARC cache's internal locking was sufficient. The assistant did not add additional synchronization around the callback invocation, assuming that the existing mutex in the ARC cache would protect against concurrent access. This was correct, as the eviction methods are called from within the cache's locked methods.

Mistakes and Incorrect Assumptions

The most notable mistake occurred not in this message but in the subsequent wiring (message 2585). When the assistant first wrote the callback lambda in retr_provider.go, it used rp.l2Cache.Put(string(key), value) as a statement, but SSDCache.Put returns void—it doesn't return a value. The Go compiler rejected this with: rp.l2Cache.Put(string(key), value) (no value) used as value. The assistant had to check the SSDCache.Put signature (messages 2586–2587) and fix the lambda (message 2588).

This mistake reveals an interesting assumption: the assistant assumed that Put returned something (perhaps a boolean success indicator or an error), when in fact the SSD cache's Put method is a fire-and-forget operation that logs errors internally. This is a common gotcha when working with unfamiliar APIs—the developer (or in this case, the AI) assumes a return type based on common patterns rather than checking the actual signature.

Another subtle assumption was that the callback should be set after both L1 and L2 caches are initialized. The assistant initially tried to set the callback immediately after creating the L1 cache (message 2584), but realized that the callback lambda needed to reference rp.l2Cache, which hadn't been initialized yet. This required reorganizing the initialization order so that the callback was set after both caches existed (message 2585).

Broader Significance

This message, while brief, illustrates a fundamental pattern in software engineering: the moment of transition between understanding and acting. The assistant had already analyzed the problem, understood the ARC cache's eviction mechanics, and added the callback field. But before it could add the SetEvictionCallback method, it needed to verify the exact insertion point. This read-before-write pattern is the software equivalent of a surgeon checking the anatomy before making an incision.

The message also demonstrates the value of incremental, testable changes. Rather than attempting a large refactoring that modified the ARC cache's constructor signature or added promotion logic directly into the cache, the assistant chose a minimal, composable approach: add an optional callback, then wire it up externally. This kept the ARC cache generic and reusable, while the promotion logic lived in the retrieval provider where it belonged.

In the broader context of the FGW project, this L1→L2 promotion mechanism was a critical piece of the data lifecycle. Without it, the two-tier cache was effectively a one-tier cache—entries evicted from memory were gone forever, and the SSD tier was only populated by direct HTTP retrievals. The promotion callback ensured that the SSD tier acted as a true second chance for evicted blocks, improving cache hit rates and reducing the need for expensive network retrievals.

Conclusion

Message 2578 is a quiet but essential moment in a larger implementation. It is the assistant reading the codebase to confirm the exact insertion point for a new method—a method that would become the bridge between eviction and promotion. The message reveals the assistant's methodical approach: understand the structure, confirm the location, then act. The subsequent messages would complete the implementation, but this read operation was the necessary precondition. In the lifecycle of a software change, such moments of verification are often invisible in the final commit, but they are where the real understanding happens.