The Investigative Pivot: Understanding Code Before Writing It

A Deep Dive Into a Single Message of a Distributed Storage System Implementation

In the course of building a horizontally scalable S3-compatible storage system for the Filecoin Gateway, there comes a moment when the developer must stop writing code and start reading it. This article examines one such message — a brief but critical investigative turn taken by an AI assistant during a marathon coding session. The message, reproduced below, captures the assistant stepping back from implementation to understand the existing interfaces and data structures before tackling one of the most critical missing pieces in the system: the Unlink method.

[assistant] ## Agent Reasoning Let me look at how the RBS interface defines Unlink and understand what the ribBatch is. [grep] type.Batch.interface|func.*Unlink Found 1 matches /home/theuser/gw/iface/iface_rbs.go: Line 35: type Batch interface {

>

[read] /home/theuser/gw/iface/iface_rbs.go <file> 00181| 00182| type GroupMeta struct { 00183| State GroupState 00184| 00185| MaxBlocks int64 00186| MaxBytes int64 00187| 00188| Blocks int64 00189| Bytes int64 00190| 00191| ReadBlocks, ReadBytes int64 00192| WriteBlocks, WriteBytes int64 00193| 00194| PieceCID, RootCID string 00195| 00196| DealCarSize int64 // todo move to DescribeGroup 00197| } 00198| 00199| type GroupStats struct { 00200| GroupCount int64 00201| TotalDataSize int64 00202| NonOff... [read] /home/theuser/gw/rbstor/rbs.go <file> 00201| } 00202| 00203| func (r ribSession) View(ctx context.Context, c []mh.Multihash, cb func(cidx int, data []byte)) error { 00204| done := map[int]struct{}{} 00205| byGroup := map[iface.GroupKey][]int{} 00206| 00207| err := r.r.index.GetGroups(ctx, c, func(cidx int, group iface.GroupKey) (bool, error) { 00208| if _, ok := done[cidx]; ok { 00209| return false, nil 00210| } 00211| done[cidx] = struct{}{} 00212| 00213| if group == iface.UndefGroupKey { 00214| return t...

This is not a message that produces code. It is a message that produces understanding. And in many ways, these investigative messages are the most important ones in any complex software project.

Context: The Critical Gap

To understand why this message was written, we must first understand the state of the project at this moment. The assistant had just received a comprehensive analysis from eight parallel subagent investigations, revealing a sobering picture: the codebase had solid foundations but critical integration gaps. Components were implemented in isolation but not wired together. Among the most critical items was the Unlink method — a function that had been left as panic("implement me") in two separate locations: rbstor/rbs.go:315 and rbstor/group.go:428.

The user's directive was clear: "Address the critical parts, test things like unlink/compaction, schema — do not define indexes until clearly needed." The assistant had already begun this work in the preceding message, reading the current implementations of Unlink in both files, examining the CarLog structure, and searching for index interfaces. But then came this message — a deliberate investigative pause.

Why This Message Was Written

The assistant had just finished reading the stubbed-out Unlink implementations. Both locations contained the same placeholder: panic("implement me"). But before the assistant could write the actual implementation, it needed to understand the broader context. The Unlink method is not a standalone operation — it is deeply embedded in the system's data lifecycle. It must coordinate between the batch session (ribBatch), the group storage layer (Group), the CQL index, the CarLog write-ahead log, and the group metadata tracking system.

The assistant's reasoning reveals a methodical approach: "Let me look at how the RBS interface defines Unlink and understand what the ribBatch is." This is not a trivial question. The ribBatch is the batch write session that wraps group operations. To implement Unlink correctly, the assistant needed to understand:

• How the Batch interface defines Unlink at the top level
• How Group.Unlink should interact with the CarLog for writing deletion records
• How the CQL index's DropGroup method removes entries from the database
• How group metadata (block counts, byte counters) should be updated after unlinking
• How the batch session tracks which groups need flushing after modifications The message shows the assistant reading two key files: the interface definition in iface/iface_rbs.go and the session implementation in rbstor/rbs.go. The interface file contains the GroupMeta struct — a critical piece of the puzzle. This struct tracks Blocks, Bytes, ReadBlocks, ReadBytes, WriteBlocks, WriteBytes, and crucially, the assistant would later realize it needed to add DeadBlocks and DeadBytes counters to properly track unlinked data. The session file shows the View method — the read counterpart to Unlink — which uses GetGroups to find which group contains each multihash, then reads the data from the group's CarLog.

The Thinking Process Visible in the Message

The assistant's reasoning, visible in the agent reasoning block, reveals a structured investigative process. The first action is a grep search: type.*Batch.*interface|func.*Unlink. This is a deliberate scoping operation — the assistant is asking: "Where is the interface I need to implement, and what does it expect?" The grep returns one match: the Batch interface at line 35 of iface/iface_rbs.go.

But the assistant doesn't just read the interface definition. It also reads the GroupMeta struct and the ribSession.View method. Why? Because understanding Unlink requires understanding the full data lifecycle. The View method shows how data is read — it groups multihashes by their group key, then reads from each group. The Unlink method must reverse this process: it must remove the mapping from multihash to group key, so that subsequent View calls return iface.UndefGroupKey for those multihashes.

The assistant is also implicitly asking: "What happens after Unlink? How does the system know that data has been removed?" The GroupMeta struct shows the answer: the system tracks block and byte counts. For Unlink to work correctly, it must update these counters. This insight would lead the assistant to add dead_blocks and dead_bytes tracking in the database schema — a decision visible in the chunk summary but rooted in this investigative moment.

Assumptions Made

This message reveals several assumptions the assistant is operating under:

1. The interface is the specification: The assistant assumes that the Batch interface in iface/iface_rbs.go is the authoritative definition of what Unlink should do. This is a reasonable assumption in well-structured Go projects, where interfaces serve as contracts.
2. The CarLog handles deletion through its existing mechanisms: The assistant is looking for how the CarLog supports deletion. It has already examined the truncate method and the WritableIndex interface. The assumption is that Unlink can be implemented by writing to the CarLog and updating the index, similar to how truncate works but at the individual block level rather than at the offset level.
3. The CQL index's DropGroup method is the right deletion mechanism: The assistant had previously found the DropGroup method in index_cql.go and is assuming this is the correct way to remove entries from the CQL database during unlinking.
4. Group metadata needs updating after unlinking: By reading the GroupMeta struct, the assistant is assuming that the block and byte counters must be updated when blocks are unlinked. This is correct — the system needs to distinguish between "live" data and "dead" (unlinked) data for garbage collection and compaction purposes.

What Knowledge Was Required to Understand This Message

To fully grasp what the assistant is doing here, a reader would need:

1. Understanding of the RBS (RIB Storage) architecture: The system uses a two-level storage model where ribBatch sessions wrap group-level operations. Groups are the unit of data organization, and the CarLog is the write-ahead log that records all mutations.
2. Knowledge of the CQL (Cassandra Query Language) index: The system uses YugabyteDB (a CQL-compatible database) as its distributed index. The DropGroup method in the CQL index removes multihash-to-group mappings from the database.
3. Familiarity with the CarLog write-ahead log: The CarLog is a persistent log that records block writes. It supports truncation (removing data after a certain offset) and has a WritableIndex interface for updating the index during mutations.
4. Understanding of the data lifecycle: In this system, blocks are written to groups, indexed in CQL, and later may be unlinked (removed from the index) to make way for garbage collection and compaction. The Unlink method is the entry point for this lifecycle.
5. Go programming conventions: The assistant is using grep to find interface definitions and reading struct definitions to understand data flow. This is standard practice in Go development.

What Knowledge Was Created by This Message

This message produced several valuable outputs:

1. A clear understanding of the implementation path: By reading the interface and the session code, the assistant confirmed that Unlink needs to work at two levels: the ribBatch level (which tracks which groups need flushing) and the Group level (which performs the actual index removal and metadata update).
2. Identification of the GroupMeta struct as a key data structure: The assistant now knows that group metadata includes block and byte counters that must be updated during unlinking. This directly informed the later decision to add dead_blocks and dead_bytes columns to the database schema.
3. Understanding of the View method's behavior with UndefGroupKey: The View method checks for iface.UndefGroupKey and returns early for unlinked multihashes. This confirms that the goal of Unlink is to set the group key to UndefGroupKey for the target multihashes.
4. A mental model of the complete data flow: The assistant now understands how data flows from write (through ribBatch → Group → CarLog → CQL index) to read (through ribSession.View → GetGroups → CarLog) and how Unlink must fit into this flow.

The Broader Significance

This message exemplifies a pattern that recurs throughout the coding session: the assistant alternates between investigation and implementation. Before writing any code, it reads the existing interfaces, data structures, and related methods to build a complete mental model of the system. This is particularly important in a codebase as complex as this distributed S3 storage system, where components span multiple packages (rbstor, carlog, iface, database) and must coordinate through carefully defined interfaces.

The message also demonstrates the value of explicit reasoning in AI-assisted coding. By articulating its investigative intent — "Let me look at how the RBS interface defines Unlink and understand what the ribBatch is" — the assistant makes its thought process visible and auditable. A human reviewer can see not just what the assistant did, but why it did it.

In the end, this investigative message directly enabled the successful implementation of Unlink that followed. The assistant would go on to implement Group.Unlink to remove multihash entries from the CQL index via DropGroup, update group metadata with dead blocks/bytes counters, handle the offloaded group case, and implement ribBatch.Unlink to wrap this process with batch session tracking. A dedicated test file would verify the full put-unlink-view cycle. All of this was built on the foundation laid in this brief but crucial investigative moment.