The Architecture of Unlink: Reasoning Through a Critical Implementation Gap

Introduction

In distributed storage systems, data deletion is rarely as simple as removing a file. When blocks are spread across multiple nodes, indexed in distributed databases, and stored in append-only logs, the concept of "deleting" data becomes a layered architectural decision. This article examines a single message from an AI-assisted coding session where the agent grappled with implementing the Unlink method in a horizontally scalable S3-compatible storage system built on YugabyteDB and CAR archives. The message, brief as it appears, captures a pivotal moment of architectural reasoning that determined how data removal would work across the entire system.

The Message

The subject message, message index 2493 in the conversation, reads as follows:

Agent Reasoning Good, withReadableGroup returns just an error. But for Unlink, we might need to modify the group. Let me think about this more carefully.

>

Actually, looking at the architecture: 1. Unlink removes index entries from CQL (MultihashToGroup table) 2. It doesn't actually delete data from the CarLog (that would require compaction) 3. It updates group counters

>

For the index removal, we don't need the group to be writable in the sense of accepting new writes - we just need to update the CQL index and group metadata. Let me check if there's a better approach.

>

Actually, looking at the Group.Unlink signature and the fact that it needs to write to the index (DropGroup), I think we should use withReadableGroup and then call index.DropGroup directly from ribBatch, OR we need a different approach.

>

Let me look at how DropGroup is used and what it does. [edit] /home/theuser/gw/rbstor/rbs.go Edit applied successfully.

Context: The Unlink Gap

To understand why this message exists, we need to trace back through the conversation. The user had issued a directive in message 2479: "Address the critical parts, test things like unlink/compaction, schema - do not define indexes until clearly needed, those are very expensive in cql, usually separate k-v ish tables are better." This was a response to a broader analysis that had identified critical implementation gaps in the Filecoin Gateway's distributed storage layer.

The most glaring gap was the Unlink method. In both rbstor/rbs.go and rbstor/group.go, the method existed only as a stub:

func (r *ribBatch) Unlink(ctx context.Context, c []mh.Multihash) error {
    //TODO implement me
    panic("implement me")
}

And similarly in group.go:

func (m *Group) Unlink(ctx context.Context, c []mh.Multihash) error {
    // write log
    // write idx
    // update head
    //TODO implement me
    panic("implement me")
}

The comments in the Group.Unlink stub — "write log", "write idx", "update head" — represented the developer's initial understanding of what needed to happen, but the actual implementation had never been written. The ribBatch.Unlink was equally empty. This was a blocking issue for the entire data lifecycle: blocks could be written and read, but they could never be removed.

The Initial Attempt and Its Failure

The assistant's first attempt at implementing Unlink came in message 2490. The reasoning at that point was straightforward: the assistant understood that Unlink should be a "logical delete" — removing index entries rather than physically erasing data. The initial implementation tried to use withWritableGroup, a method that opens a group for writing and provides a callback with the group object. However, this immediately failed with a compilation error:

ERROR [331:10] assignment mismatch: 1 variable but r.r.withWritableGroup returns 2 values

The assistant discovered in message 2491 that withWritableGroup returns (selectedGroup iface.GroupKey, err error) — two values, not one. The code had only captured one. This is where message 2492 picks up: the assistant discovers withReadableGroup, which returns just an error, and begins to question whether the group needs write access at all for an unlink operation.

The Core Architectural Insight

Message 2493 is where the assistant pauses and thinks through the architecture from first principles. This is the heart of the message and the reason it deserves close analysis.

The assistant enumerates three key facts about the architecture:

  1. Unlink removes index entries from CQL (MultihashToGroup table) — The YugabyteDB CQL table MultihashToGroup is the authoritative mapping from content identifiers to storage groups. Removing entries from this table makes blocks unreachable through normal View operations. This is the primary mechanism of deletion.
  2. It doesn't actually delete data from the CarLog (that would require compaction) — The CAR archive log (CarLog) is an append-only structure. Blocks are written sequentially and cannot be individually removed without rewriting the entire log. Physical deletion would require a compaction process that rebuilds the archive without the removed blocks. Unlink is explicitly not compaction; it's a lighter-weight operation.
  3. It updates group counters — Groups track metadata like block counts and byte sizes. When blocks are unlinked, these counters need to be decremented to reflect the removed data. This is important for capacity planning, deal management, and operational monitoring. From these three facts, the assistant draws a critical conclusion: "For the index removal, we don't need the group to be writable in the sense of accepting new writes." This is a subtle but important distinction. The group's "writable" state is about whether it can accept new data writes to its CarLog. Unlink doesn't write new data to the CarLog; it only modifies the CQL index and updates in-memory counters. Therefore, withReadableGroup — which provides read access to the group — is sufficient.## The withReadableGroup vs. withWritableGroup Distinction The choice between withReadableGroup and withWritableGroup is not merely a technical detail — it reflects a deep understanding of the system's concurrency model. In the rbstor package, groups are complex objects that manage CAR archive logs, index references, and metadata. Opening a group for writing reserves resources, potentially blocks other writers, and requires careful lifecycle management. The withWritableGroup method, as seen in group_storage.go, takes a preferred group key and a callback, and returns the selected group key along with any error. It's designed for operations that need to append data to the CarLog. withReadableGroup, by contrast, is lighter-weight. It only needs to ensure the group is accessible and its data can be read. For Unlink, the assistant realizes that the group doesn't need to be in a write-ready state — it just needs to be loaded so that its counters can be updated. The actual index removal happens at the CQL level, not through the group's CarLog. This distinction matters for correctness and performance. If Unlink required withWritableGroup, it could fail or block when the group is in a read-only state (e.g., during offloading or compaction). By using withReadableGroup, Unlink becomes a more resilient operation that can proceed even when the group isn't accepting new writes.

Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are sound but worth examining:

Assumption 1: Unlink only needs to remove index entries, not data. This is correct for the current architecture. The CQL index is the authoritative source for block location. If an entry is removed from MultihashToGroup, subsequent View calls will return iface.UndefGroupKey, indicating the block is not found. The physical data remains in the CarLog, but it becomes inaccessible through normal lookup paths. This is a classic "logical delete" pattern used in many distributed systems.

Assumption 2: Group counters need updating. This is also correct. The GroupMeta structure tracks Blocks, Bytes, ReadBlocks, ReadBytes, WriteBlocks, and WriteBytes. When blocks are unlinked, the Blocks and Bytes counters should be decremented. However, the assistant later discovers that the group metadata schema needs a dead_blocks and dead_bytes column rather than decrementing live counters — a nuance that emerges during implementation.

Assumption 3: withReadableGroup is sufficient for counter updates. This is partially correct. The group's counters are stored in the group's head metadata, which is managed by the CarLog. Reading the group provides access to this metadata, but updating it requires a mutating operation. The assistant later finds that a new UpdateGroupDeadBlocks method needs to be added to RbsDB to handle this mutation properly.

Assumption 4: The ribBatch.Unlink should orchestrate group-level unlinks. This is the correct architectural pattern. The ribBatch is the batching layer that groups operations for efficiency. It tracks which groups need flushing after unlink operations. The Group.Unlink method handles the per-group logic.

Input Knowledge Required

To fully understand this message, one needs familiarity with several concepts:

  1. CQL (Cassandra Query Language) and YugabyteDB — The system uses YugabyteDB's CQL API for distributed storage. The MultihashToGroup table is a CQL table that maps multihashes to group keys. Understanding that CQL is a distributed database with different performance characteristics than SQL is crucial — the user's warning about expensive indexes reflects real Cassandra/YugabyteDB operational knowledge.
  2. CAR archives and CarLog — The system stores blocks in CAR (Content Addressable aRchive) format, an append-only log structure. The CarLog manages these archives and provides Put, View, Commit, and truncate operations. There is no individual block deletion in CarLog — only truncation of the entire log after a certain offset.
  3. The Group abstraction — Groups are the unit of storage organization. Each group has a CarLog for data, a CQL index entry for each multihash, and metadata counters. Groups can be in various states (active, offloaded, sealed) that affect what operations are allowed.
  4. The Batch interface — The iface.Batch interface groups operations for efficiency. Unlink is a batch operation that removes multiple multihashes at once. The ribBatch implementation tracks pending operations and flushes them to the database.
  5. Go concurrency patterns — The withReadableGroup and withWritableGroup methods use callback-based patterns common in Go for resource management. Understanding the distinction between read and write access to shared resources is essential.## The Thinking Process: A Window Into Architectural Reasoning What makes message 2493 particularly valuable is the visible thinking process. The assistant doesn't just state a conclusion — it walks through the logic step by step, correcting its own initial assumptions along the way. The thought process begins with a moment of uncertainty: "Good, withReadableGroup returns just an error. But for Unlink, we might need to modify the group. Let me think about this more carefully." This self-correction is crucial. The assistant had initially tried to use withWritableGroup (in message 2490), hit a compilation error, discovered withReadableGroup (in message 2492), and is now questioning whether read-only access is sufficient for a mutating operation. The assistant then enumerates the three architectural facts listed earlier. This enumeration serves as a mental checklist — a way of grounding the implementation decision in the system's actual design rather than abstract assumptions. Each fact is verified against the codebase: the CQL index has DropGroup, the CarLog has no individual deletion, and the group metadata has counters that need updating. The conclusion — "we don't need the group to be writable in the sense of accepting new writes" — is a moment of genuine insight. It separates the concept of "writing to the group's data log" from "modifying the group's metadata." The former requires write access; the latter only requires the group to be loaded. The final thought — "I think we should use withReadableGroup and then call index.DropGroup directly from ribBatch, OR we need a different approach" — shows the assistant considering two architectural patterns. Option A: keep the unlink logic in ribBatch and call the index directly. Option B: delegate to Group.Unlink which handles the group-level details. The assistant doesn't fully resolve this tension in message 2493, but the consideration itself shapes the eventual implementation.

Output Knowledge Created

This message, while primarily reasoning, creates several forms of output knowledge:

  1. A clarified architectural model for Unlink — The three-point enumeration (index removal, no CarLog deletion, counter updates) becomes the mental model that guides all subsequent implementation. Future readers of this conversation can understand the design rationale.
  2. A resolved API choice — The decision to use withReadableGroup rather than withWritableGroup has concrete implications for the code. It means Group.Unlink doesn't need to open the group for writing, which simplifies error handling and reduces contention.
  3. A direction for the implementation — The assistant identifies that it needs to examine DropGroup more closely to understand how index removal works. This leads to the next steps in the implementation.
  4. A documented design constraint — The message makes explicit that Unlink is not compaction. This is an important distinction for anyone maintaining the system: unlinked blocks still occupy physical space until compaction runs.

Potential Mistakes and Incorrect Assumptions

While the reasoning in message 2493 is largely sound, several assumptions would be challenged or refined in subsequent implementation:

The counter update mechanism was incomplete. The assistant assumed that group counters could be updated through the existing group metadata path. However, the schema at that point only had a dead_blocks column, not dead_bytes. The assistant later had to add a schema migration to include dead_bytes and create a new UpdateGroupDeadBlocks method on RbsDB. The counters weren't simply decremented; they were tracked separately as "dead" counts.

The relationship between ribBatch.Unlink and Group.Unlink was underspecified. The assistant considered two approaches but didn't fully resolve the delegation pattern. In the eventual implementation, ribBatch.Unlink iterates over groups, calls Group.Unlink for each, and tracks which groups need flushing. This hybrid approach — orchestration at the batch level, execution at the group level — emerged through further iteration.

The assumption that "no CarLog modification" means "no CarLog interaction" was partially wrong. While Unlink doesn't delete data from the CarLog, it does need to interact with the CarLog's head metadata to update counters. The Group.Unlink implementation eventually calls m.jb.mutHead() to update the dead blocks and bytes counters in the CarLog's head structure. This is a mutation of CarLog metadata, even though it's not a data deletion.

The assumption that withReadableGroup is sufficient was correct but required additional infrastructure. The group's counters are stored in the CarLog head, which is accessible through the group object obtained via withReadableGroup. However, updating these counters required adding the UpdateGroupDeadBlocks method to RbsDB, which was not initially anticipated.

The Broader Significance

Message 2493 represents a critical inflection point in the implementation of the distributed storage system. Before this message, Unlink was a panic("implement me") stub — a known gap that blocked the data lifecycle. After this message, the assistant had a clear architectural model to guide implementation.

The message also exemplifies a pattern that appears repeatedly in successful software development: the pause to think before coding. The assistant could have continued down the path of using withWritableGroup, fixing the compilation error, and producing a working-but-suboptimal implementation. Instead, it stepped back, enumerated the architectural facts, and made a deliberate choice about the API to use.

This kind of reasoning is especially important in distributed systems, where the cost of wrong decisions is high. Choosing the wrong access pattern for a distributed operation can lead to contention, deadlocks, or inconsistent state across nodes. By reasoning carefully about whether write access is truly needed, the assistant avoided a potential source of operational complexity.

The message also demonstrates the value of understanding the full data path. The assistant traced the Unlink operation through the CQL index (DropGroup), the CarLog (no individual deletion), and the group metadata (counter updates). This end-to-end understanding is essential for implementing operations that touch multiple subsystems correctly.

Conclusion

Message 2493 is a study in architectural reasoning under uncertainty. Faced with a stub implementation and a compilation error, the assistant paused to think through the fundamental nature of the Unlink operation. By enumerating what Unlink actually does — remove index entries, skip CarLog deletion, update counters — and matching those requirements to the available APIs (withReadableGroup vs. withWritableGroup), the assistant arrived at a sound architectural decision.

The message's value lies not in the code it produces (a single edit to rbs.go) but in the thinking it documents. Future developers reading this conversation can understand why withReadableGroup was chosen over withWritableGroup, why Unlink doesn't touch the CarLog data, and what the relationship is between index removal and counter updates. In a complex distributed system, this kind of documented reasoning is as valuable as the code itself — it's the map that explains why the territory looks the way it does.