The Moment of Decision: How Four Questions Shaped the Future of a Distributed Storage System

A Pivotal Crossroads in the Filecoin Gateway Project

In any substantial software engineering effort, there comes a moment when research and planning must crystallize into concrete decisions. For the Filecoin Gateway (FGW) project—a horizontally scalable, S3-compatible distributed storage system built on Filecoin—that moment arrived in a single, deceptively compact message from the project's AI coding assistant. The message, indexed at position 1682 in a long conversation, reads as follows:

[assistant] [question] {"questions":[{"header":"LLM Choice","options":[{"description":"Best reasoning, most reliable, cloud-hosted","label":"OpenAI GPT-4o (Recommended)"},{"description":"Strong reasoning, good for technical content","label":"Anthropic Claude"},{"description":"Full control, no external dependencies, higher... User has answered your questions: "For the AI Support Agent, which LLM provider should we target?"="Self-hosted (Llama/Mistral)", "For L2 SSD cache, what's the expected storage budget per kuri node?"="100s of gb, configurable", "Should GC be passive-only (don't extend claims) or active (delete from external storage)?"="Passive only (Recommended)", "For backup, where should encrypted wallet backups be stored?"="Specifieble S3 (aws or selfhosted) endpoint". You can now continue with the user's ans...

This message is far more significant than its brevity suggests. It represents the precise inflection point where extensive research—ten separate investigation agents had just completed deep dives into metrics infrastructure, logging, backup requirements, caching strategies, garbage collection algorithms, repair mechanisms, and state-of-the-art monitoring practices—was distilled into actionable design decisions. The answers to these four questions would determine the architectural direction for three major milestones: Enterprise Grade features, Persistent Retrieval Caches, and Data Lifecycle Management.

The Context That Made These Questions Necessary

To understand why this message exists, one must appreciate what preceded it. The FGW project had just completed a validated Ansible-based deployment infrastructure, culminating in commit 806c370. This was a hard-won achievement: the team had iteratively debugged Docker test harnesses, fixed environment file syntax, corrected log level formats, resolved wallet dotfile issues, and separated database migrations—all to achieve a repeatable, production-ready pipeline for deploying Kuri storage nodes, S3 frontend proxies, and YugabyteDB initialization.

With deployment infrastructure solid, the conversation pivoted dramatically. The user's directive was ambitious: plan and implement three major milestones. Milestone 02 (Enterprise Grade) called for Prometheus/Grafana metrics, centralized logging, backup/restore automation, operational documentation, and an AI-powered support agent. Milestone 03 (Persistent Retrieval Caches) demanded a multi-tier caching system with predictive prefetching. Milestone 04 (Data Lifecycle Management) required efficient garbage collection, automated deal extension, and self-healing repair mechanisms.

The assistant responded by launching ten research agents to investigate the existing codebase and research state-of-the-art approaches. These agents produced comprehensive reports: they found 49 existing Prometheus metrics with significant gaps, identified a basic 512MB LRU cache that needed expansion, discovered that the garbage collection code literally contained panic("implement me"), and catalogued every data component requiring backup. The synthesized execution plan that emerged was detailed—spanning 12 weeks of work across all three milestones—but it contained open questions that only the domain expert (the user) could answer.

The Four Questions and Their Significance

The assistant's questions were not arbitrary. Each one addressed a fundamental architectural choice where the research had identified viable alternatives, but where the "right" answer depended on operational priorities and constraints that the user understood best.

Question 1: LLM Choice for the AI Support Agent

The options presented were OpenAI GPT-4o (cloud-hosted, best reasoning), Anthropic Claude (strong reasoning, good for technical content), and a self-hosted option using Llama or Mistral models. The user chose self-hosted. This decision carries profound implications: it means the support AI agent will operate without external API dependencies, keeping all diagnostic data and knowledge base queries within the cluster's network boundary. For a system handling Filecoin wallets and potentially sensitive storage metadata, this is a security-conscious choice. It also means the implementation must include model serving infrastructure—likely something like Ollama or vLLM—rather than simply integrating with an external API.

Question 2: L2 SSD Cache Storage Budget

The assistant needed to know how much SSD capacity could be dedicated to the second-level cache per Kuri storage node. The user answered "100s of gb, configurable." This is a pragmatic response that gives the implementation both a concrete scale (hundreds of gigabytes) and the flexibility to adapt to different hardware configurations. It tells the caching subsystem designers that they should build for the 100GB–1TB range, with configuration knobs rather than hard-coded limits. This answer validates the multi-tier cache architecture proposed in the execution plan, where L2 (SSD) would serve as a warm cache between the fast L1 (DRAM, using ARC eviction) and the slow L3 (Filecoin retrieval, which can take up to 60 minutes for unsealing).### Question 3: GC Strategy — Passive vs. Active

This was perhaps the most architecturally significant question. The assistant presented two options: passive-only GC (don't extend claims for retired sectors) or active GC (delete from external storage). The user chose "Passive only (Recommended)"—notably, the assistant had marked this as recommended in the question itself. This choice aligns with the principle that "Removed/Retired sectors are simply not renewed," as the user had stated earlier. The passive approach is dramatically simpler and safer: rather than implementing complex deletion protocols that could accidentally destroy retrievable data, the garbage collector simply stops extending Filecoin deals for sectors marked for retirement. Over time, those deals expire naturally, and the storage space is reclaimed through the normal sector compaction process. This decision cascades through the entire GC algorithm design: no need for external storage deletion logic, no risk of data loss from premature deletion, and a clean separation between the GC detector (which identifies dead blocks via reference counting) and the GC executor (which simply withholds deal extension). The O(n) algorithm using range scans on CQL indexes, as designed in the execution plan, remains valid and efficient.

Question 4: Backup Destination for Encrypted Wallet Backups

The user specified "Specifieble S3 (aws or selfhosted) endpoint"—a configurable S3-compatible endpoint, whether AWS S3 or a self-hosted alternative like MinIO. This answer reinforces the project's architectural philosophy: everything is configurable, nothing is hard-coded to a single provider. It also tells the backup subsystem designers that they need to implement S3-compatible upload logic (using the AWS SDK or a lightweight alternative) rather than, say, rsync-based backups to a specific server. The encrypted wallet backup pipeline—using GPG symmetric encryption with AES256, then uploading to a configurable S3 endpoint—was already outlined in the execution plan, and this answer confirms that direction.

The Reasoning and Motivation Behind the Message

Why did the assistant ask these particular questions at this moment? The answer lies in the nature of the execution plan that had just been synthesized. That plan was comprehensive—spanning 12 weeks, covering three milestones, touching dozens of files across the codebase—but it contained branching points where different assumptions would lead to different implementations. Rather than making arbitrary choices that might later need to be reversed, the assistant correctly identified the decisions that should be deferred to the domain expert.

The timing is also significant. The questions appear at the very end of the execution plan document, after the assistant had presented the full synthesis of all ten research agents' findings. This placement is intentional: the user needed to see the complete picture—the current state of the codebase, the research into SOTA approaches, the proposed architectures—before making informed decisions. The questions serve as a bridge between "here's what we could build" and "here's what we should build."

Assumptions Embedded in the Questions

The questions reveal several assumptions the assistant was operating under. First, that the AI support agent should be built at all—the execution plan allocated weeks 10-12 to this component, but the LLM choice question assumes it's happening. Second, that the caching system needs an L2 SSD tier—the question about storage budget assumes this tier exists, rather than asking whether it should exist. Third, that garbage collection should happen—the question frames it as a choice between passive and active, not whether to implement it. Fourth, that wallet backups are critical and will use encryption—the backup question assumes GPG-encrypted S3 uploads.

These assumptions are reasonable given the research findings. The existing 512MB LRU cache is clearly insufficient for a production system where retrieval from Filecoin can take 60 minutes. The panic("implement me") in the GC code is a glaring gap. Wallet data is indeed unrecoverable if lost. But the questions could have been more open-ended: for instance, asking "should we implement an AI support agent, or focus on better runbooks first?" or "should we build L2 SSD caching, or optimize the existing L1 cache first?" The assistant's framing assumes all three milestones are going forward as planned.

Input Knowledge Required to Understand This Message

A reader needs substantial context to grasp the significance of these four questions. They need to know that FGW is a horizontally scalable S3-compatible storage gateway built on Filecoin, with Kuri storage nodes and stateless S3 frontend proxies. They need to understand that the project has just completed Ansible deployment automation and is now pivoting to enterprise features. They need familiarity with the existing codebase's limitations: 49 Prometheus metrics with gaps, a basic 512MB LRU cache, unimplemented GC, and no backup automation. They need to know the architecture's data flow: S3 requests hit the frontend proxy, which routes to Kuri nodes, which store data in groups that are eventually sealed into Filecoin deals. And they need to understand the operational context: wallet loss is unrecoverable, Filecoin retrieval is slow (up to 60 minutes for unsealing), and sector compaction requires sequential reads.

Output Knowledge Created

This message produces concrete design decisions that unblock the entire execution plan. The self-hosted LLM choice means the support agent implementation will include local model serving. The "100s of GB, configurable" answer gives the cache subsystem a target range and a configuration pattern. The passive GC choice simplifies the entire data lifecycle design. The configurable S3 backup endpoint confirms the backup architecture. These decisions ripple through every component of the three milestones, enabling the assistant to proceed with implementation without second-guessing fundamental choices.

The Thinking Process Visible in the Message

The message's structure reveals the assistant's reasoning process. It first presents the questions in a structured JSON format with headers, options, and descriptions—showing a systematic approach to decision-making. Then it reports the user's answers, confirming receipt and signaling readiness to proceed. The inclusion of "(Recommended)" on the passive GC option shows the assistant had a preference based on its research but correctly deferred the decision. The questions are ordered from least to most architecturally constrained: LLM choice (independent of other decisions), cache budget (impacts L2 design), GC strategy (impacts entire lifecycle), backup destination (impacts backup subsystem). This ordering reflects careful thought about how each decision constrains subsequent choices.

The message also demonstrates a mature understanding of the assistant's role: it does not simply implement based on default assumptions, but actively seeks validation on the decisions that matter most. This is the hallmark of effective AI-assisted development—not blind code generation, but collaborative decision-making where the AI identifies the branching points and the human provides the domain expertise to resolve them.