From Validated Infrastructure to Enterprise-Grade Architecture: Building the Next Generation of Filecoin Gateway

Introduction

Software development at scale is rarely a linear progression from one feature to the next. More often, it follows a rhythm of consolidation, exploration, planning, and execution — a cycle that repeats as each layer of capability is added to a growing system. This article examines one such cycle in the development of the Filecoin Gateway (FGW), a horizontally scalable S3-compatible distributed storage system built on IPFS, Filecoin, and YugabyteDB. The segment of the coding session analyzed here captures a pivotal transition: the moment when a validated deployment infrastructure becomes the foundation for three ambitious milestones — Enterprise Grade monitoring and backup, Persistent Retrieval Caches, and Data Lifecycle Management.

What makes this segment particularly instructive is not the volume of code produced — though substantial — but the disciplined methodology that guided its production. The session demonstrates a repeatable pattern for moving from research to execution in complex distributed systems: investigate the existing codebase, research state-of-the-art approaches, synthesize findings into a detailed plan, make critical architectural decisions with the stakeholder, execute incrementally with testing at every step, and recognize when cognitive limits require a checkpoint. Each phase of this pattern is visible in the conversation, and each contributed to the successful delivery of four new metrics files, a distributed tracing framework, configuration extensions for JSON logging and backup, and an Adaptive Replacement Cache implementation — all within a single continuous session.

Part I: The Foundation — Completing the Ansible Deployment Pipeline

The segment opens with the culmination of a major infrastructure effort. The assistant and user had built and validated a complete Ansible-based deployment pipeline for FGW clusters, culminating in commit 806c370. This pipeline included roles for common system setup, secure wallet distribution, YugabyteDB initialization, Kuri storage node deployment, and S3 frontend proxy deployment — all validated through a Docker-based multi-container test harness that verified connectivity, database initialization, Kuri health checks, and S3 frontend readiness.

The test harness passed all four stages. The deployment was repeatable, tested, and committed. This created a natural inflection point — the foundation was solid, and the question became: what comes next? The answer, delivered in message 1676, was a strategic pivot from execution to exploration.

Part II: The Research Pivot — Investigating Three Milestones

The user's directive in message 1676 was remarkably specific. Rather than a vague instruction to "make the system better," it enumerated three milestones with explicit sub-components and algorithmic constraints:

"Plan the following work: Milestone 02: Enterprise Grade - Metrics - Log & monitoring - Backup restore - Docs - Support (support - build KB and an support AI agent); Milestone 03: Persistent Retrieval caches - Retrieval Prefetcher (per kuri node); Milestone 04: Data lifecycle - Garbage collection on Filecoin - Deal extension - Repairing process — Start agents to investigate and understand deeply the requirements, research sota approaches to open problems, write a detailed execution plan."

The user also established a critical architectural freedom: breaking changes were permitted without backward compatibility constraints. This was a greenfield mandate within a brownfield context, signaling that the team was free to redesign components rather than being constrained by existing interfaces.

The assistant responded by launching multiple parallel research agents, each investigating a specific domain of the existing codebase. The research uncovered several critical findings that would shape the execution plan:

Part III: The Synthesis — A Comprehensive Execution Plan

The research agents returned their findings, and the assistant synthesized them into a single, coherent execution blueprint spanning three milestones, 12 weeks of implementation, and dozens of new files across the Go codebase. The plan's architecture reveals a clear design philosophy: observability comes first because you cannot tune what you cannot measure, and you cannot recover what you have not backed up.

Milestone 02: Enterprise Grade

The plan called for approximately 30 new Prometheus metrics organized into four domains:

  1. Deal pipeline metrics (rbdeal/deal_metrics.go): counters for deals proposed, accepted, rejected, and failed; gauges for active deals; histograms for proposal and sealing durations.
  2. Financial and balance metrics (rbdeal/balance_metrics.go): gauges for wallet balance, market balance, and datacap remaining; counters for market top-ups and faucet requests.
  3. Database operation metrics (database/metrics.go): histograms for CQL and SQL query durations; counters for query errors; histograms for batch sizes.
  4. S3 frontend metrics (server/s3frontend/metrics.go): counters for requests by method and status; histograms for request durations; gauges for backend health. Beyond metrics, the plan included JSON structured logging with correlation IDs, automated wallet backup to configurable S3 endpoints, Grafana dashboards following the Four Golden Signals (latency, traffic, errors, saturation), and a self-hosted AI support agent using Mistral or Llama via Ollama.

Milestone 03: Persistent Retrieval Caches

The caching architecture proposed a multi-tier design replacing the 512MB LRU cache:

Milestone 04: Data Lifecycle Management

The GC design centered on a new GroupToMultihash reverse index enabling O(n) enumeration of all blocks in a group. The algorithm uses three phases:

  1. Find zero-ref blocks via range scan on the MultihashRefCount table.
  2. Enqueue for GC with the current epoch.
  3. Process GC queue items older than a grace period (to prevent collecting blocks temporarily at zero references due to concurrent transactions). The user's key constraint — "Removed/Retired sectors are simply not renewed" — shaped the entire GC strategy as passive: groups with zero live blocks simply do not get their Filecoin claims extended, rather than actively deleting data.

Part IV: The Decisions — Four Questions That Shaped the Architecture

The plan was comprehensive, but it contained open questions requiring the user's judgment. The assistant distilled these into four targeted questions: LLM provider (cloud-hosted vs. self-hosted), L2 cache budget, GC strategy (passive vs. active), and backup destination.

The user's answers were decisive and shaped the architecture in concrete ways:

  1. Self-hosted LLM (Mistral/Llama via Ollama) — signaling a preference for autonomy and data sovereignty over raw model capability. This became a YAML configuration with provider, model, and base_url fields.
  2. Configurable L2 cache in the hundreds of GB — emphasizing flexibility over a fixed default.
  3. Passive GC only — a conservative choice that minimizes risk of data loss.
  4. Configurable S3 endpoint (AWS or self-hosted MinIO) — emphasizing diverse deployment environments. The assistant immediately translated these answers into concrete Go struct definitions. The backup configuration became a struct with six environment-variable-backed fields: BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY, BACKUP_S3_REGION, and BACKUP_ENCRYPTION_KEY. The cache configuration became another struct with FGW_L1_CACHE_SIZE_MIB, FGW_L2_CACHE_SIZE_GB, FGW_L2_CACHE_PATH, and FGW_L2_CACHE_ENABLED. This translation from abstract decision to concrete configuration is the essence of architectural design — it forces specificity and creates artifacts that can be directly implemented.

Part V: The Green Light — Incremental Test-Driven Execution

With the plan documented and the decisions made, the user issued the final directive in message 1689: "execute all milestones, avoid asking questions, test incrementally as implementation progresses - unit, integration tests. Refer to milestones document as needed, generously."

This was a mandate for autonomous, test-driven execution. The assistant responded immediately by creating a structured todo list with the first task — "M02: Add deal pipeline metrics (deal_metrics.go)" — marked as in_progress. The pattern was set: break each milestone into discrete tasks, start with the first, test incrementally, and proceed without seeking further approval.

The "Read Before Write" Discipline

Before writing any code, the assistant read three existing files: retr_metrics.go (the existing metrics pattern), deal_tracker.go (the deal lifecycle logic), and deal_db.go (the database interaction layer). This "read before write" discipline ensured that new code would follow established conventions — the promauto.NewCounter pattern, the struct-based metrics organization, the naming conventions. The assistant was not just adding metrics; it was extending a system, and extension requires understanding the system's architecture.

Part VI: Building Enterprise-Grade Infrastructure — The Implementation

The implementation of Milestone 02 proceeded methodically, file by file, with build verification at each step.

Metrics Files

The assistant created four metrics files, each following the established pattern of defining a Go struct with Prometheus counter fields initialized via promauto:

Configuration Extensions

The assistant then extended the configuration system in configuration/config.go with two new structs:

type LogConfig struct {
    Format string `envconfig:"RIBS_LOG_FORMAT" default:"text"` // "text" or "json"
}

type BackupConfig struct {
    S3Endpoint    string `envconfig:"BACKUP_S3_ENDPOINT" default:"https://s3.amazonaws.com"`
    S3Bucket      string `envconfig:"BACKUP_S3_BUCKET"`
    S3AccessKey   string `envconfig:"BACKUP_S3_ACCESS_KEY"`
    S3SecretKey   string `envconfig:"BACKUP_S3_SECRET_KEY"`
    S3Region      string `envconfig:"BACKUP_S3_REGION" default:"us-east-1"`
    EncryptionKey string `envconfig:"BACKUP_ENCRYPTION_KEY"`
}

The LogFormat option, with values "text" and "json", was a small change with transformative impact — it enabled log aggregation systems like Loki to parse structured fields, turning unstructured text into queryable observability data. The configureLogFormat function was added to the LoadConfig flow, with an LSP error caught and fixed immediately — demonstrating the incremental testing discipline in action.

The Trace Package

The most architecturally significant component of Milestone 02 was the trace package in server/trace/. This package implemented:

  1. Trace ID generation — UUID-based correlation IDs
  2. Context propagation — storing trace IDs in Go's context.Context
  3. HTTP middleware — extracting or generating trace IDs from request headers
  4. Structured log fields — formatting trace context for zap logging
  5. Request injection — propagating trace IDs to backend requests The assistant wrote the implementation (trace.go) and a comprehensive test suite (trace_test.go) simultaneously, then ran go test ./server/trace/... -v and watched all seven tests pass in zero seconds. The tests covered: TestGenerateTraceID, TestFromContext, TestFromRequest, TestMiddleware, TestLogFields, TestWithLogFields, and TestInjectIntoRequest. This was the "test incrementally" mandate in action: write a focused unit of functionality, validate it immediately, then move on. A full build verification at message 1698 confirmed that all new metrics files compiled correctly across the three affected packages. The assistant had reached a natural checkpoint: the core of Milestone 02 — metrics definitions, logging configuration, trace infrastructure — was structurally complete.

Part VII: The ARC Cache — Milestone 03 Implementation

With Milestone 02's core deliverables validated, the assistant updated the todo list, marking all four metrics tasks as completed, and announced the transition: "Now let me update the todo and start on the caching (Milestone 03)."

This transition message is deceptively simple, but it carries enormous contextual weight. It marks the boundary between two fundamentally different kinds of work: observability infrastructure (metrics, logging, tracing) and performance optimization (caching, prefetching). The assistant's decision to move to caching before completing every detail of Milestone 02 (wallet backup implementation, Loki integration, AI support agent) reflects a breadth-first approach — getting each milestone to a structurally complete state before moving on, rather than finishing every detail of one before touching the next.

The ARC Algorithm

The first task of Milestone 03 was the ARC cache. The assistant created the rbcache directory, wrote arc.go (the ARC implementation) and arc_test.go (the test suite), and immediately ran the tests.

The ARC (Adaptive Replacement Cache) algorithm is a sophisticated cache eviction policy that dynamically balances between recency and frequency. Unlike LRU (which only tracks recency) or LFU (which only tracks frequency), ARC maintains four lists:

Cache Configuration

Following the ARC implementation, the assistant added cache configuration to the configuration system:

type CacheConfig struct {
    L1SizeMiB   int    `envconfig:"FGW_L1_CACHE_SIZE_MIB" default:"2048"`
    L2SizeGB    int    `envconfig:"FGW_L2_CACHE_SIZE_GB" default:"256"`
    L2Path      string `envconfig:"FGW_L2_CACHE_PATH" default:"/data/cache/l2"`
    L2Enabled   bool   `envconfig:"FGW_L2_CACHE_ENABLED" default:"true"`
}

This configuration makes the L2 SSD cache size and path configurable via environment variables, with sensible defaults (2GB L1, 256GB L2). The L2 cache can be disabled entirely, allowing deployments to use only the DRAM-based ARC cache if SSD storage is not available.

Part VIII: The Checkpoint — Recognizing Cognitive Limits

The segment concludes with a remarkable meta-cognitive moment. After the ARC cache tests passed, the user sent what appears to be an empty message (index 1716). The assistant's response was not to continue implementing — it was to create a meticulously structured checkpoint document that inventories every file created, every test passed, every design decision made, and every next step required.

This response reveals sophisticated self-awareness. The assistant recognized that the conversation had accumulated hundreds of messages and was approaching the limits of its context window. Rather than continuing to build new components that might become disconnected from earlier decisions, it paused to externalize its state into a portable document — a survival strategy for maintaining coherence across the fragility of conversational context.

The checkpoint document includes:

  1. Project overview — what FGW is and the current state
  2. Accomplishments by milestone — detailed inventory of every file created and test passed
  3. File tree — all new and modified files organized by package
  4. Task list — prioritized remaining work for each milestone
  5. Design decisions — self-hosted LLM, passive GC, configurable S3 backup
  6. Shell commands — for resuming work after context reset
  7. Contextual notes — breaking changes are allowed, algorithms must be O(n), GC is passive only This document is not written for a human reader — it is written for the model itself, to reconstruct its mental state after a context reset. It represents an advanced form of meta-cognitive tool use, where the assistant recognizes its own cognitive limitations and builds external scaffolding to compensate.

Conclusion: Patterns for Distributed Systems Development

This segment of the coding session tells a story that is larger than any single feature or bug fix. It is a case study in how disciplined engineering teams navigate the gap between foundation and feature, between execution and exploration, between planning and implementation. The arc is clear: validate the deployment infrastructure, pivot to research, synthesize findings into a coherent plan, make critical design decisions, execute incrementally with testing at every step, and recognize when cognitive limits require a checkpoint.

The patterns visible in this segment are transferable to any complex software project:

  1. Research before implementation: The assistant did not jump into coding. It launched parallel research agents to investigate the existing codebase, found 49 metrics with critical gaps, discovered the unimplemented GC, and grounded every proposal in reality.
  2. Structured decision-making: The assistant distilled open questions into four targeted choices, captured the user's answers precisely, and translated them into concrete configuration structures. This prevented ambiguity from propagating into implementation.
  3. Read before write: Before creating new metrics files, the assistant read the existing patterns — retr_metrics.go, deal_tracker.go, deal_db.go — ensuring new code followed established conventions.
  4. Test incrementally: Every unit of functionality was tested immediately after creation. The trace package had seven tests that passed in zero seconds. The ARC cache had six tests covering basic operations, eviction, promotion, scan resistance, ghost list adaptation, and updates.
  5. Breadth-first milestone progression: Rather than finishing every detail of Milestone 02 before touching Milestone 03, the assistant got each milestone to a structurally complete state and moved on. This approach maximizes the surface area of validated work.
  6. Meta-cognitive checkpointing: Recognizing when context is fragile and externalizing state into a portable document is a survival strategy for any long-running development effort, whether human or AI. At the end of this segment, the Filecoin Gateway project has a validated deployment pipeline, a comprehensive execution plan for three milestones, four new metrics files covering deal pipeline, financials, database operations, and S3 frontend, a distributed tracing framework with correlation ID propagation, JSON logging configuration, backup configuration, an ARC cache implementation with passing tests, and a checkpoint document that can survive a session reset. The foundation is laid for the next phase: L2 SSD cache, access tracking, prefetching, garbage collection, deal extension, and repair. But the patterns established in this segment — research before implementation, testing before integration, and checkpointing before context loss — will carry the work forward.