From Dashboard to Cluster: The Two Faces of Building a Distributed S3 System

Introduction

In the lifecycle of any complex distributed systems project, there are two distinct kinds of work that rarely get equal attention. The first is the work of making the system observable — building the dashboards, charts, and monitoring interfaces that let operators see what is happening inside the black box of distributed services. The second is the work of making the system runnable — creating the infrastructure scripts, Docker Compose configurations, and initialization logic that transform code into a living, breathing cluster. These two efforts are deeply interdependent, yet they often proceed on separate tracks, with different tools, different mindsets, and different measures of success.

This article examines a single chunk of an extended coding session for the Filecoin Gateway's horizontally scalable S3-compatible storage architecture — a session where both of these tracks unfolded in rapid succession, and where the tension between them revealed fundamental truths about how distributed systems are actually built. The chunk covers two major efforts: first, the design and implementation of a comprehensive cluster monitoring dashboard with real-time React components and WebSocket-driven data polling; and second, the construction of a test cluster infrastructure with Docker Compose, shell scripts, and a cascade of debugging fixes that ultimately uncovered a critical architectural error.

What makes this chunk particularly instructive is the way these two efforts interacted. The monitoring UI was built for a system that did not yet exist as a deployable test environment. The test cluster was built to support a system that had a fundamental architectural flaw. Together, they tell a story about the iterative, non-linear nature of distributed systems development — where observability and infrastructure are not sequential phases but parallel, mutually-reinforcing concerns.

Part I: Building the Dashboard — Making the Invisible Visible

The first major arc of this chunk began with a user request that signaled a shift in the project's maturity. After the assistant had completed all five phases of implementing the horizontally scalable S3 architecture — modifying Kuri storage nodes for node identification, building a stateless S3 frontend proxy with round-robin distribution and health checking, implementing YCQL-based read routing, creating a multipart upload coordination system, and writing unit tests — the user asked for something different. Not more features, not bug fixes, but observability: "Design a UI with a live cluster and data flow overview, including useful performance charts" [42].

This request marked a transition from building infrastructure to making it visible. The assistant's response was methodical and architecture-aware. Rather than jumping into React components, it first explored the existing codebase — the RIBSWeb application — to understand the tech stack (React 18.2.0, Recharts 2.7.2, Create React App 5.0.1), the real-time communication pattern (WebSocket RPC), and the existing component architecture [42]. This exploration ensured that the new dashboard would integrate seamlessly rather than feeling bolted-on.

The design that emerged was comprehensive. It specified four major React components: a ClusterTopology SVG-based diagram showing the hierarchical relationship between load balancers, proxies, and storage nodes with color-coded health status and animated bezier curves; Performance Charts using Recharts for throughput (multi-line), latency (stacked area with p50/p95/p99 and SLA reference lines), and error rates (horizontal bar charts); NodeStatistics tables separating frontend proxy metrics from storage node metrics with sortable, expandable rows; and a DataFlowOverview with real-time animated counters breaking down request routing into reads (YCQL routing), writes (round-robin), and multipart (coordinating) [44].

The design also specified five new backend RPC methods — ClusterTopology, RequestThroughput, LatencyDistribution, ErrorRates, and ActiveRequests — each with stub implementations that would need to be wired to actual metrics collection [44]. A tiered polling strategy was defined: 100ms for active requests (to feel real-time), 1s for throughput and latency (smooth chart updates), 5s for topology and node stats (slower-changing data), and real-time push for events. A color scheme using CSS custom properties with semantic names (healthy green, degraded orange, unhealthy red) completed the design specification.

What followed was a systematic implementation effort. The assistant created each component file in sequence — ClusterTopology.jsx, RequestThroughputChart.jsx, LatencyDistributionChart.jsx, ErrorRateChart.jsx, ActiveRequestsPanel.jsx, NodeStatistics.jsx, DataFlowOverview.jsx, RecentEventsTimeline.jsx — along with their corresponding CSS files, a barrel export (index.js), a navigation link in the existing routing structure, and integration into the RIBSWeb application [67][68][69][70][71][72][74][76][78][80]. Each file write was accompanied by a build verification step, and when imports failed or components couldn't be found, the assistant methodically traced the errors to their source — often discovering that a file had been created with a slightly different name than what was imported, or that a barrel export was missing a re-export [82][83][84][85][86][87][88].

The monitoring UI implementation was, in many ways, a model of systematic development. The assistant worked from a clear design specification, created files in a logical order, verified each step with build checks, and fixed issues as they arose. By the end of this arc, the RIBSWeb application had a new /cluster route with a fully functional monitoring dashboard — at least in terms of UI components and stubbed backend data.

But there was a problem. The dashboard had no cluster to monitor.

Part II: Building the Cluster — When Infrastructure Meets Reality

The second major arc began with a deceptively simple user question: "How do I run a test cluster with 2x2 nodes, data in /data/fgw-test/" [90]. This question was a pivot point. The assistant had just completed an elaborate monitoring UI for a system that did not yet exist as a deployable test environment. The dashboard was a cockpit with no aircraft, a beautiful instrument panel with no engine to connect it to.

The user's question demanded that the architecture become operational. The assistant responded by creating a complete test cluster infrastructure: a Docker Compose configuration with two Kuri storage nodes, a shared YugabyteDB instance, and a suite of shell scripts — start.sh, stop.sh, init-data.sh, logs.sh, test.sh — all parameterized with a data directory argument [92][102]. The Docker Compose file defined the service topology: Kuri nodes with environment variables for node identification, database connection strings, and API bind addresses; YugabyteDB with volume mounts for persistent storage; and a db-init service to create the required PostgreSQL database and CQL keyspace before the Kuri nodes attempted to connect.

But the infrastructure did not work on first try. What followed was a cascade of debugging that revealed the gap between configuration and reality.

The database that wasn't there. When the user ran ./start.sh /data/fgw2, both Kuri nodes crashed immediately with a deeply nested dependency injection error traceback. At the bottom of the chain: pq: database "filecoingw" does not exist [160]. The YugabyteDB container was running and healthy, but the PostgreSQL database that the Kuri nodes expected to connect to had never been created. The environment variables YSQL_DB: filecoingw and YCQL_KEYSPACE: filecoingw were set on the YugabyteDB container, but these do not automatically create the database — they only configure what the YugabyteDB process would use if it were creating databases on first startup, which is a different mechanism entirely. The fix was to add a db-init service: a one-shot container that runs after YugabyteDB is healthy, executes the SQL commands to create the database and keyspace, and then exits [160][161][162].

The dependency dilemma. Even with the db-init service in place, there was a question of timing. The assistant initially trusted Docker Compose's depends_on with condition: service_completed_successfully to ensure that Kuri nodes would wait for database initialization. But the user, when asked, chose to add an explicit wait loop in start.sh — a defensive approach that made the script's behavior visible and debuggable [170][260]. This decision reflected a preference for explicit error handling over implicit orchestration, born from operational experience with timing-dependent failures.

The port conflict that wasn't. The user raised a sharp-eyed concern: "Is the :9010 cluster mon not going to conflict with :9010 on individual kuri nodes?" [112]. The assistant's first instinct was to explain that Docker containers are isolated — internal container ports don't conflict with host-mapped ports. But the assistant quickly recognized the deeper issue: while there was no technical conflict, there was a usability and clarity problem. If someone wanted to debug an individual Kuri node by accessing its web UI directly, they couldn't, because those ports weren't mapped to the host. This triggered a cascade of fixes: disabling the web UI on individual Kuri nodes, adding explanatory comments, updating the README with a troubleshooting entry, and documenting how to re-enable individual node web UIs if needed [112][113][114][115][116][117][118][119][120].

The flag assumptions. Throughout the infrastructure work, the assistant repeatedly assumed that the Kuri daemon accepted certain command-line flags — --s3-api, --webui — that simply did not exist. The actual configuration mechanism was through environment variables like RIBS_S3API_BINDADDR, and the web UI started automatically on port 9010 with no flag to disable it [124][125][126][127][128][129][130][131][132][133][134][135][136][137][138][139][140][141][142][143][144][145][146][147][148][149][150][151][152][153][154][155][156][157][158][159]. Each assumption had to be verified by reading the Kuri source code, and each discovery required a cascade of corrections across the Docker Compose file, the shell scripts, and the documentation.

Part III: The Architecture Error That Changed Everything

The most significant discovery in this chunk was not a database initialization bug or a port conflict — it was a fundamental architectural error that the user identified by reading the project's own roadmap document.

The assistant had configured the test cluster with Kuri nodes exposing S3 APIs directly and sharing a single configuration file. But the scalable-roadmap.md document — which the assistant itself had written earlier — clearly specified that S3 frontend proxies are a separate stateless node type that routes requests to Kuri storage nodes, not the storage nodes themselves [92][102]. The roadmap described a clean three-layer design: stateless S3 frontend proxy nodes routing requests to backend Kuri storage nodes, all coordinated through a shared YugabyteDB database.

The assistant had the correct mental model — as shown in the architecture diagrams it drew — but translated it incorrectly into infrastructure. The proxies existed as containers in the Docker Compose file, but the routing logic — the crucial layer that makes requests flow through proxies to storage nodes — was either missing or misconfigured. The Kuri nodes themselves were configured as S3 endpoints, bypassing the frontend layer entirely.

This error is particularly instructive because it shows how easily architectural understanding can be lost in translation from document to implementation. The assistant had read the roadmap, understood the design, and even implemented the frontend proxy package. But when it came to wiring everything together in Docker Compose, it fell back on a simpler model: Kuri nodes are the S3 endpoints, so let's expose them directly. The proxies became an afterthought — or were simply omitted from the initial configuration [92].

The correction required a complete redesign: generating per-node independent settings files, restructuring the Docker Compose hierarchy into a proper three-layer architecture (S3 proxy on port 8078 → Kuri storage nodes → YugabyteDB), implementing the routing layer as specified in the roadmap, and ensuring each Kuri node had its own external HTTP endpoint for CAR file staging. This was not a minor fix — it was a fundamental re-architecture of the test cluster, driven by the user's recognition that the assistant had deviated from the documented design.

Part IV: The Interplay Between Observability and Infrastructure

What makes this chunk particularly rich is the way the two major efforts — building the monitoring dashboard and building the test cluster — interacted and revealed each other's assumptions.

The monitoring dashboard was designed for an idealized architecture: stateless proxies routing to storage nodes, with clean data flows and well-defined metrics. But the test cluster, when actually built, revealed that the architecture was not yet correctly implemented. The dashboard's topology diagram would have shown proxies that didn't actually route requests. The throughput charts would have displayed data from Kuri nodes acting as direct S3 endpoints, not from the stateless proxy layer. The data flow overview would have misrepresented the actual request path.

In other words, the monitoring UI, if deployed against the flawed test cluster, would have displayed a system that did not match reality. It would have shown operators a clean three-layer architecture when the actual infrastructure was a flat, misconfigured topology. This is a profound risk in distributed systems monitoring: dashboards can lie, not because the data is wrong, but because the underlying architecture is different from what the dashboard assumes.

The test cluster, by contrast, forced the assistant to confront the gap between design and implementation. Every operational failure — the missing database, the incorrect flags, the permission issues, the port conflicts — was a test of whether the architecture could actually work. The user's practical question — "How do I run this?" — created the pressure that exposed the divergence between the roadmap and the code.

This interplay reveals a deeper truth about distributed systems development: observability and infrastructure are not sequential phases but parallel, mutually-reinforcing concerns. Building the dashboard forces you to specify what metrics matter, which in turn forces you to instrument the infrastructure correctly. Building the infrastructure forces you to confront the operational reality of your architecture, which in turn reveals what your dashboard should actually display. The two efforts are a feedback loop, not a linear sequence.

Part V: Lessons for Distributed Systems Development

Several lessons emerge from this chunk that apply broadly to distributed systems engineering.

Configuration is architecture. The Docker Compose file was not just a convenience for developers — it was the concrete expression of the system's topology. By getting the configuration wrong — by making Kuri nodes direct S3 endpoints instead of routing through proxies — the assistant had effectively built a different system than the one described in the roadmap. No matter how elegant the code, no matter how thorough the design document, if the configuration that wires everything together doesn't match the architecture, the system will not behave as intended [92].

Assumptions must be verified against source code. The assistant repeatedly assumed that the Kuri daemon accepted certain command-line flags, only to discover through source code inspection that the actual configuration mechanism was through environment variables. Each assumption required a verification cycle — reading the code, discovering the discrepancy, and correcting all the files that depended on the wrong assumption. This pattern is universal in distributed systems: the documentation, the configuration, and the actual behavior of a service are three different things, and only the source code tells the truth [124][125][126][127][128][129][130][131][132][133][134][135][136][137][138][139][140][141][142][143][144][145][146][147][148][149][150][151][152][153][154][155][156][157][158][159].

Health checks are only as good as their criteria. The YugabyteDB health check verified that the database process was running and accepting connections, but it did not verify that the specific database "filecoingw" existed. The container passed its health check while still being unable to serve the application's needs. This is a classic pitfall: a health check that verifies "is the process running?" is very different from one that verifies "is the required resource available?" [160].

Explicit beats implicit in operational scripts. The user's choice to add an explicit wait loop for db-init rather than trusting Docker Compose's dependency mechanism reflected a preference for debuggability over elegance. In distributed systems testing, where failures are often silent and timing-dependent, explicit error handling can save hours of troubleshooting [170][260].

Documentation is not optional. Throughout the chunk, the assistant consistently updated documentation — README files, troubleshooting entries, architecture diagrams — as part of every fix. The port conflict resolution included a troubleshooting entry explaining how to access individual node web UIs. The stop script addition included updates to the README, the start script's output, and the test script's summary. This commitment to documentation completeness transformed the test cluster from a fragile experiment into a usable tool [112][113][114][115][116][117][118][119][120][170][171][172][173][174][175].

Conclusion

This chunk of the coding session tells a story that is both specific to the Filecoin Gateway project and universal to distributed systems development. It is the story of building a dashboard for a system that didn't yet exist, then building the system and discovering it was architecturally wrong, then fixing the architecture while simultaneously fixing the infrastructure, the configuration, the scripts, and the documentation.

The monitoring UI and the test cluster were two faces of the same effort: making the distributed S3 architecture visible and making it runnable. Neither could succeed without the other. The dashboard needed a real cluster to monitor; the cluster needed a dashboard to be understood. And both needed the architecture to be correct — which it was not, until the user asked the simple question that exposed the flaw.

In the end, what this chunk reveals is that building distributed systems is not a linear process of design, implement, test, and deploy. It is an iterative, messy, recursive process where every layer — architecture, infrastructure, configuration, observability, documentation — must be built and rebuilt in response to discoveries made at every other layer. The dashboard reveals what the infrastructure is actually doing. The infrastructure reveals whether the architecture actually works. The architecture reveals whether the design was correct. And the user's questions — "How do I run this?" — reveal whether any of it matters at all.