From Research to Reality: Deploying a Distributed Storage QA Cluster Across Three Physical Nodes

Introduction

In the lifecycle of any distributed system, there is a critical transition that separates a well-tested codebase from an operational service. It is the moment when binaries leave the build machine, land on bare-metal servers, and must actually work together across a network. This article examines a sustained engineering effort to deploy a QA test cluster for the FGW (Filecoin Gateway) distributed storage system across three physical nodes—a head node at 10.1.232.82 and two storage nodes at 10.1.232.83 and 10.1.232.84. The work spanned research, architecture design, infrastructure provisioning, security correction, and debugging, ultimately producing a fully functional three-node cluster that validated the entire distributed storage stack.

What makes this effort particularly instructive is not just the technical challenges overcome, but the methodological discipline that guided every step. The assistant did not rush to deploy. It researched before architecting, verified before assuming, and corrected course immediately when security gaps were identified. This article synthesizes the key phases of that deployment, drawing on the detailed message-level analyses captured in the companion articles, to present a coherent narrative of how a distributed storage QA cluster was built from scratch.

Phase 1: Research Before Architecture

The deployment began not with commands but with questions. When the user asked for a QA environment specification, the assistant responded by delegating four parallel research tasks covering FGW deployment requirements, YugabyteDB minimum specifications, Filecoin mainnet connectivity, and MinIO S3 requirements [33][35]. This research-before-architecture principle was a deliberate choice, explicitly instructed by the user: "Delegate agents to research anything you aren't 100% sure about."

The research yielded critical insights that shaped the entire deployment. The most important discovery was that FGW does not require a full Filecoin Lotus node—it connects to an external RPC endpoint (api.chain.love), eliminating the need for blockchain syncing and its associated hardware demands [35]. YugabyteDB could run as a single-node yugabyted instance for QA purposes, requiring only 2 CPU cores and 2 GB RAM minimum. The architecture plan that emerged from this research specified a three-node topology: two storage nodes with 1.25 TiB each running the Kuri daemon and S3 proxy, and one infrastructure node with 200 GiB hosting YugabyteDB and optional MinIO [35].

This phase established a pattern that would recur throughout the deployment: gather facts before making commitments, verify assumptions before executing commands, and document the reasoning so that future operators can understand why decisions were made.

Phase 2: From Blueprint to Bare Metal

The transition from architecture plan to physical deployment began with a simple connectivity check. The assistant SSHed into each of the three nodes, verifying hostnames, kernel versions, and OS releases [39]. The machines revealed themselves as fgw-qa-head, fgw-ribs1, and fgw-ribs2—all running Ubuntu 24.04.3 LTS with identical kernel builds. This uniformity was a stroke of luck; identical operating systems meant that Ansible roles and package versions would work across all nodes without OS-specific adjustments.

The assistant then methodically assessed hardware specifications, checked for existing Docker installations, verified sudo access, and examined the production Ansible inventory for reusable patterns [39][40][41]. This assessment phase was not merely preparatory—it was a form of due diligence that prevented countless downstream failures. By understanding exactly what each node offered (CPU cores, RAM, disk layout) and what it lacked (Docker, YugabyteDB, the fgw user), the assistant could make informed decisions about where to install each component and what prerequisites needed to be fulfilled.

Phase 3: The Database Foundation

Installing YugabyteDB on the head node appeared straightforward—a curl download, a tar extraction, a directory creation—but it immediately revealed a subtle permissions issue. The tarball was extracted with sudo, leaving files owned by root, but the yugabyted process would run as the theuser user [66]. The resulting startup failure required a chown correction, a small debugging detour that nonetheless illustrated a recurring theme: the gap between installation ownership and runtime ownership is a common source of infrastructure failures.

The real challenge, however, lay in creating the CQL keyspaces. YugabyteDB provides both a PostgreSQL-compatible SQL interface (YSQL) and a Cassandra-compatible CQL interface (YCQL). The SQL databases were created without incident using ysqlsh. But the CQL keyspaces—filecoingw_kuri_01, filecoingw_kuri_02, and filecoingw_s3—proved stubbornly elusive. The bundled ycqlsh tool crashed with a cryptic ModuleNotFoundError: No module named 'six.moves' [75][78][82].

This error triggered an extensive debugging spiral that consumed nearly thirty messages. The assistant checked Python paths, installed python3-six, created a python symlink, tried setting CQLSH_PYTHON and PYTHONPATH, and even examined the ycqlsh.py source code directly. Nothing worked. The root cause was a fragile dependency chain in the ycqlsh wrapper script: it discovered the Python interpreter through a shebang mechanism that failed to find a version with the six.moves module available [82][84].

The breakthrough came when the assistant abandoned the CLI tool entirely and used the Python Cassandra driver directly via an inline script executed over SSH [97]. The first attempt failed due to quote escaping—the CQL replication parameters required single quotes, but the shell quoting layers were converting them to double quotes. After correcting the escaping strategy, the keyspaces were created successfully. This workaround, bypassing a broken tool in favor of a direct API call, is a textbook example of pragmatic infrastructure debugging: when a tool is broken, do not fix the tool—work around it.

Phase 4: Building and Deploying Binaries

With the database foundation in place, the assistant turned to building the FGW software stack. The project's Makefile revealed three build targets: kuri (the storage node daemon, ~175 MB), gwcfg (a configuration utility, ~67 MB), and s3-proxy (the S3 frontend proxy, ~39 MB) [102]. A clean build succeeded without errors, producing binaries that were then copied to both storage nodes via scp [104].

The binary deployment step, however, contained a subtle architectural misunderstanding. The assistant copied both kuri and s3-proxy to each storage node, implicitly designing a topology where every storage node was also an S3 endpoint. This contradicted the project roadmap's requirement for separate stateless frontend proxy nodes [104]. The mistake was understandable—the earlier Docker-based test harness had co-located all components on a single machine—but it would later manifest as cross-node S3 read failures. Each node could only serve data from its local blockstore, not from the peer node. The eventual solution was to deploy the s3-proxy as a separate layer on the head node, routing requests to the correct backend based on object metadata in the shared S3 keyspace.

Phase 5: The Security Correction

Perhaps the most consequential moment in the deployment was not a technical breakthrough but a security correction. When the user provided the CIDgravity API token and wallet files, the assistant initially embedded the token directly into the settings.env configuration files in plaintext [106]. The user's response was immediate and unambiguous: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS" [119].

The assistant's response demonstrated a crucial capability: the ability to recognize a mistake, apologize, and redesign the approach from first principles. The corrected architecture separated configuration from secrets entirely. The settings.env files contained all non-sensitive parameters (node IDs, database hosts, API endpoints) but explicitly omitted the token, with a comment noting where it would be loaded from. The token was stored in a separate file at /home/fgw/.ribswallet/cidg.token with permissions 600, owned by the fgw user [121][122].

The systemd service files completed this architecture with a sophisticated runtime secret injection mechanism [125]. An ExecStartPre hook read the token from the restricted file and wrote it to a runtime environment file in /run/fgw/token.env—a memory-backed tmpfs that never touched persistent storage. The service then loaded this file via EnvironmentFile=-/run/fgw/token.env, with the leading dash ensuring graceful degradation if the file did not exist. Security hardening directives (NoNewPrivileges=true, ProtectSystem=strict, ProtectHome=read-only) transformed the service from a simple daemon launcher into a sandboxed execution environment.

This correction elevated the deployment from hobbyist-grade to enterprise-grade. It demonstrated that security is not a feature to be added after the fact but a constraint that must be baked into the architecture from the start. The separation of configuration from secrets, the use of restricted-permission vault files, and the runtime-only token loading pattern became the standard for the entire QA deployment.

Phase 6: Initialization and the CQL Error

The final phase of the deployment involved running kuri init on both storage nodes to create the database schemas. The command executed successfully for the SQL tables—creating the garbage collection infrastructure (MultihashRefCount, GCQueue)—but then encountered a cryptic CQL error: (ql error -12)) [126][127].

This error, truncated by a tail -20 pipe, was the symptom of a deeper problem. The CQL keyspaces contained dirty = true flags in their schema_migrations tables, left behind by prior test suite runs. These dirty flags prevented the CQL schema migrations from executing, causing the kuri init command to fail silently. The assistant did not immediately investigate this error, proceeding to run the same command on the second node before diagnosing the failure on the first [127].

The root cause—dirty migration state—would be resolved in the subsequent chunk of the session by manually updating the migration flags to false using CQL UPDATE statements. This debugging detour illustrated a critical lesson about distributed database deployments: schema migration systems are stateful, and their state must be carefully managed across test runs. A failed or interrupted migration in one session can leave artifacts that block all future migrations until manually cleaned.

Themes and Lessons

Several themes emerge from this deployment effort that are relevant beyond the specific context of FGW:

Research before architecture. The assistant spent significant effort researching requirements before committing to a design. This principle, explicitly instructed by the user, produced a more defensible and accurate architecture than guesswork would have.

Verification at every layer. Each phase of the deployment was preceded by verification steps: checking connectivity before deploying, checking permissions before starting services, checking keyspace existence before initializing schemas. This layered verification prevented cascading failures.

Security as a first-class constraint. The vault principle—secrets must never be stored in configuration files—was enforced by the user and internalized by the assistant. The resulting architecture of runtime secret injection via systemd ExecStartPre is a pattern worth replicating in any deployment.

Pragmatic workarounds over tool fixation. When the ycqlsh tool proved unfixable, the assistant bypassed it entirely and used the Cassandra driver directly. This willingness to abandon broken tools in favor of direct approaches is a hallmark of effective infrastructure engineering.

The cost of architectural assumptions. The decision to co-locate the S3 proxy with storage nodes, carried forward from the Docker test environment, created a topology that could not serve cross-node S3 reads. Assumptions from simplified test environments do not always survive contact with multi-node physical deployments.

Conclusion

The deployment of the FGW QA cluster across three physical nodes was not a single event but a sustained, multi-phase effort spanning research, architecture, provisioning, debugging, security correction, and initialization. Each phase built on the previous one, and each phase revealed assumptions that had to be tested and corrected. The final result—a working three-node cluster with secured credentials, properly initialized schemas, and a functional S3 proxy layer—validated the entire distributed storage stack on real hardware.

The most valuable lesson from this effort is not about any single technical decision but about the methodology that guided the work. The assistant researched before architecting, verified before assuming, corrected course immediately when errors were discovered, and maintained a structured approach throughout. In distributed systems deployment, where the gap between a working prototype and an operational cluster is often vast, this methodological discipline is the difference between success and prolonged debugging. The three-node QA cluster that emerged from this process was not just a test environment—it was a testament to the power of systematic, security-conscious, research-driven infrastructure engineering.## References

[33] "The Research-Before-Architecture Principle: How One Message Launched a QA Cluster" — Analyzes the assistant's decision to delegate parallel research tasks before committing to an architecture design.

[35] "Architecting a QA Test Environment for Distributed S3 Storage: A Study in Research-Driven Infrastructure Planning" — Examines the comprehensive architecture plan that emerged from the research phase, including node specifications, port tables, and deployment methodology.

[39] "From Blueprint to Bare Metal: The Pivot from Planning to Deployment in a Distributed Storage Cluster" — Details the transition from architecture planning to physical node assessment, including SSH connectivity checks and hostname discovery.

[40] "The Infrastructure Assessment That Precedes Every Successful Deployment" — Examines the methodical hardware and software assessment performed across all three nodes before deployment.

[41] "QA Cluster Privilege Check: A Pivotal Moment in Deployment" — Analyzes the verification of sudo access and system prerequisites.

[66] "The Quiet Infrastructure Decision: Installing YugabyteDB on a QA Cluster" — Unpacks the YugabyteDB installation command and the permissions issue that caused a startup failure.

[75] "Persistence of Error: Debugging a Python Import Failure in YugabyteDB CQL Keyspace Creation" — Documents the initial ycqlsh failure and early debugging attempts.

[78] "The Debugger's Litmus Test: How a One-Line Python Import Unlocked a Distributed Database" — Analyzes the moment when the assistant verified that the Cassandra driver itself worked.

[82] "Reading the Source: A Deep Dive Into One Shell Command" — Examines the assistant's examination of the ycqlsh.py source code to understand the shebang-based Python discovery mechanism.

[84] "The Persistence of a Python Import Error: Debugging YugabyteDB's ycqlsh in a Distributed Storage Deployment" — Traces the continued debugging of the six.moves import error.

[97] "The Art of the Workaround: Escaping Quotes to Create YugabyteDB Keyspaces in a QA Cluster" — Analyzes the breakthrough moment when the assistant bypassed ycqlsh and used the Cassandra driver directly with correct quoting.

[100] "The Pivot Point: How a Single Status Message Reveals the Architecture of Systematic Infrastructure Deployment" — Examines the transition message that marked the database layer as complete.

[102] "The Build That Made It Real: Compiling the FGW Distributed Storage Stack" — Analyzes the make clean && make all command that produced the kuri, gwcfg, and s3-proxy binaries.

[104] "The Binary Deployment Step: When a Simple Copy Reveals Architectural Misunderstanding" — Examines the scp-based binary deployment and the hidden assumption about S3 proxy placement.

[106] "The Credential Handoff: A Pivotal Moment in Infrastructure Automation" — Analyzes the assistant's request for the CIDgravity token and wallet files.

[119] "The Seven Words That Saved a Deployment: Why 'Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS' Matters" — Examines the user's security intervention.

[121] "The Vault Principle: How a Single Security Correction Reshaped Configuration Architecture in a Distributed Storage Deployment" — Analyzes the redesigned configuration files that separated secrets from settings.

[122] "The Security Correction: How a Configuration File Became a Lesson in Secrets Management" — Examines the corrected settings.env files.

[125] "The Art of Secret Injection: A Systemd Service Architecture for Runtime Token Loading" — Analyzes the systemd service files with ExecStartPre token loading, RuntimeDirectory, and security hardening.

[126] "The Moment the CQL Error Spoke: A Turning Point in QA Cluster Deployment" — Examines the kuri init command on the first storage node and the (ql error -12) output.

[127] "The Silent Error: A CQL Migration Failure in the FGW QA Cluster Deployment" — Analyzes the kuri init command on the second storage node and the dirty migration state that caused the error.