The Vault Principle: How a Single Security Correction Reshaped Configuration Architecture in a Distributed Storage Deployment

The Message

The following message, sent by an AI assistant during an infrastructure deployment session, represents a critical pivot point in the configuration of a three-node QA cluster for the Filecoin Gateway (FGW) distributed storage system. It is the assistant's response immediately after the user flagged a serious security violation.

[assistant] Let me create config files WITHOUT the token, and store the token separately with proper permissions:
[bash] cat > /tmp/settings-kuri1.env << 'EOF'
# Kuri settings for kuri_01 - QA Environment
# NOTE: CIDGRAVITY_API_TOKEN loaded from /home/fgw/.ribswallet/cidg.token

FGW_NODE_ID="kuri_01"
FGW_NODE_TYPE="storage"
RIBS_DATA="/data/fgw"
RIBS_MAX_LOCAL_GROUP_COUNT="30"
RIBS_MAX_STAGING_GROUP_COUNT="10"
RIBS_FILECOIN_API_ENDPOINT="https://api.chain.love/rpc/v1"

# CIDGravity - token loaded separately
CIDGRAVITY_API_ENDPOINT_GBAP="https://service.cidgravity.com/private/v1/get-best-available-providers"
CIDGRAVITY_API_ENDPOINT_GOCD="https://service.cidgravity.com/private/v1/get-on-chain-deals"
CIDGRAVITY_MAX_CONNECTIONS="4"

# Deal Settings
RIBS_MINIMUM_REPLICA_COUNT="3"
RIBS_MAXIMUM_REPLICA_COUNT="5"
RIBS_MINIMUM_RETRIEVABLE_COUNT="3"
RIBS_RETRIEVALBLE_REPAIR_THRESHOLD="2"
RIBS_DEAL_START_TIME="96"
RIBS_DEAL_DURATION="530"
RIBS_DEAL_REMOVE_UNSEALED="false"
RIBS_DEAL_SKIP_IPNI_ANNOUNCE="false"
RIBS_DEAL_CHECK_INTERVAL="30s"
RIBS_RUN_SP_CRAWLER="true"

# YugabyteDB
RIBS_YUGABYTE_SQL_HOST="10.1.232.82"
RIBS_YUGABYTE_SQL_PORT="5433"
RIBS_YUGABYTE_SQL_DB="filecoingw_kuri_01"
RIBS_YUGABYTE_SQL_USER="yugabyte"
RIBS_YUGABYTE_SQL_PASS="yugabyte"
RIBS_YUGABYTE_CQL_HOSTS="10.1.232.82"
RIBS_YUGABYTE_CQL_PORT="9042"
RIBS_YUGABYTE_CQL_KEYSPACE="filecoingw_kuri_01"

# S3 Shared Keyspace
RIBS_S3_CQL_HOSTS="10.1.232.82"
RIBS_S3_CQL_PORT="9042"
RIBS_S3_CQL_KEYSPACE="filecoingw_s3"

# S3 API
RIBS_S3API_BINDADDR=":8079"
RIBS_S3API_REGION="EU"
RIBS_S3API_AUTH_ENABLED="false"

# LocalWeb
EXTERNAL_LOCALWEB_PATH="/data/fgw/cardata"
EXTERNAL_LOCALWEB_URL="http://10.1.232.83:7010"
EXTERNAL_LOCALWEB_BUILTIN_SERVER="true"
EXTERNAL_LOCALWEB_SERVER_PORT="7010"
EXTERNAL_LOCALWEB_SERVER_TLS="false"
EXTERNAL_LOCALWEB_MAX_CONCURRENT_UPLOADS_PER_DEAL="15"

FGW_INTERNAL_API_BINDADDR=":9090"
RIBS_PROMETHEUS_PORT="2112"
RIBS_LOGLEVEL=".*:.*=info"
IPFS_PATH="/data/fgw/ipfs"
EOF

At first glance, this appears to be a routine configuration file creation. But to understand its significance, one must appreciate the chain of events that led to this moment—a chain that reveals deep truths about infrastructure security, the relationship between human operators and automated assistants, and the architectural decisions that separate hobbyist deployments from enterprise-grade systems.

The Context: A Security Violation Caught Mid-Flight

Moments before this message, the assistant had committed a serious security error. The user had provided a CIDgravity API token—a credential that grants access to a service that helps storage providers on the Filecoin network find the best deals. The assistant, in its eagerness to move quickly, had embedded this token directly into two configuration files (settings-kuri1.env and settings-kuri2.env) that were destined for deployment across three physical nodes. These files contained the token in plaintext, alongside dozens of other configuration parameters.

The user's response was immediate and unambiguous: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS." This was not a suggestion or a gentle nudge. It was a hard boundary, delivered with the urgency of someone who has seen credentials leaked, systems compromised, and the painful aftermath of security shortcuts.

The assistant's reply in message 1992 acknowledged the mistake explicitly: "You're absolutely right - I apologize for that security mistake." It then deleted the compromised files and announced a new plan: store the token in a separate restricted file and reference it at runtime via environment variable expansion in the systemd service.

Message 1993—the subject of this article—is the execution of that plan for the first node (kuri_01). It is the moment where a security principle was translated into concrete configuration architecture.## Why This Message Was Written: The Reasoning and Motivation

The message exists because of a fundamental tension in infrastructure automation: the conflict between speed and security. The assistant was operating in a mode of rapid iteration, building out a QA cluster across three physical nodes (head node at 10.1.232.82, kuri1 at 10.1.232.83, kuri2 at 10.1.232.84). It had already deployed YugabyteDB, built and transferred binaries, created user accounts, and set up directory structures. The configuration files were the last piece before initializing the Kuri storage nodes and starting services.

When the user provided the wallet and CIDgravity token in message 1979, the assistant saw an opportunity to complete the configuration quickly. It created two comprehensive settings files with the token embedded directly—a pattern that worked in earlier Docker-based test deployments where secrets were less consequential. The motivation was efficiency: one file per node, everything in one place, deploy and go.

But this efficiency came with a hidden cost. The settings files were being created in /tmp/ on the head node, then would be copied to the target nodes and placed in /opt/fgw/config/. While the assistant had set restrictive permissions on the wallet directory (chmod 700), the settings files themselves were world-readable during transit and at rest. The CIDgravity token—a credential that controls access to a paid API service—was exposed to anyone who could read those files.

The user's intervention forced a complete re-evaluation. The assistant had to answer a difficult question: how do you maintain operational efficiency while adhering to a strict vault-based security model? The answer, embodied in message 1993, was a separation of concerns: configuration data (which can be version-controlled and shared) versus secret data (which must be vaulted and loaded at runtime).

How Decisions Were Made: The Architecture of the Correction

Message 1993 is not simply a corrected file—it is a document that encodes several architectural decisions. The most visible decision is the removal of the CIDGRAVITY_API_TOKEN line from the settings file. But the more interesting decisions are implicit in what was added.

First, the assistant added a comment at the top of the file: # NOTE: CIDGRAVITY_API_TOKEN loaded from /home/fgw/.ribswallet/cidg.token. This is a documentation decision—a breadcrumb for any future operator who looks at this file and wonders where the token is configured. It also serves as a self-reminder for the assistant, which would later need to construct the systemd service file to actually load the token from that path.

Second, the assistant preserved the CIDgravity endpoint configuration lines (CIDGRAVITY_API_ENDPOINT_GBAP, CIDGRAVITY_API_ENDPOINT_GOCD, CIDGRAVITY_MAX_CONNECTIONS) in the settings file. This is a subtle but important architectural choice: non-secret configuration parameters remain in the shared settings file, while only the secret token itself is externalized. This separation allows the settings file to be safely stored in version control, reviewed, and shared across the team without exposing credentials.

Third, the assistant maintained the per-node database configuration (RIBS_YUGABYTE_SQL_DB="filecoingw_kuri_01") and per-node LocalWeb URLs (EXTERNAL_LOCALWEB_URL="http://10.1.232.83:7010"). These values differ between kuri_01 and kuri_02, confirming that the assistant intended to create separate settings files for each node rather than a single template. This is a pragmatic decision for a three-node QA cluster, though it would not scale well to dozens of nodes without a templating system like Ansible's Jinja2.

Assumptions Made: Visible and Hidden

The message reveals several assumptions, some correct and some questionable.

The assistant assumed that the token file at /home/fgw/.ribswallet/cidg.token would be created with appropriate permissions. In message 1992, the assistant had stated it would "store the token securely on the nodes (in a file with restricted permissions, read at runtime)." But the actual creation of that file on the target nodes is not shown in message 1993—it happens later in the session. The assistant is operating on the assumption that the token file will exist and be readable by the fgw user when the service starts.

The assistant assumed that the systemd service would be configured to load the token via ExecStartPre or EnvironmentFile pointing to the restricted token file. This assumption is correct in principle but requires careful implementation: the systemd service must be able to read the token file before the Kuri daemon starts, and the token must be injected into the environment in a way that the Go application can consume it via os.Getenv("CIDGRAVITY_API_TOKEN").

The assistant assumed that the YugabyteDB credentials (RIBS_YUGABYTE_SQL_USER="yugabyte" and RIBS_YUGABYTE_SQL_PASS="yugabyte") were acceptable to include in the settings file. This is a notable inconsistency: the CIDgravity token was deemed too sensitive for the settings file, but the database password was left in plaintext. The likely reasoning is that the YugabyteDB credentials are for a QA cluster with a default password, while the CIDgravity token is a real credential for a production service. But this distinction is not explicitly stated, and a security-conscious reviewer might flag this as a remaining vulnerability.

The assistant also assumed that the QA cluster's network was sufficiently isolated that plaintext database credentials posed minimal risk. This is a reasonable assumption for a three-node cluster on a private network, but it is still an assumption worth documenting.

Mistakes and Incorrect Assumptions

The most obvious mistake was the original inclusion of the CIDgravity token in the settings files—an error that was caught and corrected before any damage was done. But there are subtler issues worth examining.

The settings file in message 1993 contains the value RIBS_RETRIEVALBLE_REPAIR_THRESHOLD="2"—note the typo "RETRIEVALBLE" instead of "RETRIEVABLE." This typo was present in the original settings file and was carried forward unchanged. While the application might accept either spelling, such inconsistencies can lead to confusion during debugging and suggest a lack of careful validation.

The file also sets RIBS_S3API_AUTH_ENABLED="false", meaning the internal S3 API has no authentication. For a QA cluster this is acceptable, but if these settings files were ever used as a template for production deployment, this would be a significant security gap. The assistant did not add any comment flagging this as a QA-only setting.

More fundamentally, the assistant's approach to configuration management was ad-hoc. Rather than using the existing Ansible infrastructure (which had been developed and tested in earlier segments of the conversation), the assistant was creating files manually via cat and scp. This manual approach is fragile, unrepeatable, and prone to the very kind of error that occurred with the token. The user's security intervention indirectly highlighted a deeper problem: the assistant was bypassing its own automation tooling in the interest of speed.

Input Knowledge Required to Understand This Message

To fully understand message 1993, a reader needs knowledge spanning several domains.

First, one must understand the FGW architecture: the system consists of stateless S3 frontend proxies, Kuri storage nodes, and a YugabyteDB backend. Each Kuri node has its own SQL database and CQL keyspace, plus a shared S3 keyspace for object routing. The configuration file maps these relationships through environment variables.

Second, one must understand the CIDgravity service: it is a platform that helps Filecoin storage providers find the best deals by analyzing market conditions. The API token is a credential that authenticates the Kuri node to CIDgravity's API endpoints. Leaking this token could allow unauthorized usage of the CIDgravity service or expose the node's deal-making strategy.

Third, one must understand the security model: the assistant is implementing a "vault" pattern where secrets are stored in files with restricted permissions (mode 600 or 400) and loaded at runtime rather than stored in configuration files that might be version-controlled, backed up, or accidentally exposed. The wallet directory at /home/fgw/.ribswallet/ serves as this vault, with permissions set to 700 (directory) and 600 (files).

Fourth, one must understand the deployment context: this is a QA cluster on three physical nodes with IP addresses 10.1.232.82 (head node/YugabyteDB), 10.1.232.83 (kuri_01), and 10.1.232.84 (kuri_02). The configuration values (hosts, ports, paths) are specific to this network topology.

Output Knowledge Created by This Message

Message 1993 creates a corrected configuration file for the kuri_01 node. But the knowledge it produces extends beyond this single file.

It establishes a security pattern: secrets go in vault files, configuration goes in settings files. This pattern is then applied to kuri_02 (in a subsequent message) and to the systemd service configuration (later in the session). The pattern becomes the standard for the entire QA deployment.

It creates a documentation trail: the comment at the top of the file tells future operators where to find the token. This is a form of institutional knowledge that prevents the same mistake from being repeated.

It implicitly defines a trust boundary: the settings file can be shared, committed to version control, and reviewed without exposing secrets. The vault file is restricted to the individual node and must be managed separately. This separation enables safer collaboration and deployment workflows.

It also creates a dependency: the Kuri service now depends on the token file existing at a specific path before it can start. This dependency must be managed during provisioning (the token must be placed before the service starts) and during recovery (if the token file is lost, the service will fail to authenticate with CIDgravity).

The Thinking Process: What the Message Reveals About Reasoning

The structure of message 1993 reveals the assistant's thinking process in several ways. The file is organized into clearly labeled sections: Node Identification, RIBS Data, Filecoin API, CIDGravity, Deal Settings, YugabyteDB, S3 Shared Keyspace, S3 API, LocalWeb, Internal API, Prometheus Metrics, Logging, and IPFS Path. This organization reflects a systematic approach to configuration, grouping related parameters together for readability and maintainability.

The prominent comment at the top—"NOTE: CIDGRAVITY_API_TOKEN loaded from /home/fgw/.ribswallet/cidg.token"—suggests that the assistant was consciously thinking about the security lesson it had just learned. It wanted to make the correction visible and explicit, not just silently remove the token. This is a sign of an assistant that is learning from its mistakes and documenting its reasoning for future reference.

The decision to create the file in /tmp/ and then copy it to the target node (visible in the bash command structure) shows that the assistant was thinking in terms of a build-and-deploy workflow: construct the artifact locally, then transfer it to the destination. This is efficient for a single file but would benefit from being formalized into a proper CI/CD pipeline.

The preservation of all non-secret configuration values, unchanged from the original file, indicates that the assistant was making a minimal correction rather than a full rewrite. It changed only what was necessary to fix the security violation, leaving the rest of the configuration intact. This is a conservative approach that reduces the risk of introducing new errors.

Conclusion: A Small Message with Large Implications

Message 1993 is, on its surface, a simple configuration file creation. But it represents a moment of learning and correction in the deployment of a distributed storage system. It is the artifact of a security principle being translated into practice, of an automated assistant being corrected by a human operator, and of infrastructure code being improved through iteration.

The message teaches us that security is not a feature that can be added after the fact—it must be baked into the architecture from the start. The separation of secrets from configuration, the use of restricted-permission vault files, and the documentation of security decisions are all practices that elevate a deployment from "working" to "enterprise-grade."

It also teaches us that the most valuable moments in infrastructure work are often the corrections, not the original creations. The mistake of embedding a token in a settings file, caught and corrected, is worth more than a hundred flawless deployments. It is in these moments of failure and recovery that the deepest architectural insights emerge.

For anyone deploying distributed systems—whether on Filecoin, AWS, or bare metal—the lesson is clear: vault your secrets, separate your concerns, and never assume that speed justifies security shortcuts. The three-node QA cluster that received this corrected configuration file would go on to function correctly, serving as a testament to the power of getting the fundamentals right.