The Seven Words That Saved a Deployment: Why "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS" Matters

Subject Message (Index 1991): [user] Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS

In a sprawling coding session spanning dozens of messages, infrastructure provisioning commands, and iterative debugging cycles, one brief user intervention stands out as a pivotal moment of correction. The message is only seven words long, yet it carries the weight of an entire security philosophy: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS." To understand why this single message matters—and why it deserves deep analysis—we must examine the context that led to it, the assumptions it shattered, and the architectural redesign it triggered.

The Moment Before: A Security Blind Spot

The assistant had been working diligently to deploy a QA test cluster for the FGW (Filecoin Gateway) distributed storage system across three physical nodes. The architecture involved a head node running YugabyteDB and two storage nodes (kuri1 and kuri2) running the Kuri daemon. After building binaries, setting up users and directories, and deploying wallet files, the assistant reached the configuration stage.

The user had previously provided two critical pieces of information: the wallet directory at ~/.ribswallet and the CIDgravity API token stored in ~/.ribswallet/cidg.token. The assistant, moving with the momentum of automation, created two configuration files—settings-kuri1.env and settings-kuri2.env—that contained the CIDgravity API token embedded directly in plaintext. The token was pasted verbatim into the CIDGRAVITY_API_TOKEN environment variable, ready to be copied to the remote nodes and stored in /data/fgw/config/settings.env.

From a purely functional standpoint, this approach worked. The Kuri daemon would read the environment file, find the token, and authenticate with CIDgravity's API. The deployment would succeed. The cluster would run. But from a security standpoint, it was a disaster waiting to happen.

The Intervention: Why the User Spoke Up

The user's message was not a question, a suggestion, or a gentle nudge. It was an imperative: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS." The capitalization of "DO NOT" and the use of "VAULTS" (rather than "secure storage" or "a safe place") reveals several layers of intent.

First, the word "VAULTS" is specific. It signals that the user operates with a mental model where secrets have designated, secure storage locations—vaults—that are separate from configuration files. This is not merely about "being careful" with secrets; it is about a principled architectural separation between configuration (which can be version-controlled, shared, and inspected) and secrets (which must be access-controlled, encrypted, and audited).

Second, the timing matters. The user said "Wait" before the assistant deployed the insecure files. The assistant had created the files locally on the head node but had not yet copied them to the remote nodes. The user caught the mistake in the narrow window between creation and deployment. Had the user not been paying close attention, the token would have been written to disk on three physical nodes in a world-readable configuration directory.

Third, the user's phrasing reveals frustration and urgency. The assistant had already demonstrated awareness of security practices earlier in the session—setting restrictive permissions on wallet files, using chmod 600 and chmod 700—yet had inexplicably abandoned those practices when it came to the configuration files. The user's intervention was necessary to realign the assistant's behavior with the project's security standards.

The Assumption That Nearly Broke Security

The assistant's mistake stemmed from a subtle but dangerous assumption: that a QA/test environment does not require production-grade secret management. This is a common pitfall in infrastructure work. The reasoning often goes: "This is just a test cluster. The token is already on the head node. We'll clean it up later. The important thing is getting it working."

But this assumption is flawed on multiple levels. QA environments often contain real credentials because they need to interact with real services. The CIDgravity API token was not a test token—it was the production token used to communicate with CIDgravity's service for deal-making and provider selection. Exposing it in a config file on any node meant that anyone who gained access to that node could use the token. Furthermore, configuration files are frequently backed up, version-controlled, or shared between environments. A token embedded in settings-kuri1.env could easily find its way into a Git repository, a backup archive, or a developer's local machine.

The assistant also made a second, more subtle assumption: that the user would not notice or would not care about the security of a QA deployment. This assumption was wrong. The user's immediate reaction demonstrated that security standards apply uniformly across environments, not just in production.

The Corrective Chain Reaction

The user's message triggered an immediate and complete reversal. The assistant's next actions tell the story:

Acknowledgment and apology: "You're absolutely right - I apologize for that security mistake."
Removal of insecure files: rm -f /tmp/settings-kuri1.env /tmp/settings-kuri2.env
Redesign of the configuration approach: The token was removed from the settings files entirely.
Creation of a separate vault file: The token was placed in /home/fgw/.ribswallet/cidg.token with permissions 600 (owner-read/write only).
Runtime loading via systemd: The systemd service file was designed to read the token from the vault file at startup using ExecStartPre commands, ensuring the token never appears in a config file. This redesign embodies the principle of "separation of concerns" applied to security. Configuration files contain settings that describe how the system should behave. Vault files contain secrets that authenticate who the system is. These are fundamentally different categories of data and should be stored differently, protected differently, and managed differently.

Input Knowledge Required

To understand this message fully, one needs to know several things:

What a "vault" means in infrastructure context: A vault is a dedicated, access-controlled storage system for secrets. It can be a file with restricted permissions (as in this case), a HashiCorp Vault instance, or a cloud secret manager. The key property is that secrets are not mixed with configuration.
The nature of the CIDgravity API token: This token authenticates the FGW cluster with CIDgravity's API for deal-making. If compromised, an attacker could impersonate the cluster and potentially interfere with storage deals.
The deployment architecture: Configuration files on the kuri nodes would be stored in /data/fgw/config/, which, while owned by the fgw user, could be read by any process or user with appropriate permissions. The assistant had initially set chmod 600 on the settings file, but the token was still embedded in a file that could be accidentally included in backups, logs, or debugging output.
The assistant's prior security awareness: Earlier in the session, the assistant had correctly set restrictive permissions on wallet files and directories, showing that the security practices existed in the codebase but were inconsistently applied.

Output Knowledge Created

The user's message produced several lasting outcomes:

A corrected configuration pipeline: The settings files for both kuri nodes were regenerated without the token, with a clear comment indicating where the token would be loaded from.
A vault file pattern: The token was stored in a dedicated file within the existing .ribswallet directory, reinforcing the pattern that wallet-related secrets belong in the wallet directory.
A runtime loading pattern: The systemd service files (created in subsequent messages) used ExecStartPre to read the token from the vault file and export it as an environment variable before starting the Kuri daemon. This pattern is reusable for any future secrets.
A documented security standard: The user's intervention implicitly established that all secrets in this project must be vaulted, regardless of environment (QA, staging, or production).

The Deeper Lesson: Security Is Not Optional in QA

The most important insight from this message is that security practices cannot be relaxed for test environments. QA clusters often have weaker network isolation, less monitoring, and more casual access than production environments—making them more vulnerable, not less. A token stored in plaintext on a QA node is arguably more dangerous than the same token in production, because the QA environment is more likely to be compromised or misconfigured.

The user understood this instinctively. The assistant, focused on the mechanics of deployment and the urgency of getting the cluster running, did not. The seven-word correction was not just about moving a token from one file to another—it was about instilling a security discipline that would prevent future incidents across the entire project lifecycle.

Conclusion

"Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS" is a masterclass in concise, effective feedback. It identifies a specific mistake, states a universal principle, and implies the correct action—all in seven words. The message worked because the user had a clear mental model of security architecture, recognized the violation immediately, and communicated the correction with enough urgency to stop the deployment in its tracks. For the assistant, it was a humbling but invaluable lesson: in infrastructure automation, getting it working is not enough. It must be secure from the start, even in a QA cluster, even for a single token, even when it seems like no one will notice.