The Art of Secret Injection: A Systemd Service Architecture for Runtime Token Loading

Introduction

In the sprawling landscape of distributed systems deployment, few moments carry as much weight as the one captured in message 1997 of this coding session. What appears at first glance to be a routine systemd service file creation is, in reality, a carefully constructed response to a security failure that occurred only minutes earlier. This message represents the culmination of a rapid learning cycle: the assistant made a mistake, the user corrected it, and the assistant designed an elegant, production-grade solution that transformed a security vulnerability into a hardened deployment pattern.

The message itself is deceptively simple—two nearly identical systemd unit files deployed to two remote storage nodes (kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84). But embedded within those unit files is a sophisticated secret injection mechanism that separates configuration from credentials, enforces runtime-only token loading, and applies systemd-level security hardening. This is the story of how a single user admonishment—"DO NOT STORE SECRETS OUTSIDE OF VAULTS"—rippled through the assistant's decision-making and produced a design worth studying.

The Full Text of the Message

The assistant wrote:

Now create the systemd service files that load the token from the secure file:

``` [bash] ssh 10.1.232.83 "sudo tee /etc/systemd/system/kuri.service > /dev/null << 'EOF' [Unit] Description=FGW Kuri Storage Node After=network-online.target Wants=network-online.target

[Service] Type=simple User=fgw Group=fgw WorkingDirectory=/data/fgw

# Load environment from config file EnvironmentFile=/data/fgw/config/settings.env

# Load token from secure file ExecStartPre=/bin/bash -c 'echo CIDGRAVITY_API_TOKEN=\$(cat /home/fgw/.ribswallet/cidg.token) > /run/fgw/token.env' EnvironmentFile=-/run/fgw/token.env

ExecStart=/opt/fgw/bin/kuri daemon Restart=always RestartSec=10

# Security hardening NoNewPrivileges=true ProtectSystem=strict ProtectHome=read-only ReadWritePaths=/data/fgw /var/log/fgw RuntimeDirectory=fgw RuntimeDirectoryMode=0700

# Resource limits LimitNOFILE=1048576 MemoryMax=24G

[Install] WantedBy=multi-user.target EOF echo 'kuri1 service created'" ```

(The identical unit file was then deployed to kuri2 at 10.1.232.84.)

The Immediate Context: A Security Breach and Its Correction

To understand why this message was written, we must rewind to the events that precipitated it. Just a few messages earlier, the assistant had been building configuration files for the FGW (Filecoin Gateway) distributed storage system's kuri nodes. The assistant created settings.env files for both nodes that contained every configuration parameter the kuri daemon needed—including the CIDgravity API token embedded directly in plaintext.

The CIDgravity token is a sensitive credential that authenticates the storage node with the CIDgravity marketplace, which handles deal-making and provider selection on the Filecoin network. Exposing this token in a world-readable configuration file would be a serious security lapse. The user caught this immediately with a sharp rebuke: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS."

The assistant's response in message 1992 was immediate and unambiguous: "You're absolutely right - I apologize for that security mistake." The assistant deleted the insecure files and outlined a two-part plan: store the token in a file with restricted permissions, and reference it at runtime via environment variable expansion in the systemd service.

Messages 1993 through 1995 executed the first part of that plan. The assistant created new settings.env files that explicitly omitted the token, adding a comment noting it would be loaded separately. Then the assistant deployed these config files to both nodes with chmod 600 permissions, and separately deployed the CIDgravity token to /home/fgw/.ribswallet/cidg.token on each node, also with chmod 600 and owned by the fgw user.

Message 1997 completes the second part of the plan: the systemd service files that wire everything together.

Architectural Decisions Embedded in the Service Design

The systemd unit files in this message encode several deliberate architectural decisions that reveal the assistant's thinking process:

1. The Two-Layer Environment Loading Strategy

The most interesting design choice is the dual EnvironmentFile mechanism. The service loads configuration from /data/fgw/config/settings.env using a standard EnvironmentFile directive. This file contains all the non-sensitive configuration: node IDs, database connection parameters, API endpoints, port bindings, and logging levels. But crucially, it contains no secrets.

The token is loaded through a separate mechanism. An ExecStartPre hook runs a bash command that reads the token from the restricted file at /home/fgw/.ribswallet/cidg.token and writes it to a runtime file at /run/fgw/token.env. The leading dash in EnvironmentFile=-/run/fgw/token.env tells systemd to ignore the file if it doesn't exist—a graceful degradation pattern that prevents service startup failure if the token file hasn't been created yet.

This two-layer approach is significant because it separates the concerns of configuration management from secret management. The settings file can be version-controlled, backed up, and audited without exposing secrets. The token file lives in a separate, tightly restricted location and is only read at service startup.

2. The Runtime Directory as a Secure Scratch Space

The assistant specified RuntimeDirectory=fgw with RuntimeDirectoryMode=0700. This tells systemd to create /run/fgw/ at boot time with permissions 0700 (owner-only access), owned by the fgw user and group. The token environment file is written into this directory by the ExecStartPre script.

This is a clever use of systemd's runtime directory feature. The /run filesystem is a tmpfs (temporary filesystem stored in memory), which means the token file never touches persistent storage. It exists only in memory for the lifetime of the service. When the service stops or the system reboots, the file disappears. This is a textbook example of the principle of least privilege applied to secret lifetimes.

3. Security Hardening Directives

The service file includes four systemd security hardening options that the assistant likely selected deliberately:

NoNewPrivileges=true: Prevents the kuri process and its children from gaining any new privileges through setuid/setgid binaries or capabilities. This contains potential damage if the process is compromised.
ProtectSystem=strict: Makes most of the filesystem read-only for the service, with only explicitly listed paths being writable. This prevents an attacker from modifying system binaries or configuration even if they gain code execution within the kuri process.
ProtectHome=read-only: Makes /home read-only. This is particularly interesting because the token file lives in /home/fgw/.ribswallet/cidg.token. The service can read the token at startup (via the ExecStartPre script, which runs before the security policies are fully applied), but the running daemon cannot modify it.
ReadWritePaths=/data/fgw /var/log/fgw: Explicitly grants write access only to the data directory and log directory. The kuri daemon needs to write blocks, CAR files, and logs, but nothing else. These hardening directives transform the service from a simple daemon launcher into a sandboxed execution environment. The assistant was thinking not just about getting the service running, but about containing potential damage if it were ever compromised.

4. Resource Limits for Production Readiness

The LimitNOFILE=1048576 and MemoryMax=24G directives set resource limits appropriate for a storage node that may handle thousands of concurrent connections and large data volumes. The file descriptor limit of 1,048,576 (2^20) is generous but reasonable for a node serving S3 API requests. The 24 GB memory limit prevents the process from consuming all system memory in case of a memory leak or unexpected load spike.

What the Assistant Assumed

Every design carries assumptions, and this message is no exception:

The assistant assumed the token file already existed at the specified path. This was a safe assumption because messages 1993-1995 had just deployed the token to that exact location. But the assumption is baked into the service design: if the file doesn't exist, the ExecStartPre script will fail, and the service will not start.

The assistant assumed the /run/fgw/ directory would be created by systemd. The RuntimeDirectory=fgw directive ensures this, but only if the systemd version supports it (it has been available since systemd 235, released in 2017). On older systems, this directive would be silently ignored, and the ExecStartPre script would fail trying to write to a non-existent directory.

The assistant assumed both nodes had identical configurations. The two service files are byte-for-byte identical, which is correct because the node-specific configuration (database keyspace, node ID, port numbers) lives in the settings.env file, not in the service unit. The service file is generic; the configuration file is specific.

The assistant assumed the kuri binary was already deployed to /opt/fgw/bin/kuri. This was established earlier in the session when the assistant built the binary and copied it to both nodes.

What the Message Created: Output Knowledge

This message produced concrete, deployable artifacts: two systemd service files installed on remote machines. But it also created several forms of output knowledge:

A reusable secret injection pattern: The two-layer environment loading with ExecStartPre is a pattern that can be applied to any systemd-managed service that needs to load secrets from restricted files at runtime.
A security baseline for the kuri service: The combination of NoNewPrivileges, ProtectSystem, ProtectHome, and ReadWritePaths establishes a minimum security posture that can be refined as the deployment matures.
Documentation of the token loading mechanism: The comment in the settings file ("NOTE: CIDGRAVITY_API_TOKEN loaded from /home/fgw/.ribswallet/cidg.token") and the ExecStartPre script together serve as self-documenting infrastructure. Any operator reading these files can trace the token's origin.
A foundation for observability: With Restart=always and RestartSec=10, the service will automatically recover from crashes, and the logging configuration (referenced in settings.env) will capture diagnostic output.

Mistakes and Incorrect Assumptions

While this message represents a significant improvement over the earlier insecure approach, it is not without potential issues:

The token is still written to a file on disk (in /run/fgw/token.env), albeit temporarily. An attacker with root access could read this file before the service starts or after it crashes. A more secure approach would be to pass the token directly as an environment variable in the ExecStart command, avoiding any intermediate file. However, systemd's ExecStart does not support command substitution in the same way, so the file-based approach is a practical compromise.

The ExecStartPre script runs as root. The systemd service runs as fgw user, but ExecStartPre hooks execute before the user switching occurs. This means the bash command that reads the token file runs with root privileges. If the token file path could be manipulated (e.g., via a symlink attack), this could be exploited. The assistant mitigated this by setting strict permissions on the token file and its parent directory, but the architectural risk remains.

The service does not validate the token before starting. If the token file is empty, corrupted, or contains a newline character that breaks the environment file syntax, the service will start with a malformed or missing token. The kuri daemon would then fail to authenticate with CIDgravity, but the service would appear healthy from systemd's perspective (since the process started successfully). Adding a validation step in ExecStartPre or in the kuri daemon itself would improve robustness.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the service file. The ordering of directives tells a story:

First, the assistant establishes the basic service metadata (description, dependencies). Then the execution context (user, group, working directory). Then the environment loading, which is the core architectural challenge this message solves. The ExecStartPre + EnvironmentFile combination is placed before the ExecStart to make the dependency chain explicit: the token must be loaded before the daemon starts.

The security hardening directives come next, forming a protective shell around the running process. The assistant likely thought: "If we're going to run a network-facing daemon that handles sensitive data, we need to limit what it can do if compromised."

Finally, the resource limits and installation target complete the service definition. The overall structure follows a logical flow: identity → environment → execution → security → resources → installation.

Conclusion

Message 1997 is a case study in how a single user correction can transform a routine deployment task into a thoughtfully engineered solution. The assistant could have simply removed the token from the config file and called it done. Instead, it designed a multi-layered secret injection system that separates configuration from credentials, uses memory-backed runtime directories for ephemeral secrets, applies systemd security hardening, and documents its own design through comments and structure.

The message also reveals something about the assistant's operating model: it learns from mistakes and incorporates feedback into subsequent actions. The original sin of embedding a secret in a config file was corrected not by a minimal fix, but by a comprehensive redesign that addressed the root cause—the lack of a secure secret delivery mechanism. The systemd service file is the final piece of that redesign, the component that wires the secure token file into the running daemon's environment.

For anyone deploying distributed systems with systemd, this message offers a practical pattern for runtime secret injection. It is not the most sophisticated approach possible—dedicated secret vaults like HashiCorp Vault or Kubernetes Secrets would be more robust in large-scale deployments—but for a three-node QA cluster, it strikes an excellent balance between security, simplicity, and maintainability. The assistant's willingness to tear down and rebuild when shown a better way is, in the end, the most valuable pattern visible in this message.