The $export Trap: When systemd's EnvironmentFile Silently Rejects Shell Conventions

A Single Line of Debugging That Unraveled an Infrastructure Assumption

In the middle of a grueling session debugging Ansible deployment scripts for a distributed Filecoin Gateway (FGW) cluster, the assistant issued a message that, on its surface, seems almost trivial: "I need to remove all export prefixes from the template since systemd's EnvironmentFile doesn't support them." The message then reports writing a single file—a Jinja2 template for environment variables. But this brief intervention represents the culmination of a much longer debugging chain, a moment where a subtle mismatch between two different interpretation contexts (shell scripting vs. systemd configuration) was finally identified and corrected. Understanding why this message was written, what assumptions it challenged, and what knowledge it created reveals the nature of infrastructure-as-code debugging at its most granular.

The Message

[assistant] I need to remove all export prefixes from the template since systemd's EnvironmentFile doesn't support them. Let me fix this: [write] /home/theuser/gw/ansible/roles/kuri/templates/settings.env.j2 Wrote file successfully.

The Debugging Chain That Led Here

To understand why this message exists, one must trace the events immediately preceding it. The assistant had been iterating through a Docker-based test harness that validates Ansible playbooks for deploying FGW clusters. The test had reached the point of deploying Kuri storage nodes—the core distributed storage daemons—and was failing. The test harness reported that the kuri-kuri-01.service systemd unit was in a perpetual restart loop, exiting with code 1.

The assistant checked the service status and journal logs (message 1578) and found the critical clue:

Jan 31 17:46:46 kuri-01 systemd[1]: kuri-kuri-01.service: Ignoring invalid environment assignment 
'export RIBS_MINIMUM_RETRIEVABLE_COUNT=1': /data/fgw/kuri-01/config/settings.env

Systemd was explicitly logging that it was ignoring every line in the environment file because each line began with the export keyword. The EnvironmentFile= directive in systemd unit files expects a specific format: plain KEY=VALUE lines, with no shell syntax whatsoever. No export, no quotes beyond the value delimiter, no variable expansion. This is a well-documented but easily overlooked constraint—especially when the same file is intended to be sourced by a shell during initialization.

The assistant then read the template file (message 1579) and confirmed the problem: every variable assignment in settings.env.j2 was prefixed with export, as is conventional in shell environment files meant to be sourced with . or source. The template looked like:

export FGW_NODE_ID="{{ fgw_node_id }}"
export FGW_NODE_TYPE="{{ fgw_node_type }}"
export RIBS_MINIMUM_RETRIEVABLE_COUNT=1

This is perfectly valid shell syntax. When a user runs source settings.env in bash, the export keyword ensures the variables are propagated to child processes. But systemd's parser is not a shell—it reads the file line by line, splits on the first =, and rejects anything that doesn't match its strict format. Every line starting with export was silently discarded, meaning critical variables like IPFS_PATH, FGW_NODE_ID, and database connection strings were never set. The Kuri daemon then started with default values, looked for IPFS data in the wrong directory (/home/fgw/.ipfs instead of /data/fgw/kuri-01/ipfs), and crashed.

The Decision: Remove Export, But Handle the Consequence

The assistant's decision to remove all export prefixes from the template was the obvious fix for the systemd issue, but it created a secondary problem: the same file was also used during the kuri init step, where it was sourced by a shell to set up the environment before running the initialization command. Without export, sourcing the file would set the variables in the current shell, but they wouldn't be inherited by child processes—including the kuri init command itself.

This tension between two consumers of the same file—systemd's EnvironmentFile= (which forbids export) and bash's source (which needs export for child processes)—is the core architectural challenge. The assistant recognized this and, in the very next message (message 1582), addressed the consequence:

The problem is that: 1. systemd needs plain KEY=VALUE format (no export) 2. But when we source the file in bash for kuri init, we need export to actually export the vars

>

Let me update the init task to use export explicitly when sourcing

The solution was to make the template use plain KEY=VALUE format (satisfying systemd), and then modify the Ansible task that runs kuri init to explicitly use export when sourcing—for example, by running set -a; source settings.env; set +a or by wrapping the init command with explicit variable exports. This two-pronged approach correctly serves both consumers from a single source file.

Assumptions and Mistakes

Several assumptions were embedded in the original design, and the debugging process exposed each one:

Assumption 1: "Environment files are shell scripts." The developer who created the template likely thought of settings.env as a shell script fragment, following the widespread convention of using export in .env files. Many tools (docker-compose, direnv, various dotenv libraries) accept or even require the export prefix. But systemd's EnvironmentFile= is not one of them.

Assumption 2: "One file can serve both systemd and shell sourcing without modification." This is a reasonable aspiration—DRY (Don't Repeat Yourself) principles discourage maintaining two separate environment files. But the differing syntax requirements mean that either the file must be transformed between uses, or the shell sourcing must be adapted to handle the non-export format.

Assumption 3: "The error would be obvious." The systemd log message "Ignoring invalid environment assignment" is actually quite clear, but it's easy to miss in a wall of journal output. The assistant had to specifically check the service status and journal logs to find this needle in the haystack.

Mistake: Not testing the systemd unit format earlier. The environment template was created as part of a large batch of Ansible role files, and the first test execution revealed the problem. A unit test or a simple validation step—like parsing the generated file with a mock systemd parser—could have caught this before the test harness run.

Input Knowledge Required

To understand and diagnose this issue, the assistant needed:

  1. Knowledge of systemd unit file syntax, specifically the EnvironmentFile= directive and its strict KEY=VALUE format requirement. This is documented in systemd.exec(5) but is a detail that many developers overlook.
  2. Knowledge of shell environment semantics, including the difference between export and non-export variable assignments, and how source propagates variables to child processes.
  3. Knowledge of the Ansible Jinja2 templating system and how settings.env.j2 is rendered into settings.env on the target host.
  4. Knowledge of the Kuri daemon's startup behavior, including which environment variables it reads (IPFS_PATH, FGW_NODE_ID, database connection strings) and what happens when they are missing.
  5. Knowledge of the Docker test harness infrastructure, including how to execute commands inside containers (docker compose exec), check systemd service status, and read journal logs.
  6. Knowledge of the overall deployment architecture: that settings.env is consumed both by systemd (at service start) and by Ansible tasks (during kuri init), and that both paths must work correctly.

Output Knowledge Created

The fix created several valuable pieces of knowledge:

  1. A corrected Jinja2 template (settings.env.j2) that uses plain KEY=VALUE format, compatible with systemd's EnvironmentFile= directive.
  2. A modified Ansible task (in the subsequent message) that adapts the shell sourcing to use explicit export when running kuri init, bridging the gap between the two consumers.
  3. A documented constraint: the team now knows that any future environment variables added to this template must avoid shell syntax, and that the kuri init task must handle the export explicitly.
  4. A debugging pattern: the journal log message "Ignoring invalid environment assignment" is now a known diagnostic signal for this class of problem. Future developers encountering similar failures will know to check for export prefixes in environment files consumed by systemd.
  5. A hardened test harness: the Docker-based test suite successfully caught this bug, validating its role as a regression detection mechanism for deployment infrastructure.

The Broader Pattern

This debugging episode illustrates a recurring theme in infrastructure engineering: the tension between human-friendly conventions and machine-parser strictness. Shell environment files with export prefixes are natural for humans to write and read—they look like shell scripts, they work with source, and they're familiar. But systemd's parser is not a shell; it's a line-oriented configuration reader that makes no allowances for syntactic sugar. The mismatch is a classic "leaky abstraction" problem, where the convenience of one context (shell scripting) leaks into another context (systemd configuration) where it breaks.

The fix—removing export from the template and adjusting the shell sourcing to compensate—is a small change with a large impact. It makes the environment file format explicit about its primary consumer (systemd), and it pushes the adaptation logic to the secondary consumer (the init task), where it belongs. This is the right architectural choice: the file format should match the strictest consumer, and other consumers should adapt.

Conclusion

The assistant's message at index 1580 is a single line of intent followed by a file write operation. But it represents the resolution of a debugging chain that traversed container logs, systemd journal entries, Jinja2 templates, Ansible task ordering, and shell semantics. The fix was small—removing the word export from a template—but the reasoning that led to it required deep knowledge of multiple systems and the ability to trace a failure from "service won't start" to "environment variable not set" to "systemd parser rejected the format." This is the essence of infrastructure debugging: finding the one line in a thousand that silently fails, understanding why it fails, and applying the minimal correction that makes the entire system converge to a working state.