The Subtle Art of Container Recreation: Debugging Environment Caching in Docker Compose

In a sprawling distributed systems debugging session spanning multiple days, one seemingly minor message captures a critical moment of diagnostic reasoning. The assistant, having just committed a CQL batcher and loadtest improvements to a horizontally scalable S3 architecture, receives a simple question from the user: "Does ./docker-compose.yml still work for single-node mode?" What follows is a chain of troubleshooting that culminates in a single, deceptively simple insight about how Docker Compose handles environment file changes.

The subject message reads:

Still failing. The container might have cached the old env. Let me recreate it: `` cd /home/theuser/gw && docker compose up -d --force-recreate filecoingw && sleep 10 && docker logs gw-filecoingw-1 2>&1 | tail -20 ``

This message, brief as it is, represents a moment where the assistant correctly identifies that Docker Compose's restart command does not re-read environment files, and that a full container recreation is required. Understanding why this matters—and what led to it—requires tracing the debugging path that preceded it.

The Road to Single-Node Validation

The session had been focused on building and debugging a multi-node test cluster with three layers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB database. After extensive work—including correcting a fundamental architectural error where Kuri nodes were mistakenly used as direct S3 endpoints—the assistant had committed significant changes: a CQL batcher for high-throughput metadata writes, loadtest improvements that distinguished timeouts from actual corruption, and test-cluster configuration fixes.

But the user's question about the root docker-compose.yml (the one in the project root, not the test-cluster subdirectory) revealed that this simpler single-node configuration had been neglected. The assistant dutifully checked it and immediately hit a problem: the configuration file data/config/settings.env did not exist. Docker Compose requires this file to be present—it is referenced in the compose file via env_file—and without it, docker compose config refused to validate.

The assistant created a minimal settings.env based on the test-cluster's per-node configuration files. The config validated. But when the containers started, the Kuri node (named filecoingw in this compose file) failed to initialize with an error about a missing external offload module. Investigation revealed that the EXTERNAL_LOCALWEB_URL environment variable was required for the LocalWeb external storage module to initialize. The assistant added it to settings.env and ran docker compose restart filecoingw.

The Failure That Triggered the Insight

The restart command produced a confusing set of errors:

Error: ipfs configuration file already exists!
Reinitializing would overwrite your keys
Configuration load failed: %w invalid log level:

The IPFS error was expected—the container had already initialized IPFS on the first run, and restarting shouldn't reinitialize it. But the "invalid log level" error was more troubling. It suggested that the configuration was being loaded incorrectly, possibly from a cached or corrupted state.

This is where the subject message's reasoning becomes visible. The assistant's hypothesis—"The container might have cached the old env"—is a sophisticated diagnostic leap. Docker Compose's restart command stops and starts the same container. It does not destroy and recreate it. The environment variables that were injected when the container was first created (from the original settings.env without EXTERNAL_LOCALWEB_URL) persist in the container's metadata. Even though the assistant had updated the settings.env file on disk, the running container was still using the old values.

The decision to use --force-recreate instead of restart was therefore not arbitrary. It was based on a specific understanding of Docker Compose's lifecycle: restart reuses the existing container object (keeping its environment), while up --force-recreate destroys the old container and creates a new one, which causes Docker Compose to re-read the env_file and inject the updated variables.

Assumptions and Their Validity

The assistant made several assumptions in this message, most of which proved correct:

First, that the environment variables were indeed cached in the container metadata. This is accurate—Docker stores the environment at container creation time, and restarting does not refresh them. The env_file directive in Docker Compose is evaluated at docker compose up/create time, not at start/restart time.

Second, that --force-recreate would resolve the issue. This was correct—the subsequent message (msg 1419) shows the assistant declaring "Now it's working" and running a sanity test.

Third, that the only remaining problem was environment caching. This assumption was partially incorrect. While the force-recreate did resolve the startup failure, it revealed a new problem: the Kuri storage node's S3 server was rejecting PUT requests with "invalid content sha256" errors. The loadtest tool was not sending the x-amz-content-sha256: UNSIGNED-PAYLOAD header that the storage node's S3 handler required. This was a separate issue that had been masked by the environment problem.

Input Knowledge Required

To understand this message, one needs knowledge of Docker Compose's container lifecycle model. The distinction between restart (which preserves the container's configuration) and up --force-recreate (which rebuilds the container from scratch) is crucial. Without this understanding, the assistant's decision to force-recreate might seem like cargo-cult debugging—randomly trying different commands.

Additionally, one needs to understand the project's architecture: that the root docker-compose.yml runs a single Kuri storage node directly (not behind an S3 proxy), and that this node reads its configuration from environment variables sourced from settings.env. The assistant had been working with a multi-node test-cluster that used per-node settings files and a separate S3 proxy layer, so the single-node setup was a different configuration path.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

  1. Docker Compose does not re-read env_file on restart. This is a concrete operational insight that applies to any project using Docker Compose with external environment files.
  2. The single-node configuration required EXTERNAL_LOCALWEB_URL to be set, and this variable was not present in the initial settings.env the assistant created.
  3. After force-recreating, the Kuri node started successfully, confirming that the environment caching hypothesis was correct.
  4. The successful startup revealed a second issue: the storage node's S3 handler validates the x-amz-content-sha256 header, which the loadtest tool was not sending. This led to a subsequent fix where the loadtest was updated to include the UNSIGNED-PAYLOAD header.

The Thinking Process

The reasoning visible in this message follows a classic debugging pattern: observe a failure, form a hypothesis about its cause, test the hypothesis with a targeted intervention, and observe the result. The key insight—that environment variables are cached at container creation time—came from understanding Docker Compose's internals rather than from any error message. The assistant didn't see a "cached environment" warning; they inferred the caching from the behavior.

The phrase "Still failing" connects back to the previous restart attempt, showing that the assistant is tracking the state across multiple commands. The hedge "might have cached the old env" shows appropriate uncertainty—the assistant is proposing a hypothesis, not stating a certainty. The decision to use --force-recreate is the test of that hypothesis.

The message also shows operational discipline: the assistant chains commands with && so that the logs are only shown if the recreate succeeds, and uses sleep 10 to give the container time to initialize before reading logs. The tail -20 ensures they see the most recent output without being overwhelmed by the full log.

Broader Significance

This message, while small, illustrates a fundamental challenge in distributed systems debugging: the difference between a service appearing to restart and actually reinitializing with new configuration. In complex systems with multiple configuration sources—environment variables, config files, databases, network state—it is easy to assume that a restart is sufficient when a full recreation is required.

The assistant's willingness to question whether the container was truly picking up the new configuration, rather than assuming that restart would work, is the kind of skepticism that separates effective debugging from frustrating trial-and-error. It is a reminder that in systems programming, the most valuable tool is not a debugger or a profiler, but a precise mental model of how the system's components actually behave.