The Art of Parameterization: A Docker Compose Refactoring in a Distributed S3 Test Cluster

Introduction

In software engineering, some of the most instructive moments come not from grand architectural designs but from small, precise corrections that reveal deeper assumptions about how systems should be built. This article examines a single message from an opencode coding session—message index 197—in which an assistant updates a docker-compose.yml file to replace hardcoded paths with environment variable substitution. On its surface, the message appears trivial: a one-line reasoning note followed by a file write operation. Yet this small change sits at the intersection of several important software engineering themes: the tension between convenience and portability, the iterative refinement of infrastructure code, and the importance of listening to user feedback about configuration design.

The message reads in its entirety:

[assistant] ## Agent Reasoning I need to update the docker-compose.yml to use environment variables for the data directory paths. Let me replace the hardcoded paths with variable substitution. [write] /home/theuser/gw/test-cluster/docker-compose.yml Wrote file successfully.

To understand why this message matters, we must examine the conversation that led to it, the decisions embedded in its brevity, and the assumptions it both corrects and perpetuates.

The Context: Building a Test Cluster for Horizontally Scalable S3

The broader coding session concerns the implementation of a horizontally scalable S3-compatible storage architecture for a Filecoin Gateway system. The architecture follows a clean three-layer design: stateless S3 frontend proxies handle request routing and load balancing, backend Kuri storage nodes maintain independent blockstore data, and a shared YugabyteDB instance coordinates object placement metadata.

Earlier in the conversation, the user had asked a straightforward question: "How do I run a test cluster with 2x2 nodes, data in /data/fgw-test/?" The assistant responded by creating a comprehensive test cluster infrastructure in /home/theuser/gw/test-cluster/, including a docker-compose.yml, initialization scripts, a start script, a test script, a logs viewer, and a detailed README. This initial implementation was generous in its automation but hardcoded in its configuration: the data directory /data/fgw-test/ was baked into the docker-compose file, and the scripts referenced absolute paths under /home/theuser/.

The user's response was concise and corrective: "In the script: Require data dir as param, don't refer to /home/theuser." This feedback identified two problems simultaneously. First, the scripts should accept the data directory as a runtime parameter rather than assuming a fixed location. Second, the hardcoded reference to /home/theuser/—the assistant's own home directory—made the configuration non-portable and potentially broken for any other user or deployment environment.

The assistant responded by refactoring in three steps. First, init-data.sh was rewritten to accept a data directory parameter. Second, start.sh was similarly updated. Third—and this is the subject of our analysis—the docker-compose.yml was updated to use environment variable substitution instead of hardcoded paths.

Why This Message Was Written: Completing the Refactoring

The motivation for message 197 is straightforward but significant: consistency. Having already updated the two shell scripts to accept a parameterized data directory, the assistant recognized that the docker-compose file remained as the last holdout of hardcoded configuration. The docker-compose.yml contained lines like:

volumes:
  - /data/fgw-test/yugabyte:/root/var

These paths needed to become:

volumes:
  - ${DATA_DIR}/yugabyte:/root/var

Without this change, the refactoring would be incomplete. A user who passed a custom data directory to start.sh would find that the docker-compose services still mounted volumes from the original hardcoded location. The cluster would either fail to start (if /data/fgw-test/ didn't exist) or would use a different data directory than expected, leading to confusing behavior where uploaded objects seemed to vanish.

The reasoning block in the message reveals this awareness explicitly: "I need to update the docker-compose.yml to use environment variables for the data directory paths." The assistant understood that the refactoring was a three-part operation and that the docker-compose file was the remaining piece. This demonstrates a systematic approach to code modification—not stopping when the immediate task appears done, but following the dependency chain to ensure all related files are updated consistently.

How Decisions Were Made: The Choice of Variable Substitution

The decision to use Docker Compose's built-in environment variable substitution (the ${VAR} syntax) rather than other approaches reveals several implicit design choices. The assistant could have chosen to generate the docker-compose.yml dynamically from a shell script, or to use Docker Compose's --env-file option, or to rely on shell variable expansion in the start.sh script that calls docker-compose up. Each approach has trade-offs.

Environment variable substitution within docker-compose.yml itself is the most idiomatic solution. Docker Compose has supported this feature since version 2.x, and it is widely understood by DevOps practitioners. The syntax is declarative rather than imperative: the docker-compose file declares that a value will be provided at runtime, rather than being generated by a script. This makes the file self-documenting—anyone reading it can see that DATA_DIR is a required configuration parameter.

The assistant's choice also reflects an understanding of how the scripts would work together. The start.sh script, already updated to accept a data directory parameter, would set the DATA_DIR environment variable before invoking docker-compose up. The shell script acts as the orchestrator, translating a command-line argument into an environment variable that Docker Compose can consume. This creates a clean chain of parameterization: user input → shell variable → Docker Compose substitution → container volume mount.

An alternative approach would have been to use Docker Compose's --env-file option to load variables from a configuration file. This would have been more explicit but less convenient for quick command-line use. Another alternative would have been to use sed or similar tools to rewrite the docker-compose file at runtime, but this would have been fragile and error-prone. The assistant's choice of environment variable substitution represents a pragmatic middle ground: flexible enough for the test cluster use case, standard enough to be understood by other developers, and simple enough to implement in a single edit.

Assumptions Made by the User and Agent

Both the user and the assistant operated under several assumptions during this exchange, some of which are worth examining critically.

The assistant's initial assumption was that hardcoding the data directory was acceptable for a test cluster. This assumption is common in infrastructure code: developers often hardcode paths in test configurations because they assume the tests will only be run by themselves or in a specific environment. The assistant likely thought, "This is a test cluster for development purposes; hardcoding the path is fine." The user's correction challenged this assumption, asserting that even test infrastructure should be portable and parameterized.

The user assumed that the assistant would understand the request without needing explicit examples. The feedback "Require data dir as param, don't refer to /home/theuser" is concise but assumes the assistant can infer the full scope of changes needed. The assistant's response—updating three files in sequence—validates this assumption. However, the user might have been more explicit about whether they wanted a command-line argument, an environment variable, a configuration file, or some combination.

Both parties assumed that environment variable substitution in docker-compose.yml was the right mechanism. This is a reasonable shared assumption given the Docker-based infrastructure, but it's worth noting that other container orchestration systems (Kubernetes, Nomad, etc.) handle configuration differently. The assumption here is that the test cluster will continue to use Docker Compose, which is consistent with the development-stage nature of the project.

The assistant assumed that the refactoring should be done incrementally—updating one file at a time—rather than rewriting all files in a single operation. This assumption reflects a cautious, iterative approach to code modification. Each file was read, understood, and updated individually, with the assistant verifying each step before proceeding to the next.

Mistakes and Incorrect Assumptions

The most significant mistake in this exchange was the original hardcoding of both /data/fgw-test/ and /home/theuser/ paths. The /home/theuser/ reference is particularly problematic because it embeds the assistant's own username and home directory into the configuration. Any other developer cloning the repository and running the test cluster would encounter broken paths. This is a classic example of the "works on my machine" problem, where infrastructure code is written to suit the author's environment without consideration for portability.

A subtler issue is that the assistant's refactoring, while correct in mechanism, may not have gone far enough. The docker-compose.yml likely contained other hardcoded values—port numbers, image tags, network names—that were not parameterized. The user's feedback specifically targeted the data directory and the home directory reference, so the assistant correctly limited the scope of changes. But a more thorough refactoring might have considered whether other values should also be configurable. For example, the YugabyteDB image version (yugabytedb/yugabyte:2024.2.5.0-b59) is hardcoded, which could cause issues if the image is updated or if a different version is needed.

Another potential mistake is the lack of validation. The updated scripts accept a data directory parameter, but they do not validate that the directory exists or is writable before proceeding. A user who passes a non-existent directory would encounter confusing Docker errors rather than a clear message. Similarly, there is no default value for DATA_DIR in the docker-compose.yml, which means that running docker-compose up directly (without going through start.sh) would fail with an unset variable error. This is arguably intentional—forcing users to go through the provided scripts—but it could be surprising.

Input Knowledge Required to Understand This Message

To fully understand message 197, a reader needs knowledge in several areas. First, familiarity with Docker Compose and its variable substitution syntax is essential. The ${DATA_DIR} syntax is specific to Docker Compose and differs from shell variable expansion or other templating systems. A reader who has never used Docker Compose might not understand what the message is changing or why.

Second, understanding the broader architecture of the test cluster is necessary. The reader needs to know that there are three layers (frontend proxies, Kuri storage nodes, YugabyteDB), that each service mounts volumes for persistent data, and that the data directory must be consistent across all services. Without this context, the change to volume paths seems trivial rather than structurally important.

Third, knowledge of the previous two refactoring steps (updating init-data.sh and start.sh) is required to see the full picture. Message 197 is the third in a sequence, and its significance depends on understanding that the shell scripts were already parameterized. A reader seeing only this message might wonder why only the docker-compose file is being changed and not the scripts.

Fourth, the reader needs to understand the user's feedback from message 193: "In the script: Require data dir as param, don't refer to /home/theuser." This feedback is the catalyst for the entire refactoring, and without it, the motivation for message 197 is opaque.

Output Knowledge Created by This Message

Message 197 produces a single concrete output: an updated docker-compose.yml file at /home/theuser/gw/test-cluster/docker-compose.yml. The file now uses ${DATA_DIR} instead of hardcoded paths for all volume mounts. This change has several implications:

1. Portability: The test cluster configuration can now be used by any developer, on any machine, by simply providing a different DATA_DIR value. The configuration is no longer tied to a specific filesystem layout.
2. Reproducibility: Multiple test clusters can run simultaneously on the same machine by using different data directories. This is important for testing scenarios that require isolated environments.
3. Documentation: The docker-compose file now explicitly declares its configuration dependency. Anyone reading the file can see that DATA_DIR must be set before running the cluster.
4. Consistency: The docker-compose file now matches the parameterization pattern used by the shell scripts. The entire test cluster infrastructure follows a uniform configuration strategy. Beyond the file itself, the message creates knowledge about the refactoring process. It demonstrates that the assistant recognized the need for consistency across multiple files and followed through on the complete change rather than stopping after the first two updates. This systematic approach is a model for how infrastructure refactoring should be done.

The Thinking Process Visible in the Reasoning

The reasoning block in message 197 is brief but revealing: "I need to update the docker-compose.yml to use environment variables for the data directory paths. Let me replace the hardcoded paths with variable substitution."

This reasoning shows several cognitive steps compressed into a single sentence. First, the assistant has identified the remaining file that needs modification. This implies a mental checklist: "I've updated init-data.sh, I've updated start.sh, now I need to update docker-compose.yml." The assistant is tracking the state of the refactoring across multiple files.

Second, the assistant has chosen the specific technique: environment variable substitution rather than any other approach. This choice is not explained in the reasoning—it's presented as settled—but it reflects an understanding of Docker Compose's capabilities and the idiomatic way to parameterize Docker Compose files.

Third, the assistant has scoped the change to "the data directory paths" specifically. The reasoning does not say "parameterize everything" or "make the entire file configurable." It focuses on the specific paths that need to change. This demonstrates precise understanding of the user's feedback: the user asked for the data directory to be a parameter, not for a complete configuration overhaul.

The absence of any hesitation or alternative consideration in the reasoning is also notable. The assistant does not write "I could use an env file, or I could generate the file dynamically, but I'll use variable substitution." The reasoning presents the solution as obvious and straightforward. This could reflect either genuine confidence in the approach or a simplification in the recorded reasoning (the assistant may have considered alternatives but did not record them).

Conclusion

Message 197 is a small but instructive moment in a larger engineering conversation. It demonstrates how a single piece of user feedback—"Require data dir as param"—can trigger a systematic refactoring across multiple files. The assistant's response shows the importance of following through on changes completely, ensuring that all related files are updated consistently rather than leaving a partially parameterized system.

The message also illustrates several software engineering principles in action: the value of portability over convenience, the importance of listening to user feedback about configuration design, and the power of idiomatic tool usage (Docker Compose variable substitution) over ad-hoc solutions.

Most importantly, this message reminds us that infrastructure code is never truly finished. Each round of feedback reveals new assumptions to examine, new paths to parameterize, and new opportunities to make the system more robust and more usable. The hardcoded path that seemed harmless at first became, under scrutiny, a barrier to portability and collaboration. The parameterized path that replaced it is not just a technical improvement—it is an expression of respect for the other developers who will use this code, whoever and wherever they may be.