When Read-Only Volumes Block Deployment: Debugging Ansible Test Infrastructure

In the middle of an intensive session building and validating Ansible deployment scripts for a horizontally scalable S3 storage architecture, a single message captures a moment of friction that every infrastructure engineer knows well: the silent assumption that a volume is writable, only to be met with a cascade of read-only errors. Message 1560 in this coding conversation is a snapshot of iterative debugging — a moment where the assistant discovers that binaries intended for deployment targets were never actually installed, and then attempts to fix the situation by manually copying them, only to run headlong into a Docker volume mounted as read-only. This message is not a triumphant breakthrough; it is the gritty, necessary work of uncovering hidden assumptions in a test harness.

The Message

Here is the message in full, exactly as it appeared in the conversation:

[assistant] The binaries weren't copied. Let me copy them again: [bash] cd /home/theuser/gw/ansible/test/docker && for host in kuri-01 kuri-02 s3-fe-01; do container="ansible-test-$host" docker exec "$container" mkdir -p /opt/fgw/bin docker cp "./binaries/kuri" "$container:/opt/fgw/bin/" docker cp "./binaries/s3-proxy" "$container:/opt/fgw/bin/" docker exec "$container" chmod +x /opt/fgw/bin/ docker exec "$container" chown -R fgw:fgw /opt/fgw/bin/ echo " $host binaries installed" done 2>&1 time="2026-01-31T18:38:21+01:00" level=error msg="Can't add file /home/theuser/gw/ansible/test/docker/binaries/kuri to tar: io: read/write on closed pipe" time="2026-01-31T18:38:21+01:00" level=error msg="Can't close tar writer: io: read/write on closed pipe" Error response from daemon: mounted volume is marked read-only Error response from daemon: mounted volume is marked read-only zsh:6: no matches found: /opt/fgw/bin/ chown: changing ownership of '/opt/fgw/bin/': Read-only file system kuri...

Context: The Ansible Test Harness

To understand why this message exists, one must understand what came before it. The assistant had been building a comprehensive Ansible-based deployment system for a distributed S3 storage cluster called FGW (Filecoin Gateway). The architecture involves three layers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB database. The assistant had created seven Ansible roles, five playbooks, and an inventory structure for production deployment.

But before trusting these playbooks on real hardware, the assistant needed a test harness — a Docker-based environment that would simulate target hosts. The test infrastructure consisted of:

• A YugabyteDB container (the database)
• Three target containers running Ubuntu 24.04 with systemd and SSH (two Kuri nodes and one S3 frontend)
• An Ansible controller container that would run the playbooks against the targets The assistant had been iterating through this test harness, fixing issues one by one: the YugabyteDB health check needed the correct hostname, the controller needed psql, cqlsh, and sshpass installed, the test inventory was missing group_vars for the Kuri and S3 frontend groups, target volumes had read-only mount issues, and pam_nologin was blocking SSH logins after container startup. By message 1559, the assistant had just run the test suite and discovered that the binaries weren't being installed in /opt/fgw/bin/ on the target hosts. The directory existed but was empty. This was the immediate trigger for message 1560.

Why This Message Was Written: Reasoning and Motivation

The assistant's motivation is straightforward: the Ansible playbook test run failed because the target hosts had no binaries. The playbooks presumably try to copy or execute kuri and s3-proxy from /opt/fgw/bin/, and when those files don't exist, the deployment fails. The assistant's reasoning chain appears to be:

1. "The binaries aren't being installed" (discovered in message 1559 by inspecting the container)
2. "Let me copy them again" — a direct, manual intervention to bypass whatever mechanism was supposed to install them
3. The assistant writes a shell loop that iterates over all three target hosts, creates the target directory, copies the two binaries, sets permissions, and sets ownership The thinking here reveals an important assumption: the assistant is treating the symptom (empty binary directory) rather than the root cause (the Ansible role or Docker volume configuration that should have populated it). This is a pragmatic debugging strategy — by manually verifying that the binaries can be placed in the target directory, the assistant can narrow down whether the problem is in the copy mechanism or in something else entirely.

The Cascade of Errors and What They Reveal

The output of the command is a masterclass in cascading failure modes. Each error tells a different story:

"Can't add file ... to tar: io: read/write on closed pipe" — The docker cp command works by creating a tar archive of the file and streaming it to the container. This error suggests that the write end of the pipe was closed before the tar could be completed. This is almost certainly a consequence of the read-only volume: the Docker daemon attempted to write the tar stream to the container's filesystem, found it was read-only, and closed the pipe, leaving the tar writer in a broken state.

"Error response from daemon: mounted volume is marked read-only" — This is the root cause, stated plainly. The /opt/fgw/bin/ directory in the container is a Docker volume mounted with the :ro flag. The docker cp command cannot write to a read-only mount.

"zsh:6: no matches found: /opt/fgw/bin/*" — Because the docker cp commands failed, no files were placed in the directory. When the shell tries to expand /opt/fgw/bin/* for the chmod command, the glob matches nothing, and in zsh (the shell inside the container), this is a fatal error by default.

"chown: changing ownership of '/opt/fgw/bin/': Read-only file system" — Even the chown command fails because the entire mount point is read-only, not just the individual files.

The final line, kuri..., is truncated — it's the echo output for the first host in the loop, but the loop likely aborted after the errors on subsequent hosts.

Assumptions Made

This message reveals several assumptions, some of which turned out to be incorrect:

1. The binaries directory is writable. The assistant assumed that /opt/fgw/bin/ on the target containers was a regular directory that could be written to via docker cp. In reality, it was a Docker volume bind-mounted from the host as read-only. This assumption was baked into the Docker Compose configuration from earlier in the session.
2. Manual copy will work where Ansible failed. The assistant assumed that if the Ansible playbook couldn't install binaries, a direct docker cp would succeed. This bypasses the actual deployment mechanism and tests a different path entirely.
3. The glob will match. Using chmod +x /opt/fgw/bin/* without checking whether files exist first is fragile. In bash this would be harmless (the glob would pass through literally), but in zsh it's a fatal error.
4. The loop will complete. The assistant wrote a for loop over three hosts, but the errors on the first host likely caused the loop to continue in a broken state for the remaining hosts.

Mistakes and Incorrect Assumptions

The most significant mistake is not the read-only volume itself — that was a deliberate configuration choice earlier in the session — but rather the assistant's failure to recognize it before running the manual copy command. The read-only mount had been configured in the Docker Compose file as a deliberate design decision (perhaps to simulate production-like restrictions or to prevent accidental modifications). The assistant had even encountered read-only issues earlier in the session (message 1531 mentions "The ansible directory is mounted read-only").

The second mistake is the approach itself. Rather than fixing the underlying issue — either by changing the volume mount to read-write or by modifying the Ansible role to handle binary deployment correctly — the assistant attempted a manual workaround that was doomed to fail against the same read-only constraint.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

• Docker fundamentals: how docker cp works (it streams a tar archive), how volume mounts work, what the :ro flag means
• Container filesystem concepts: the difference between a bind mount and a container's native filesystem, and how read-only mounts affect write operations
• Shell scripting: how for loops work in bash/zsh, how glob expansion differs between shells, and how error handling (or lack thereof) affects command sequences
• The FGW architecture: understanding that kuri and s3-proxy are the two main binaries, that they go into /opt/fgw/bin/, and that they need to be present on all three target hosts
• The debugging context: the previous messages showing the test harness setup, the Ansible playbook runs, and the discovery that the binaries directory was empty

Output Knowledge Created

This message produces several pieces of actionable knowledge:

1. The read-only volume is confirmed as the blocker. The error messages definitively prove that /opt/fgw/bin/ is mounted read-only, which is why neither the Ansible playbook nor the manual copy can populate it.
2. The Docker Compose configuration needs fixing. The binaries volume in docker-compose.yml needs to be changed from read-only to read-write, or the binary deployment strategy needs to be rethought entirely.
3. The manual copy approach is not viable. Even if the volume were made writable, manually copying binaries with docker cp is not a sustainable deployment strategy — it bypasses the Ansible automation that the entire test harness is meant to validate.
4. Shell compatibility matters. The zsh: no matches found error reveals that the target containers use zsh, which has different globbing behavior than bash. This is a subtle but important detail for any future shell commands run inside these containers.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in the structure of the command itself. The sequence of operations reveals a mental model of what "installing binaries" entails:

1. Create the target directory (mkdir -p)
2. Copy the files (docker cp)
3. Make them executable (chmod +x)
4. Set the correct ownership (chown -R fgw:fgw) This is a classic four-step installation procedure, and the assistant is replicating what the Ansible role should be doing. The fact that step 2 fails immediately tells us that the problem is at the filesystem level, not at the permission or ownership level. The assistant also chose to iterate over all three hosts in a single loop, which is efficient but also means that a failure on one host could cascade. The output shows errors for kuri-01 (the first host), and it's unclear whether the loop continued for the other hosts.

Broader Significance

This message is a microcosm of the entire debugging session. The assistant is building a complex, multi-layered test infrastructure, and each layer introduces assumptions that can silently conflict with other layers. The Docker Compose file assumed read-only mounts would be fine; the Ansible playbook assumed writable binary directories; the manual debugging assumed neither of those constraints existed. The collision of these assumptions is what produces the error cascade.

In the messages immediately following (1561-1563), the assistant fixes the Docker Compose file by removing the :ro flag, restarts the containers, and successfully copies the binaries. The read-only volume was a configuration choice, not a fundamental limitation — and once recognized, it was trivially fixable. But recognizing it required running headlong into the error, reading the message, and understanding what it meant.

This is the essence of infrastructure debugging: not brilliance, but persistence. The assistant tried something, it failed, the error message told a story, and the assistant listened.