The Binary Deployment That Almost Worked: A Study in Shell Globbing and Infrastructure Automation

Introduction

In the middle of an intense debugging session for a horizontally scalable S3 storage architecture, a single bash command was executed that perfectly encapsulates the tension between automation and the subtle edge cases that plague infrastructure tooling. Message 1563 in this coding session shows an engineer attempting to deploy compiled binaries into freshly restarted Docker containers, only to be met with a cryptic zsh globbing error that silently undermines the operation. This message, seemingly mundane at first glance, reveals deep truths about the assumptions we make when scripting infrastructure, the dangers of shell expansion across context boundaries, and the iterative nature of building reliable deployment pipelines.

The Message

The message reads as follows:

cd /home/theuser/gw/ansible/test/docker && sleep 15 && for host in kuri-01 kuri-02 s3-fe-01; do
    container="ansible-test-$host"
    docker cp "./binaries/kuri" "$container:/opt/fgw/bin/"
    docker cp "./binaries/s3-proxy" "$container:/opt/fgw/bin/"
    docker exec "$container" chmod +x /opt/fgw/bin/*
    docker exec "$container" chown -R fgw:fgw /opt/fgw/bin/
    docker exec "$container" ls -la /opt/fgw/bin/
    echo "  $host binaries installed"
done 2>&1

And the output:

zsh:5: no matches found: /opt/fgw/bin/*
total 208920
drwxr-xr-x 1 fgw fgw        24 Jan 31 17:39 .
drwxr-xr-x 1 fgw fgw        18 Jan 31 17:30 ..
-rwxr-xr-x 1 fgw fgw 174769544 Jan 31 17:34 kuri
-rwxr-xr-x 1 fgw fgw  39158272 Jan 31 17:34 s3-proxy
  kuri-01 binaries installed
zsh:5: no matches found: /opt/fgw/bin/*
total 208920
drwxr-xr-x 1 fgw fgw        24 Jan 31 17:39 .
drwxr-xr-x 1 fgw fgw        18 Jan 31 17:30 ..
-rwxr-xr-x 1 fgw fgw 174769544 Jan 31 17:34 kuri
-rwxr-xr-x 1 fgw fgw  391582...

Why This Message Was Written: The Context

To understand why this message exists, we must trace the debugging session that preceded it. The engineer was building an Ansible-based deployment system for a distributed S3 storage cluster called FGW (Filecoin Gateway). The architecture consists of three layers: stateless S3 frontend proxies, Kuri storage nodes, and a shared YugabyteDB backend. The deployment automation includes seven Ansible roles and five playbooks, and the engineer had created a Docker-based test harness to validate these scripts before running them against real infrastructure.

The test harness uses Docker Compose to spin up multiple containers: a YugabyteDB instance, three target hosts running Ubuntu 24.04 with systemd and SSH (kuri-01, kuri-02, s3-fe-01), and an Ansible controller container. The target hosts need the FGW binaries (kuri and s3-proxy) installed in /opt/fgw/bin/ before the Ansible playbooks can configure them.

In the immediately preceding messages, the engineer had discovered a critical problem: the Docker Compose file was mounting the binaries volume as read-only (binaries:/opt/fgw/bin:ro). When they tried to copy binaries into the containers, they received "read-only file system" errors. They fixed this by editing the compose file to make the volume writable, then ran docker compose down && docker compose up -d to restart the containers with the new configuration.

Message 1563 is the first attempt to deploy binaries into the freshly restarted containers. It represents a moment of cautious optimism — the read-only volume problem has been fixed, the containers are restarting, and now it's time to verify that the fix actually works.

The Hidden Bug: Shell Globbing Across Context Boundaries

The most interesting aspect of this message is the subtle bug revealed by the error zsh:5: no matches found: /opt/fgw/bin/*. This error occurs because the engineer's shell (zsh) is expanding the glob pattern /opt/fgw/bin/* before passing it to the docker exec command. Since the host machine doesn't have a /opt/fgw/bin/ directory with matching files, zsh's default behavior (configured with nomatch or similar) causes it to abort with a "no matches found" error.

This is a classic and pernicious class of bug in shell scripting. The engineer's intent was clear: run chmod +x on all files inside /opt/fgw/bin/ inside the container. But because the glob pattern was not quoted, the shell on the host machine interpreted it first. The fix would be to escape the asterisk or quote it so that it gets passed literally to the container's shell: docker exec "$container" chmod +x '/opt/fgw/bin/*' or docker exec "$container" sh -c 'chmod +x /opt/fgw/bin/*'.

The output reveals that despite the error, the docker cp commands preceding it succeeded — the binaries were copied into the container. The ls -la command that follows also works because it doesn't use a glob pattern that would trigger expansion issues. So the binaries are present and listed, but they may not have the correct permissions or ownership because the chmod and chown commands failed silently (well, not silently — the error was printed, but the script continued).

This is a critical observation: the engineer's confirmation message kuri-01 binaries installed is displayed even though the permission-setting steps failed. The script has no error checking or conditional logic. This is typical of rapid prototyping and debugging sessions where the goal is to move fast, but it also represents a reliability gap that could cause subtle failures later.

Assumptions Made

Several assumptions underpin this message, some of which turned out to be incorrect:

Assumption 1: The containers would be fully booted after 15 seconds. The sleep 15 at the beginning of the command assumes that the containers, which were just restarted with docker compose down && docker compose up -d, will be ready to accept docker cp and docker exec commands within that timeframe. In practice, this is usually sufficient for Docker containers, but the target containers run systemd which can take longer to initialize. Earlier in the session (message 1544), the engineer had to wait for systemd to reach "running" state, which took about 10 seconds after container start.

Assumption 2: The glob pattern would be interpreted by the container's shell. This was the critical mistake. The engineer assumed that * in docker exec "$container" chmod +x /opt/fgw/bin/* would be expanded inside the container. Instead, zsh on the host expanded it first.

Assumption 3: The binaries would be executable and owned by fgw after the copy. The docker cp command preserves file permissions from the source, but the source binaries might have specific permissions that need adjustment. The engineer correctly anticipated this by running chmod +x and chown, but the globbing bug prevented these from executing properly.

Assumption 4: The script would halt on error. The 2>&1 redirect suggests the engineer expected to see errors, but there's no set -e or explicit error handling. The script continues despite the globbing error, which means the engineer may not have noticed the problem immediately.

Input Knowledge Required

To fully understand this message, one needs:

  1. Docker fundamentals: Understanding of docker cp (copying files into containers) and docker exec (running commands inside containers), as well as the concept of container restart cycles.
  2. Shell scripting: Knowledge of glob expansion, the difference between quoted and unquoted wildcards, and how zsh handles glob matching failures differently from bash.
  3. The FGW architecture: Understanding that the system has three layers (S3 proxy, Kuri storage, YugabyteDB) and that each node type needs specific binaries. The kuri binary (174MB) is the storage node executable, while s3-proxy (39MB) is the frontend proxy.
  4. The debugging history: The read-only volume issue from the previous messages, the Docker Compose configuration, and the overall goal of validating Ansible playbooks.
  5. Linux file permissions: Understanding why chmod +x and chown -R fgw:fgw are necessary after copying binaries into a container where the target directory exists but may have different ownership.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Verification that the read-only fix worked: The docker cp commands succeed, confirming that changing the volume mount from :ro to writable in the Docker Compose file resolved the previous error.
  2. Binary sizes and presence: The output confirms that kuri is 174MB and s3-proxy is 39MB, and both are present in /opt/fgw/bin/ on each target host. This validates that the build process produced the correct binaries and that they were successfully transferred.
  3. A discovered bug: The zsh globbing error reveals a flaw in the deployment script. This is valuable knowledge — even though the engineer may not have addressed it immediately, the error is visible in the output and could be caught in a review.
  4. Container naming convention: The mapping between host names (kuri-01, kuri-02, s3-fe-01) and container names (ansible-test-kuri-01, etc.) is demonstrated and validated.
  5. A working pattern for binary deployment: Despite the globbing issue, the overall approach of using docker cp followed by permission adjustment is shown to work (modulo the shell expansion bug).

The Thinking Process Visible in the Message

The structure of this command reveals the engineer's thought process. The sleep 15 at the beginning shows an awareness that containers need time to initialize after restart — a lesson learned from earlier in the session when they had to wait for systemd. The loop over three hosts shows a systematic approach: apply the same operation to all nodes rather than handling them individually. The use of docker cp rather than mounting a volume shows a preference for explicit file transfer over shared filesystem approaches, which gives more control over permissions and ownership.

The ordering of operations within the loop is logical: copy binaries first, then set permissions, then verify with ls, then print a confirmation. This is a classic "do, verify, report" pattern that reflects disciplined debugging practices.

The 2>&1 at the end of the command suggests the engineer wanted to capture both stdout and stderr together, perhaps to see the full output in one stream. This is common when running commands in CI/CD or test scripts where you want a complete log.

The fact that the engineer didn't immediately fix the globbing issue after seeing the error is interesting. It could mean they were focused on the higher-level goal (getting binaries into containers) and considered the chmod/chown steps secondary, or it could mean they simply didn't notice the error among the output. The session continues with further debugging, suggesting that the binaries were usable despite the permission issue, or that the error was addressed later.

Broader Significance

This message is a microcosm of the challenges in infrastructure automation. It demonstrates how a single unquoted asterisk can undermine an otherwise well-structured deployment script. It shows the importance of understanding shell expansion rules when crossing the boundary between host and container. And it illustrates the iterative nature of building reliable systems: fix one bug (read-only volume), discover another (shell globbing), and continue the cycle.

For anyone building deployment pipelines, this message serves as a cautionary tale. Always quote glob patterns that are meant for remote execution. Use set -e or explicit error handling in deployment scripts. And never trust a "success" message without verifying that every step in the chain actually executed correctly.

The binaries ended up in the containers — the ls output proves that. But whether they had the correct permissions and ownership remains an open question, a loose thread in the fabric of the deployment that could unravel under different circumstances. This is the reality of debugging complex systems: progress is measured not in perfect solutions, but in the gradual accumulation of working pieces, each held together by assumptions that may or may not hold tomorrow.