The Perils of Shell Globbing Across Permission Boundaries

In the middle of deploying a distributed storage cluster across three physical nodes, a seemingly straightforward file-copy operation fails with a cryptic error. The assistant's command—designed to transfer cryptographic wallet files to remote servers and secure them—returns chown: cannot access '/home/fgw/.ribswallet/*': No such file or directory. This single failure message, buried in a long session of infrastructure automation, reveals a subtle but instructive bug that touches on Unix permission models, shell glob semantics, and the hidden complexities of privilege escalation.

The Message

The subject message (index 1984) contains two nearly identical bash commands, each targeting one of two remote storage nodes:

[assistant] [bash] scp ~/.ribswallet/MRSWMYLVNR2A ~/.ribswallet/O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS 10.1.232.83:/tmp/ && ssh 10.1.232.83 "sudo mv /tmp/MRSWMYLVNR2A /tmp/O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS /home/fgw/.ribswallet/ && sudo chown fgw:fgw /home/fgw/.ribswallet/* && sudo chmod 600 /home/fgw/.ribswallet/*"
chown: cannot access '/home/fgw/.ribswallet/*': No such file or directory

[bash] scp ~/.ribswallet/MRSWMYLVNR2A ~/.ribswallet/O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS 10.1.232.84:/tmp/ && ssh 10.1.232.84 "sudo mv /tmp/MRSWMYLVNR2A /tmp/O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS /home/fgw/.ribswallet/ && sudo chown fgw:fgw /home/fgw/.ribswallet/* && sudo chmod 600 /home/fgw/.ribswallet/*"
chown: cannot access '/home/fgw/.ribswallet/*': No such file or directory

Both commands fail with the same error. The wallet file names have been left unredacted in this article because they appear to be opaque cryptographic identifiers rather than secrets—the actual sensitive content is the file contents, not their names. The CIDgravity token and wallet key material are not exposed in this message.

Context: Deploying a Distributed Storage Cluster

To understand why this message was written, we need to step back. The assistant was in the middle of deploying a QA test cluster for the Filecoin Gateway (FGW) distributed storage system across three physical machines: a head node (10.1.232.82) running YugabyteDB, and two storage nodes (kuri1 at 10.1.232.83 and kuri2 at 10.1.232.84). The architecture required each storage node to run a kuri daemon that manages IPFS data, CAR file staging, and S3-compatible object storage, all backed by a shared YugabyteDB cluster.

Earlier in the session, the user had provided wallet files located at ~/.ribswallet/ on the development machine, along with a CIDgravity API token. The wallet contained two key files with opaque names—cryptographic material used for Filecoin network identity. The assistant had already created the fgw system user on both remote nodes, set up data directories, and deployed the compiled Go binaries. The next logical step was to transfer the wallet files to each node so the kuri daemon could authenticate with the Filecoin network.

What Went Wrong: The Shell Glob Permission Trap

The command appears straightforward: use scp to copy two wallet files to /tmp/ on each remote node, then via SSH, use sudo to move them into the fgw user's home directory (/home/fgw/.ribswallet/), change ownership to fgw:fgw, and set restrictive permissions (chmod 600). The use of a wildcard (*) in the chown and chmod commands was intended to apply the operation to all files in the directory without needing to name them explicitly.

The error message—"No such file or directory" for the literal path /home/fgw/.ribswallet/*—is the shell's way of saying that the glob pattern matched nothing. But why? The sudo mv command should have placed the files there. The answer lies in a subtle interaction between Unix file permissions and shell glob expansion.

The directory /home/fgw/.ribswallet/ had been created in a previous step (message 1983) with chmod 700 and ownership fgw:fgw. This means only the fgw user can read the directory's contents. The SSH connection, however, runs as the theuser user (the user initiating the SSH session). When the remote shell processes the command string, it performs glob expansion before executing any commands, including sudo. The shell, running as theuser, attempts to expand /home/fgw/.ribswallet/* by listing the directory—and fails because permission is denied. The shell cannot see the files, so the glob matches nothing, and the literal string * is passed as an argument to chown and chmod. These utilities then report that a file literally named * does not exist in the directory.

This is a classic Unix pitfall: shell glob expansion happens at the shell level, under the identity of the shell process, not under the identity that sudo will later assume. The sudo command only elevates privileges for the command itself, not for the shell's preparatory work like pathname expansion.## Why the Wildcard Was Used

The assistant's choice to use a wildcard in the chown and chmod commands was driven by a desire for generality and future-proofing. Rather than hardcoding the two specific wallet filenames, a glob pattern would automatically apply the correct ownership and permissions to any files present in the directory. This is a reasonable instinct in automation: avoid repeating filenames that might change, and make the command resilient to variations. In the context of an Ansible-driven deployment pipeline, this pattern of "apply to everything in the directory" is common.

However, the assistant overlooked a critical detail: the sudo command does not cause the shell to re-evaluate globs under the elevated user's identity. The shell expands wildcards before sudo ever runs, using the permissions of the shell process itself. This is a fundamental aspect of Unix shell architecture that catches many automation authors off guard.

The Immediate Aftermath

The error message was not the end of the story. Immediately after the failed command, the assistant ran diagnostic commands to understand what happened:

[assistant] [bash] ssh 10.1.232.83 "ls -la /home/fgw/.ribswallet/"
ls: cannot access '/home/fgw/.ribswallet/': Permission denied

This confirmed the root cause: the directory was owned by fgw with mode 700, making it invisible to the theuser user running the SSH session. Even though sudo mv had successfully placed the files inside (as sudo runs with elevated privileges), the subsequent sudo chown and sudo chmod commands failed because the shell had already expanded the glob—or rather, failed to expand it—before sudo could execute.

The assistant then used sudo ls to verify the files were indeed present:

[assistant] [bash] ssh 10.1.232.83 "sudo ls -la /home/fgw/.ribswallet/"
total 16
drwx------ 2 fgw     fgw     4096 Jan 31 22:16 .
drwxr-x--- 3 fgw     fgw     4096 Jan 31 22:16 ..
-rw------- 1 theuser theuser   80 Jan 31 22:16 MRSWMYLVNR2A
-rw------- 1 theuser theuser   80 Jan 31 22:16 O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS

Note that the files still showed ownership as theuser:theuser—the sudo mv had preserved the original ownership from the copy operation, and the chown command had never executed correctly. The assistant then issued a corrected command (message 1987) that explicitly named each file instead of using a wildcard:

ssh 10.1.232.83 "sudo chown fgw:fgw /home/fgw/.ribswallet/MRSWMYLVNR2A /home/fgw/.ribswallet/O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS && sudo chmod 600 /home/fgw/.ribswallet/MRSWMYLVNR2A /home/fgw/.ribswallet/O5QWY3DFOQWWMMJVNZUWWZLVOVVWO2TQNMZXEZRTNZSG26DMNJTXE33BM5VDKMTWMRTXU3TLMN3XS"

This worked because the filenames were provided explicitly, bypassing the shell glob entirely.

A Deeper Security Lesson

This episode occurred in the shadow of an even more important security correction. Just before the wallet deployment, the assistant had made a more serious mistake: embedding the CIDgravity API token directly into environment configuration files (settings.env) that would be stored in version control and readable by anyone with access to the filesystem. The user caught this immediately with the sharp rebuke: "Wait, DO NOT STORE SECRETS OUTSIDE OF VAULTS."

The assistant deleted the insecure config files and redesigned the approach. The token was stored in a separate file (/home/fgw/.ribswallet/cidg.token) with restrictive permissions (chmod 600, owned by fgw). The systemd service was configured to load it at runtime using an ExecStartPre script that reads the token and writes it to a runtime environment file in /run/fgw/token.env. This meant the token never appeared in version-controlled files or world-readable locations—a proper implementation of the principle of least privilege.

This security-conscious redesign made the wallet deployment step more critical. The wallet files, like the CIDgravity token, were sensitive cryptographic material. They needed to be placed in the fgw user's home directory with the same restrictive permissions: owned by fgw, mode 600. The assistant's intent was correct; the execution was flawed only in the mechanical detail of the shell glob.

Assumptions and Their Consequences

The message reveals several assumptions made by the assistant:

1. That sudo would elevate the glob expansion. This is the central mistake. The assistant assumed that wrapping a command in sudo would cause all parts of the command—including shell metacharacter expansion—to run with elevated privileges. In reality, sudo only elevates the command itself, not the shell's preparatory parsing.
2. That the wildcard would match the newly moved files. The assistant assumed the mv command would complete before the chown and chmod commands ran, which it did (the && operator ensures sequential execution). But the glob expansion happens before any of these commands execute, at parse time.
3. That the fgw home directory was accessible. The assistant had created /home/fgw/.ribswallet/ with chmod 700 and ownership fgw:fgw in a previous step, which was correct for security but created the glob expansion problem.
4. That the wallet filenames were stable. The assistant used wildcards partly because the filenames were long, opaque strings that were inconvenient to type. The corrected command had to spell them out explicitly.

Input Knowledge Required

To understand this message, a reader needs knowledge of:

• Unix file permissions: The distinction between chmod 700 (directory accessible only by owner) and chmod 600 (file readable/writable only by owner).
• Shell glob expansion: How * is expanded by the shell before command execution, and how this interacts with sudo.
• SSH command execution: How a command string passed to ssh is parsed by the remote shell.
• The deployment context: That wallet files are cryptographic identity material for the Filecoin network, requiring careful permission management.
• The architecture: Two remote storage nodes (kuri1, kuri2) receiving the same configuration, with a shared YugabyteDB backend.

Output Knowledge Created

This message produced several insights:

1. A documented failure mode: The error "cannot access '/home/fgw/.ribswallet/*': No such file or directory" is now clearly understood as a shell glob permission problem, not a missing file problem.
2. A corrected procedure: The follow-up commands (messages 1985-1987) demonstrate the proper approach: use explicit filenames with sudo, or alternatively, use sudo -u fgw bash -c 'chmod 600 /home/fgw/.ribswallet/*' to run the entire glob expansion under the target user's identity.
3. A security pattern: The overall approach of separating secrets into restricted files, loading them at runtime, and never storing them in version-controlled configuration files is a reusable pattern for any distributed deployment.

The Broader Pattern: Automation and Edge Cases

This bug is a microcosm of a broader class of automation failures. When writing infrastructure scripts, developers routinely make assumptions about the execution environment that turn out to be wrong. The shell glob permission trap is particularly insidious because it produces an error message that seems to say "file not found" when the real problem is "directory not readable." The error message is technically accurate—no file named * exists—but deeply misleading as to the root cause.

The assistant's debugging process demonstrates good troubleshooting methodology: verify the directory exists, check permissions, use sudo to inspect, and then correct the command by eliminating the problematic pattern. The fix—explicitly naming each file—is not the only possible solution. An alternative would be to wrap the entire command in a sudo -u fgw bash -c '...' invocation, which would run the shell (and its glob expansion) under the fgw user's identity:

ssh 10.1.232.83 "sudo -u fgw bash -c 'chown fgw:fgw /home/fgw/.ribswallet/* && chmod 600 /home/fgw/.ribswallet/*'"

This approach would work because the shell process itself runs as fgw, giving it permission to list the directory and expand the glob. The assistant did not choose this path, instead opting for explicit filenames—a simpler and more transparent solution for a one-time deployment step.

Conclusion

The failed wallet deployment in message 1984 is a textbook example of a subtle Unix permission bug that can derail infrastructure automation. The assistant's command was logically correct but mechanically flawed: the shell expanded a wildcard before privilege escalation, and the wildcard matched nothing because the directory was invisible to the unprivileged shell process. The error message, while technically accurate, pointed to the wrong cause, requiring diagnostic commands to uncover the real issue.

This episode also highlights the importance of security-conscious infrastructure design. The same deployment step that tripped over a shell glob was part of a larger pattern of proper secret management—storing tokens in restricted files, loading them at runtime, and never committing them to version control. The wildcard bug was a mechanical failure in service of a sound security goal.

For anyone writing automation scripts that cross user boundaries—whether via sudo, containerization, or remote execution—this pattern is worth remembering: shell globs expand under the identity of the shell, not the identity of the command. When in doubt, be explicit, or run the entire shell under the target user's identity. A few extra characters of typing can save minutes of debugging a misleading error message.