The Permission Denied That Stopped a Deployment: A Case Study in SSH, Unix Permissions, and Production Deployments

Introduction

In the course of a complex engineering session spanning deployment infrastructure, SSH connectivity fixes, and production management tooling, a single message stands out as a microcosm of the challenges inherent in remote deployment. Message 3788 captures a moment where an AI assistant attempts to deploy an updated binary to a production server, only to be stopped by a seemingly simple permission error. This article examines that message in depth—the reasoning behind it, the assumptions that led to failure, and the knowledge it reveals about production deployment practices.

The Message

The subject message is a single bash command executed via SSH:

ssh theuser@10.1.2.104 'cp /usr/local/bin/vast-manager /usr/local/bin/vast-manager.bak && cp /tmp/vast-manager.new /usr/local/bin/vast-manager && chmod +x /usr/local/bin/vast-manager'

The result was immediate and unambiguous:

cp: cannot create regular file '/usr/local/bin/vast-manager.bak': Permission denied

This three-line exchange—one command, one error—encapsulates a pivotal moment in a larger deployment workflow. To understand why this message was written and what it reveals, we must examine the context that led to it.

The Context: A Chain of SSH Troubleshooting

The assistant had been working on a critical infrastructure problem. The vast-manager—a Go binary that manages Vast.ai cloud instances for running cuzk/curio proving workloads—was experiencing SSH connectivity failures across all managed nodes. Every connection attempt returned the cryptic error "exit status 255," SSH's generic failure code, with no diagnostic information about the actual cause.

The root cause was twofold. First, the Go code used cmd.Output() to execute SSH commands, which captures only standard output while discarding standard error—the very channel where SSH reports authentication failures, connection refused errors, and other diagnostics. Second, stale ControlMaster sockets from SSH's connection-sharing mechanism could cause subsequent connections to fail if the master process had died unexpectedly.

The assistant's fix addressed both issues: it modified the SSH execution to capture stderr via a bytes.Buffer, added retry logic that cleans up stale control sockets on failure, and ensured error messages would include the actual SSH diagnostic text. After implementing these changes in the Go source code, the assistant built the new binary and was instructed to deploy it to the manager host at IP address 10.1.2.104.

The Deployment Attempt: A Step-by-Step Breakdown

The deployment sequence reveals a methodical, if ultimately flawed, approach. The assistant first attempted to copy the binary directly to the target directory as the root user using scp:

scp /tmp/czk/vast-manager root@10.1.2.104:/usr/local/bin/vast-manager.new

This failed with a "Received message too long" error—a classic symptom of shell startup files (like .bashrc or .profile) producing output during non-interactive SSH sessions, which corrupts the SCP protocol. Adding the -O flag to use the legacy SCP protocol bypassed this issue but revealed a different problem: the remote server was configured to reject root login entirely, redirecting to the theuser user instead.

The assistant then successfully copied the binary to the /tmp/ directory as the theuser user, verified the file's integrity, and located the running vast-manager process on the remote host. At this point, the assistant had a new binary on the remote machine at /tmp/vast-manager.new and needed to install it to /usr/local/bin/vast-manager.

WHY This Message Was Written

The message was written to complete the deployment in a single, efficient SSH invocation. The assistant had three goals:

  1. Create a backup of the current binary (vast-manager.bak) to enable quick rollback if the new version caused issues.
  2. Install the new binary by copying it from /tmp/ to /usr/local/bin/.
  3. Ensure correct permissions by making the new binary executable. Chaining these operations with && operators was a deliberate design choice. It ensured that if any step failed, the subsequent steps would not execute, preventing a partially deployed state. The backup-first approach reflected sound operational practice—always have a fallback before making destructive changes.

HOW Decisions Were Made

The assistant's decision-making process is visible in the structure of the command and the sequence of operations that preceded it. Several key decisions stand out:

Decision to use a single SSH session: Rather than making three separate SSH calls (one for backup, one for copy, one for chmod), the assistant bundled all operations into a single command string. This minimized latency, reduced the risk of partial failure, and ensured that the operations were executed in a predictable order on the remote host.

Decision to backup before overwriting: The backup step demonstrates an awareness of production deployment risk. By creating vast-manager.bak, the assistant preserved the ability to revert to the previous version quickly if the new binary exhibited problems.

Decision to use cp rather than mv: The assistant used cp for both the backup and the installation, rather than mv which would be atomic on the same filesystem. This choice is slightly unusual—if both source and destination are on the same filesystem, mv is typically preferred because it's an atomic rename operation. Using cp introduces a window where the destination file is being written and could be in an inconsistent state if read concurrently. However, cp works across filesystem boundaries, which may have been a consideration.

Decision to verify file integrity before deployment: In the preceding messages, the assistant used ls -la and file to verify that the copied binary had the correct size and was a valid ELF executable. This verification step shows an awareness that file transfers can silently corrupt data.

Assumptions Made by the Assistant

The deployment failure reveals several assumptions that turned out to be incorrect:

Assumption 1: The theuser user has write permission to /usr/local/bin/. This was the critical incorrect assumption. The assistant observed in message 3787 that the existing binary at /usr/local/bin/vast-manager was owned by theuser (with permissions -rwxrwxr-x). However, file ownership does not imply directory write permission. The /usr/local/bin/ directory is typically owned by root:root with permissions drwxr-xr-x, meaning only root can create or remove files in it. The fact that theuser owned the existing file was likely because it was created via sudo or by a previous deployment that used privilege escalation.

Assumption 2: The backup step would succeed. The assistant assumed that if the user could write to /usr/local/bin/ (to install the new binary), they could also create a backup file in the same directory. This assumption was reasonable but incorrect, as the directory permissions prevented any file creation by the theuser user.

Assumption 3: The deployment could proceed without privilege escalation. The assistant had been using the theuser user for all SSH operations and assumed this user had sufficient privileges for the deployment. The earlier SCP failure with root login should have been a signal that privilege escalation (via sudo) would be needed for system directory operations.

Assumption 4: The chained command would provide clear error reporting. While the && chaining did correctly stop execution on the first failure, the error message was somewhat ambiguous. "Permission denied" could refer to the source file, the destination directory, or the backup operation. The assistant had to infer from context that it was the backup operation that failed.

Mistakes and Incorrect Assumptions

The primary mistake was not verifying directory permissions before attempting the deployment. The assistant had access to all the information needed—it could have run ls -la /usr/local/bin/ to check directory permissions, or touch /usr/local/bin/test to test write access. Instead, it assumed that file ownership implied directory write permission, which is a common but dangerous assumption in Unix systems.

A secondary mistake was not including sudo in the command. The assistant knew that root login was disabled (from the earlier SCP failure), but didn't consider that the theuser user might have sudo privileges. In many server configurations, users in the sudo or wheel group can execute commands as root without providing a password when the session originates from a trusted source. The command could have been:

ssh theuser@10.1.2.104 'sudo cp /usr/local/bin/vast-manager /usr/local/bin/vast-manager.bak && sudo cp /tmp/vast-manager.new /usr/local/bin/vast-manager && sudo chmod +x /usr/local/bin/vast-manager'

A third mistake was the choice of cp over mv for the backup. Using cp to create a backup of a binary file involves reading the entire file and writing it to a new location, which is slower and more resource-intensive than a simple rename. If the backup was intended to be a quick rollback mechanism, mv would have been more appropriate, though it would destroy the original file in the process.

Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

Unix file permissions: Understanding the difference between file ownership and directory write permissions is essential. The /usr/local/bin/ directory typically has permissions 755 (drwxr-xr-x), meaning only the owner (root) can create or delete files, while others can read and execute. File ownership of individual files within the directory does not grant directory write access.

SSH and remote execution: Knowledge of how SSH executes commands, how scp and sftp protocols work, and how shell startup files can interfere with non-interactive sessions is necessary to understand the preceding deployment attempts.

Go build and deployment: Understanding that the assistant compiled a Go binary locally and needed to transfer it to a remote server for deployment provides context for why the binary needed to be moved.

Production deployment practices: The concepts of backup-before-deployment, atomic operations, and rollback planning are standard in production operations and inform the structure of the deployment command.

The vast-manager application: Knowledge that vast-manager is a management service for Vast.ai cloud instances running cuzk/curio proving workloads, and that it had been experiencing SSH connectivity issues that motivated the binary update.

Output Knowledge Created

This message, despite being a "failure" in the sense that the deployment did not complete, creates valuable knowledge:

The deployment strategy needs revision: The permission denied error reveals that the current approach of copying to /tmp/ and then moving to /usr/local/bin/ as the theuser user is insufficient. The deployment must either use sudo for privilege escalation or deploy to a location where the user has write permission.

The backup mechanism is viable but blocked: The fact that the backup step failed first (before the copy step) confirms that the issue is with directory write permission, not with any specific file operation. This narrows the problem space considerably.

The SSH connectivity to the remote host is working: Despite the deployment failure, the SSH connection itself succeeded. The assistant was able to authenticate as theuser, execute commands on the remote host, and receive output. This is important context—the SSH fix from earlier messages was not the cause of the deployment failure.

The chained command structure is effective: The && operators correctly halted execution after the first failure, preventing a scenario where the new binary was copied but the backup was not created. This demonstrates the value of defensive command chaining in deployment scripts.

The Thinking Process Visible in Reasoning

While message 3788 itself contains only a bash command and its output, the thinking process is visible in the surrounding messages and in the structure of the command itself.

The assistant's reasoning appears to follow this logic:

  1. "I have a new binary at /tmp/vast-manager.new on the remote host."
  2. "The current binary is at /usr/local/bin/vast-manager and is owned by theuser."
  3. "I need to replace it with the new version, but I should create a backup first."
  4. "I can do all three operations (backup, copy, chmod) in one SSH command."
  5. "Using && will ensure atomicity—if any step fails, the rest won't execute."
  6. "The backup should come first, so if something goes wrong, we can always revert." The chain of reasoning that led to the incorrect assumption about directory permissions is also traceable. The assistant had just verified (in message 3787) that the existing binary was owned by theuser with permissions -rwxrwxr-x. From this, it appears the assistant inferred that the theuser user had write access to the directory containing the file. This is a logical leap—file ownership and directory permissions are independent in Unix, but the connection is not always obvious, especially under time pressure. The assistant also appears to have been operating under the assumption that since the file was writable by the owner and group (rwxrwx), the user who owned it could modify it freely. This is true for the file itself, but creating a new file (the backup) requires directory write permission, which is a separate concern.

Broader Implications for Production Deployments

This single message illustrates several important principles for production deployment:

Verify permissions before deploying: Always test that the deployment user has the necessary permissions for all operations—creating backups, writing files, setting permissions—before attempting the actual deployment. A simple touch /usr/local/bin/test && rm /usr/local/bin/test can save time and prevent partial deployments.

Use privilege escalation explicitly: When deploying to system directories, use sudo explicitly rather than assuming the deployment user has write access. Even if the user owns files in the directory, directory permissions may prevent new file creation.

Test the backup mechanism: The backup step is only useful if it can actually be executed. Testing the backup before making changes ensures that rollback is possible.

Consider atomic deployment strategies: Using mv (rename) instead of cp for the installation step provides atomicity—the new binary appears instantaneously at the target path. Combined with a backup, this allows for clean rollback by simply renaming the backup back to the original name.

Plan for failure at every step: The chained command structure was correct in principle—it prevented partial deployment. However, the failure mode (permission denied on backup) left the system in its original state, which was the best possible outcome given the circumstances.

Conclusion

Message 3788 captures a moment of failure that is rich with meaning. A single permission denied error, occurring during what should have been a routine binary deployment, exposes the hidden complexity of production infrastructure. The assistant's methodical approach—building a new binary, copying it to the remote host, verifying its integrity, and attempting a safe deployment with backup—was sound in every respect except one: it failed to verify that the deployment user had write permission to the target directory.

This failure is instructive. It reminds us that in Unix systems, file ownership and directory permissions are independent concerns, and that assumptions about one do not translate to the other. It demonstrates the value of defensive command chaining, the importance of testing permissions before deployment, and the need for explicit privilege escalation when operating in system directories.

Most importantly, it shows that even failed operations produce valuable knowledge. The permission denied error, while frustrating, provided clear diagnostic information that guided the next steps. The assistant learned that the theuser user needed sudo access for system directory operations, and the deployment strategy was revised accordingly. In production engineering, a clean failure with clear diagnostics is often more valuable than a silent success.