The Moment of Discovery: When SSH Replaces the Docker Entrypoint

Introduction

In the complex dance of deploying distributed GPU proving infrastructure on Vast.ai, there comes a moment when theory meets practice—and practice wins. Message 921 of this coding session captures exactly such a moment: the instant when the assistant, having rebuilt a Docker image, pushed it, destroyed an old instance, and created a shiny new one, SSHes into the running container only to discover that nothing is working as expected. The entrypoint never ran. The instance never registered with the manager. And the root cause, visible in a single ps aux command, reveals a fundamental truth about how Vast.ai's SSH launch mode works: it replaces the Docker ENTRYPOINT entirely.

This article examines that message in depth: the reasoning that led to it, the assumptions that were broken, the knowledge it produced, and the debugging methodology on display.

The Subject Message

The message reads as follows:

No registration yet. Let me SSH into the instance to see what's happening — is the entrypoint running or did SSH mode replace it?

>

``bash ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 -p 48106 root@70.69.192.6 "ps aux | head -30" 2>&1 Warning: Permanently added '[70.69.192.6]:48106' (ED25519) to the list of known hosts. Welcome to vast.ai. If authentication fails, try again after a few seconds, and double check your ssh key. Have fun! USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.1 0.0 2800 1032 ? Ss 00:31 0:00 /bin/sh -c while [ ! -e /.launch ]; do sleep 1; done; bash /.launch root 38 0.0 0.0 4324 3092 ? S 00:31 0:00 bash /.launch ``

At first glance, this is a simple debugging step: the assistant notices the instance hasn't registered with the manager, so it connects via SSH to investigate. But the output tells a deeper story. PID 1—the process that should be the Docker ENTRYPOINT—is instead running Vast.ai's own init script, /.launch. The assistant's own entrypoint.sh, carefully crafted to start the portavailc tunnel, run the benchmark, and register with the manager, is nowhere to be seen.

Context and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the thread of events that led to it. The broader session (Segment 7 of the conversation) is about deploying and managing GPU proving instances on the Vast.ai marketplace. The assistant and user have been iterating on a Docker-based system for running Filecoin proof computations (CuZK proofs) on rented GPU hardware.

The immediate preceding events are critical. In messages 900–918, the assistant:

  1. Rebuilt the Docker image (Dockerfile.cuzk) with updated scripts
  2. Pushed it to Docker Hub as theuser/curio-cuzk:latest
  3. Destroyed the old instance (32709851) that was running stale software
  4. Created a new instance (32710414) using the vastai create instance command with the --ssh flag The creation command was: vastai create instance 29854362 --image theuser/curio-cuzk:latest --disk 250 --env '-e PAVAIL=... -e PAVAIL_SERVER=...' --ssh --direct The --ssh flag was included deliberately—it provides SSH access to the container for debugging, which the user and assistant had relied on throughout the session. The assumption was that --ssh would layer SSH access on top of the existing Docker ENTRYPOINT, allowing both the automated lifecycle (entrypoint.sh) and interactive debugging (SSH) to coexist. After creation, the assistant waited (messages 918–920) for the instance to start and register. The instance showed as "running" with an SSH port available, but the vast-manager logs showed no new registration. After waiting 30+ seconds, the assistant decided to investigate directly—and message 921 is that investigation. The motivation is therefore diagnostic: "Why isn't the instance registering?" The assistant suspects a specific hypothesis—that SSH mode might have replaced the entrypoint—and tests it with a single ps aux command. The question embedded in the message ("is the entrypoint running or did SSH mode replace it?") is both a rhetorical framing for the reader and a genuine hypothesis the assistant is about to confirm.

The Thinking Process: A Model of Debugging Methodology

The assistant's reasoning in this message is a textbook example of hypothesis-driven debugging. Several cognitive steps are visible:

Step 1: Observation of anomaly. The instance is running (status: "success, running theuser/curio-cuzk_latest/ssh"), the SSH port is open, but no registration occurred. The manager's monitor loop shows only 2 cached instances (the old ones), not 3. Something is wrong.

Step 2: Formation of competing hypotheses. The assistant implicitly considers two possibilities: (a) the entrypoint is running but hasn't completed yet (perhaps the portavailc tunnel is slow, or the parameter download is taking time), or (b) the entrypoint isn't running at all because SSH mode replaced it. The phrasing "is the entrypoint running or did SSH mode replace it?" shows these two hypotheses being weighed.

Step 3: Selection of a diagnostic test. The assistant chooses ps aux | head -30 as the diagnostic tool. This is a smart choice: it shows all running processes, including PID 1 (the init process of the container), which directly reveals whether the Docker ENTRYPOINT or Vast's init script is in control.

Step 4: Interpretation of results. The output is unambiguous. PID 1 is /bin/sh -c while [ ! -e /.launch ]; do sleep 1; done; bash /.launch. This is Vast.ai's SSH launch script, not the Docker ENTRYPOINT. The assistant's hypothesis (b) is confirmed.

What's notable is the economy of the debugging step. The assistant doesn't check logs, doesn't try to restart services, doesn't look at environment variables. It goes straight to the root cause with a single, well-chosen command. This reflects deep familiarity with both Docker internals (PID 1 is always the entrypoint) and Vast.ai's platform behavior (the /.launch script is the hallmark of SSH mode).

Assumptions Made and Broken

This message is rich with assumptions—some validated, some broken.

Broken assumption: --ssh preserves the Docker ENTRYPOINT. The most critical broken assumption is that adding --ssh to the vastai create instance command would layer SSH access on top of the existing entrypoint. In reality, Vast.ai's SSH launch mode replaces the ENTRYPOINT entirely with its own init script (/.launch). This is documented behavior, but it's easy to miss in the vast documentation landscape. The assistant had read about environment variable propagation earlier (message 897) but hadn't internalized this specific detail about entrypoint replacement.

Validated assumption: The instance is actually running. The SSH connection succeeds, proving the container is alive and the network is working. This rules out infrastructure-level failures (network down, container crash, OOM kill).

Validated assumption: The manager's lack of registration is due to the entrypoint not running. The assistant had been waiting for registration, suspecting a timing issue. The ps aux output confirms it's not a timing issue—the entrypoint literally never executed.

Implicit assumption: ps aux reveals PID 1. The assistant relies on the Unix convention that PID 1 is the first process in the container (the entrypoint). This is correct for Docker containers and is the key insight that makes the diagnostic work.

Implicit assumption: The /.launch script is Vast.ai's SSH init. The assistant recognizes /.launch as Vast's own script. This requires prior knowledge of Vast.ai's platform internals—someone unfamiliar with the platform might not know what /.launch signifies.

Input Knowledge Required to Understand This Message

To fully grasp what's happening in message 921, a reader needs several pieces of background knowledge:

  1. Docker container fundamentals: Understanding that PID 1 in a container is the ENTRYPOINT process, and that replacing it changes the entire lifecycle of the container.
  2. Vast.ai platform architecture: Knowing that Vast.ai offers different "launch modes" (--ssh, --jupyter, --entrypoint, --direct) that determine how the container starts. The --ssh mode injects an init script that sets up SSH and runs an optional onstart.sh script.
  3. The project's architecture: Understanding that the Docker image has an entrypoint.sh that handles: starting the portavailc tunnel (for secure port forwarding), running the benchmark, and registering the instance with the vast-manager service running on a central controller host.
  4. The vast-manager system: Knowing that instances self-register by contacting the manager API, and that the manager's monitor loop periodically caches instance lists from the Vast API.
  5. The preceding conversation: Understanding that the assistant had just rebuilt the Docker image, pushed it, destroyed an old instance, and created a new one—all in the expectation that the new instance would automatically run the lifecycle.
  6. Unix process listing: Knowing that ps aux shows all processes with their PIDs, and that PID 1 is special in Unix containers.

Output Knowledge Created by This Message

Message 921 produces several pieces of valuable knowledge:

  1. Empirical confirmation that --ssh replaces the ENTRYPOINT. Before this message, the assistant suspected this might be the case (the question in the message text shows it was a hypothesis). After this message, it's a confirmed fact. The /.launch script as PID 1 is definitive evidence.
  2. The exact process tree of an SSH-mode container. The output shows /.launch running as PID 1, with a child process also running bash /.launch. This reveals the startup sequence: Vast's init script polls for the existence of /.launch, then executes it.
  3. A concrete debugging methodology. Future readers of this conversation learn that when an instance doesn't behave as expected, SSHing in and checking ps aux is a fast way to determine whether the intended entrypoint is running.
  4. The need for an alternative deployment strategy. This message implicitly creates the requirement for a different approach. If --ssh replaces the ENTRYPOINT, then the automated lifecycle must be triggered through Vast's onstart mechanism or by using a different launch mode. The assistant's next steps (messages 922–924) involve destroying this instance and recreating it with an --onstart-cmd that calls the entrypoint in the background.
  5. Documentation of a platform quirk. For anyone deploying on Vast.ai, this message serves as a warning: SSH mode is not a simple addition to your Docker setup—it fundamentally changes how your container starts. Your carefully crafted ENTRYPOINT is ignored unless you use the --entrypoint launch mode or trigger it via --onstart-cmd.

Mistakes and Corrective Insights

While the assistant's debugging is sound, there are mistakes and near-misses worth examining:

The mistake of using --ssh without understanding its implications. The assistant had read Vast.ai documentation earlier (message 897) about environment variables not being visible in SSH sessions, but hadn't fully absorbed the corollary: SSH mode replaces the ENTRYPOINT. The documentation page on Docker execution environment mentions this, but it's easy to overlook when focused on the environment variable issue.

The near-miss of the PAVAIL environment variable. Earlier (messages 911–916), the assistant had a separate issue where the PAVAIL secret was mangled because shell expansion happened incorrectly. That was fixed, but it shows how multiple configuration details can go wrong simultaneously—and how the assistant had to iterate through several failures before reaching this one.

The correct decision to SSH in rather than wait longer. One could imagine waiting 60–90 seconds, assuming the entrypoint was slow. But the assistant's decision to investigate early saved time. The ps aux command immediately revealed the root cause, whereas waiting would have been futile.

The implicit trust in the Docker build. The assistant assumes the Docker image is correct (entrypoint.sh is properly set as ENTRYPOINT in the Dockerfile). This assumption is validated by the fact that the image built and pushed successfully. The problem is not the image but how Vast.ai launches it.

Broader Significance: The Gap Between Local and Cloud

This message illustrates a recurring theme in cloud infrastructure work: the gap between how software behaves locally and how it behaves on a cloud platform. The Docker image works perfectly when run with docker run on the build host—the ENTRYPOINT executes, the portavailc tunnel starts, the benchmark runs, the manager registration happens. But on Vast.ai, the same image behaves differently because the platform interposes its own init system.

This is not a bug in Vast.ai—it's a design choice. SSH mode is designed for interactive debugging, where you want a shell before anything else runs. The --entrypoint mode is designed for automated workloads. The mistake was using the wrong mode for the wrong purpose.

The lesson is broader than Vast.ai: every cloud platform has its own quirks about how containers start, what environment variables are available, and what init scripts run. Heroku, AWS ECS, Google Cloud Run, Kubernetes—each has its own startup lifecycle. The portable Docker image is a myth; in practice, you must understand the platform's startup sequence to deploy successfully.

Conclusion

Message 921 is a small but pivotal moment in a much larger debugging journey. In one SSH command and its output, the assistant discovers why the automated deployment failed, learns a crucial fact about the Vast.ai platform, and sets the stage for the correct solution (using --onstart-cmd instead of relying on the ENTRYPOINT).

The message is a masterclass in efficient debugging: form a hypothesis, choose the minimal diagnostic test, interpret the results immediately, and act on the new knowledge. The ps aux output tells a story that no log file could: the story of a Docker ENTRYPOINT that never got to run, replaced by a platform init script that the assistant didn't know would take over.

In the end, this message is about the gap between intention and reality in distributed systems. The assistant intended for the entrypoint to run. The reality was /.launch. And the bridge between them was a single, well-aimed SSH command.