The $0.28/hr Breakthrough: How a Docker Entrypoint Workaround Unlocked Automated Filecoin Proving on Vast.ai

On the surface, message [msg 926] appears unremarkable: a brief verification that a cloud GPU instance has started successfully. The assistant runs a vastai show instance command after a 40-second sleep, receives a JSON blob confirming the instance is running with an SSH port, and moves on. But this message represents the culmination of a multi-hour debugging odyssey—a moment where a deeply counterintuitive platform quirk was finally tamed, and the entire deployment architecture clicked into place.

The Message Itself

The message reads in full:

Instance 32710471 created. Let me wait for it to come up and verify:

>

[bash] sleep 40 && ssh 10.1.2.104 "vastai show instance 32710471 --raw 2>/dev/null" | jq '{id: .id, status_msg: .status_msg, public_ipaddr: .public_ipaddr, ssh_port: .ports["22/tcp"][0].HostPort, label: .label, actual_status: .actual_status, extra_env: .extra_env}'

>

{ "id": 32710471, "status_msg": "success, running theuser/curio-cuzk_latest/ssh", "public_ipaddr": "70.69.192.6", "ssh_port": "48191", "label": null, "actual_status": "running", "extra_env": { "PAVAIL": "[REDACTED_PAVAIL_SECRET]", "PAVAIL_SERVER": "45.33.141.226:22222" } }

Beneath this calm exterior lies a story of failed assumptions, platform-specific debugging, and a clever workaround that reconciled two conflicting requirements: the need for SSH access for debugging, and the need for a custom Docker entrypoint to manage the proving worker lifecycle.

The Debugging Trail That Led Here

To understand why this message matters, one must trace the path that preceded it. The assistant had been building a system to deploy Filecoin proof workers (using the CuZK proving engine) on rented GPU instances from Vast.ai. The architecture required a custom Docker entrypoint (entrypoint.sh) that would:

  1. Start a portavailc tunnel for secure communication back to a controller host
  2. Register the instance with a central manager service (vast-manager)
  3. Run benchmarks and eventually participate in live proving The first attempt at deploying this system used vastai create instance ... --ssh --direct, which seemed like the natural choice—it provides SSH access for debugging while running the user's Docker image. But instance 32710414, created moments earlier, never registered with the manager. When the assistant SSH'd into it (see [msg 921]), the root cause became clear:
root           1  0.1  0.0   2800  1032 ?        Ss   00:31   0:00 /bin/sh -c while [ ! -e /.launch ]; do sleep 1; done; bash /.launch

PID 1 was Vast.ai's own /.launch script, not the Docker entrypoint. The assistant had discovered a critical Vast.ai platform behavior: the --ssh launch mode replaces the Docker image's ENTRYPOINT with Vast's own init system. This is documented but easy to miss, and it completely broke the deployment model. The entrypoint that was supposed to start the portavailc tunnel, register with the manager, and manage the worker lifecycle was simply never executed.

The Reasoning Behind the Fix

The assistant's thinking process in [msg 922] and [msg 923] reveals a careful evaluation of alternatives:

  1. Use --entrypoint mode instead of --ssh: This would preserve the Docker entrypoint but sacrifice SSH access entirely, making debugging impossible if something went wrong.
  2. Put startup logic in an onstart script: Vast's /.launch script does run /root/onstart.sh if it exists, so the entrypoint logic could be placed there.
  3. Not use --ssh and configure SSH manually: This would require modifying the Docker image to include and configure an SSH server, adding complexity and potential security issues. The chosen solution was elegant: use --ssh --onstart-cmd to run the entrypoint in the background. The --onstart-cmd flag tells Vast to execute a command after its own initialization completes, but crucially, it runs alongside the SSH daemon rather than replacing it. The command used was:
nohup /usr/local/bin/entrypoint.sh > /var/log/entrypoint.log 2>&1 &

This runs the entrypoint as a background process, redirecting its output to a log file for debugging. The SSH daemon continues to run in the foreground, providing interactive access. Both requirements are satisfied.

Assumptions Made and Validated

This message validates several critical assumptions:

That --onstart-cmd would work with --ssh: The assistant had to infer this from the vastai create instance --help output and the structure of the /.launch script. The help text mentions --onstart-cmd but doesn't fully explain its interaction with SSH mode. The assistant's reasoning—that the .launch script runs onstart.sh in the background, and --onstart-cmd would do the same—proved correct.

That 40 seconds is sufficient startup time: The sleep 40 before verification assumes the instance will be fully initialized within that window. This is a reasonable heuristic based on previous experience, but it's fragile—network conditions or host load could cause failures that manifest as timeouts rather than clear errors.

That the environment variables would be preserved: The PAVAIL and PAVAIL_SERVER env vars, which were mangled in a previous attempt (see [msg 912] where $(cat /etc/portavaild/secret) was passed literally), are now correctly set as literal strings. The assistant had learned from that mistake and passed the actual secret value.

That the label field being null is acceptable: The instance's label is null, but the assistant had previously confirmed (in [msg 898]) that VAST_CONTAINERLABEL is set by Vast.ai in every container—it's just not visible in SSH sessions. The entrypoint would receive it correctly via the environment, so a null label in the API response is not a problem.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message produces several valuable outputs:

  1. Confirmation that instance 32710471 is healthy: The actual_status: "running" and status_msg: "success, running theuser/curio-cuzk_latest/ssh" indicate the Docker image loaded correctly and the SSH daemon is operational.
  2. SSH connection details: Port 48191 on IP 70.69.192.6 provides the access point for debugging. The assistant immediately uses this in the next message ([msg 927]) to check the entrypoint log.
  3. Validation of the --onstart-cmd workaround: The fact that the instance reaches "running" state with SSH access proves that the approach works. The subsequent message confirms the entrypoint is actually executing—the log shows [entrypoint] 00:33:20 Starting portavailc tunnel to 45.33.141.226:22222 ....
  4. A reusable deployment pattern: The combination of --ssh --onstart-cmd with a backgrounded entrypoint becomes the standard pattern for all future instance creation in this session. It's applied again in <msg id=...> for instances on hosts 93197 and 88910.

The Broader Significance

This message represents a transition from platform-specific debugging to operational deployment. The earlier part of the session was consumed by understanding Vast.ai's quirks—the invisible VAST_CONTAINERLABEL, the entrypoint replacement in SSH mode, the mangling of shell-expanded environment variables. With instance 32710471, all of these issues have been addressed. The assistant can now focus on the actual proving workload: running benchmarks, diagnosing performance issues, and scaling across multiple GPU instances.

The 40-second wait and the JSON output are the quiet signal that the foundation is solid. The real work is about to begin.