The $5 Million Python Import Error

When a Symlink Isn't Enough

In the sprawling infrastructure of a decentralized GPU proving network, the smallest details can bring the entire system to a halt. Message 820 of this opencode session captures one such moment — a single bash command that reveals a fundamental mismatch between how the assistant thought the system worked and how it actually worked. The message itself is deceptively brief:

[assistant] [bash] ssh 10.1.2.104 'sudo vastai show instances --raw 2>&1 | head -5'
Traceback (most recent call last):
  File "/usr/local/bin/vastai", line 5, in <module>
    from vast import main
ModuleNotFoundError: No module named 'vast'

A Python traceback. A missing module. On the surface, it looks like a routine dependency error — the kind any developer encounters when setting up a new environment. But this message sits at the convergence of several threads: the deployment of a fleet management service, the subtle differences between user and system Python environments, and the assumptions that accumulate when building complex distributed systems. To understand why this message matters, we must trace the path that led to it.

The Context: Deploying the Vast-Manager

The assistant had been building and deploying a "vast-manager" service — a Go-based HTTP API that tracks GPU instances rented through vast.ai, a marketplace for cloud compute. The manager runs on a controller host (10.1.2.104) and communicates with the vast.ai API through the vastai command-line tool, a Python package installed via pip. The manager service itself runs as root under systemd, which means any tool it invokes must be accessible to the root user.

Earlier in the session, the assistant had installed the vastai CLI for the user theuser — a standard user account on the controller host. The pip package was installed in the user's local site-packages (/home/theuser/.local/lib/python*/site-packages/), and the vastai binary was placed at /home/theuser/.local/bin/vastai. When the assistant needed to make this tool available system-wide for the root-run manager service, it took what seemed like the obvious shortcut: create a symlink.

In message 818, the assistant ran:

sudo ln -sf /home/theuser/.local/bin/vastai /usr/local/bin/vastai

This placed a symlink at /usr/local/bin/vastai pointing to the user's local binary. The assistant then verified that vastai was findable via which and that it could list instances. But crucially, this verification was done without sudo — it ran as the regular user, whose Python environment included the vast package.

The First Sign of Trouble

In message 819, the assistant took the next logical step: copying the vast.ai API key to root's home directory and testing the command under sudo. The command was:

sudo vastai show instances --raw 2>&1 | python3 -c "import json,sys; data=json.load(sys.stdin); print(f\"{len(data)} instances\")"

The output was a truncated Python traceback showing a json.decoder.JSONDecodeError. At first glance, this looked like a data formatting issue — perhaps the vast API returned non-JSON output, or the --raw flag didn't work as expected. The assistant couldn't see the full error because the output was cut off (indicated by the trailing ...). The JSON parsing error was the visible symptom, but the root cause was something else entirely.

The Subject Message: Peeling the Onion

In message 820, the assistant tries again, this time redirecting stderr to stdout (2&gt;&amp;1) and piping through head -5 to capture just the beginning of the output. This is a classic debugging technique: when a command fails silently or produces confusing output, capture both stdout and stderr and examine the first few lines to understand what's actually happening.

The result is unambiguous:

Traceback (most recent call last):
  File "/usr/local/bin/vastai", line 5, in <module>
    from vast import main
ModuleNotFoundError: No module named 'vast'

The symlink works — the shell finds /usr/local/bin/vastai and executes it. But the file is a Python script, and on line 5 it tries to from vast import main. Python's import system searches for the vast package in its module search path (sys.path). When running as root via sudo, Python does not include the user's local site-packages directory in its search path. The vast package, installed only for theuser, is invisible to root's Python interpreter.

This is the fundamental issue: the vastai binary is not a standalone executable — it's a thin wrapper script that imports a Python package. Symlinking the wrapper is insufficient; the underlying package must also be accessible. The assistant's assumption that symlinking the binary would be equivalent to installing the tool system-wide was incorrect because it conflated two separate concerns: the executable's location in PATH and the Python package's location in sys.path.

Assumptions and Their Consequences

The assistant made several interconnected assumptions that this message exposes:

  1. Symlink sufficiency: The assumption that creating a symlink from a user-local binary to a system-wide location is enough to make a tool work for all users. This is true for compiled binaries (ELF executables) but not for Python scripts that depend on user-local packages.
  2. Environment transparency: The assumption that sudo preserves enough of the user's Python environment for imports to work. In reality, sudo resets PATH, PYTHONPATH, and other environment variables to system defaults, and root's Python has no knowledge of theuser's pip-installed packages.
  3. Error interpretation: In message 819, the assistant saw a JSON decode error and likely assumed the vast CLI was working but producing malformed output. The actual error was a Python import error that occurred before any JSON was produced — the python3 -c pipeline component then tried to parse the error message as JSON and failed, producing the secondary error that the assistant saw.
  4. Installation parity: The assistant assumed that copying the API key to root's home directory was the only configuration step needed. It overlooked that the tool itself (the Python package) needed to be installed system-wide, not just the configuration.

Input Knowledge Required

To fully understand this message, the reader needs:

  #!/usr/bin/python3
  from vast import main
  main()

The actual logic lives in the vast package, not in the wrapper script.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The exact failure point: Line 5 of /usr/local/bin/vastai, the from vast import main import statement. This is precise diagnostic information that tells us exactly where the Python import system fails.
  2. The missing dependency: The vast module is not installed for the Python interpreter that runs under sudo. This is a system-wide installation gap, not a configuration or permissions issue.
  3. The symlink's inadequacy: The symlink correctly resolves, but the tool still fails. This proves that the problem is not about finding the executable but about finding its dependencies.
  4. A clear remediation path: The fix must involve installing the vast Python package system-wide (e.g., sudo pip3 install vastai) or adjusting Python's module search path for root (e.g., setting PYTHONPATH or using sudo -E with the right environment).

The Thinking Process Visible in This Message

The assistant's reasoning unfolds across messages 818-820 in a clear debugging progression:

Message 818: The assistant creates the symlink and verifies it works for the current user. The verification command (which vastai &amp;&amp; vastai show instances --raw ...) runs without sudo, succeeding because it uses theuser's Python environment. The assistant concludes the setup is correct.

Message 819: The assistant escalates to testing under sudo, which is how the manager service will actually invoke the tool. It copies the API key to root's home and runs sudo vastai show instances --raw | python3 -c .... The output is a truncated JSON decode error. The assistant sees the secondary error (the Python one-liner failing to parse the output) but not the primary error (the import failure from vastai). This is a classic debugging pitfall: the pipeline obscures the root cause.

Message 820 (the subject): The assistant refines the approach. It adds 2&gt;&amp;1 to merge stderr with stdout (suspecting that the error might be on stderr) and pipes through head -5 to capture only the beginning (avoiding the truncation that plagued the previous attempt). This reveals the true error: ModuleNotFoundError: No module named &#39;vast&#39;. The assistant now has the precise diagnostic it needs.

This progression shows a methodical debugging approach: verify in the simple case, test in the realistic case, observe the failure, refine the diagnostic technique, and isolate the root cause. The assistant doesn't jump to conclusions or apply random fixes — it iteratively improves its understanding of the failure mode.

The Broader Lesson

This message is a microcosm of a class of bugs that plague Python-based infrastructure tools. The distinction between "installed for a user" and "installed system-wide" is invisible in day-to-day development work but becomes critical when tools are invoked by system services, cron jobs, or other non-user contexts. The symlink approach — so natural for compiled binaries — is a trap for Python tools because it addresses only the executable location, not the dependency resolution.

The fix, as shown in the subsequent message (msg 821), is to install the package system-wide with sudo pip3 install vastai --break-system-packages. But even that runs into complications (a conflicting urllib3 package installed by the system package manager), leading to yet another layer of debugging. Each layer reveals new assumptions and new constraints.

In the end, message 820 is a turning point: it transforms a confusing, truncated error into a clear, actionable diagnosis. It's a reminder that in complex systems, the most valuable output of any command is often not the expected result but the precise error message that tells you what's actually wrong.