The Vanishing Virtual Environment: Debugging Python Activation Over SSH in an ML Deployment Pipeline

Introduction

In the course of deploying a 744-billion-parameter Mixture-of-Experts language model (GLM-5-NVFP4) across eight NVIDIA RTX PRO 6000 Blackwell GPUs, a seemingly trivial problem derailed the entire installation pipeline: the Python virtual environment would not activate properly over SSH. Message [msg 92] captures a single, deceptively simple debugging step in this saga — a moment where the assistant realizes that source activation isn't working through a remote shell, attempts a workaround using bash -l -c, and hits yet another wall. This message, barely a dozen lines long, encapsulates a class of problems that plague every ML engineer working with remote GPU servers: environment management across non-interactive SSH sessions.

The full text of the message reads:

[assistant] The source didn't activate properly over SSH. Let me use the full path. [bash] ssh 10.1.230.175 'bash -l -c "source ~/ml-env/bin/activate && which pip && pip install --upgrade pip 2>&1 | tail -5"' /usr/bin/pip

See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.

The Context: Why This Message Exists

To understand why this message was written, we must trace back through the preceding conversation. The session began with the ambitious goal of setting up a full ML environment on Ubuntu 24.04 with eight Blackwell GPUs ([msg 80]). The assistant had already successfully installed NVIDIA drivers (590.48.01), CUDA Toolkit 13.1, a secondary CUDA 12.8 toolkit for PyTorch compatibility, and a full Python environment using uv — including PyTorch 2.9.1, flash-attn 2.8.3, vLLM 0.15.1, and a dozen other ML libraries.

The user then issued a new directive ([msg 81]): deploy the GLM-5-NVFP4 model using sglang, which likely required a main-branch or nightly build. The assistant verified all eight GPUs were visible ([msg 87]), explored the local research repository for load testing tools ([msg 89]), and then turned to installing sglang.

Message [msg 90] checked whether sglang was already installed — it was not. Message [msg 91] attempted the obvious: pip install "sglang[all]" after activating the venv with source ~/ml-env/bin/activate. But the command failed with an externally-managed-environment error — a PEP 668 protection that prevents pip from installing packages system-wide when Python is managed by the OS package manager.

This error is the smoking gun. It tells us that the virtual environment was not active when the pip command ran. Instead, the system Python at /usr/bin/pip was invoked. The source activation had silently failed.

Message [msg 92] is the assistant's response to discovering this failure. It is a debugging pivot: the assistant recognizes the activation didn't work, hypothesizes that SSH is the culprit, and tries a more robust invocation strategy.## The Reasoning: Why source Fails Over SSH

The assistant's reasoning in [msg 92] is sharp and practical. The first sentence — "The source didn't activate properly over SSH" — is a diagnosis, not a guess. The evidence is clear: the error message came from /usr/bin/pip, not from the venv's pip. The activation command source ~/ml-env/bin/activate had run without error in the previous message, but it had no lasting effect.

This is a well-known subtlety of SSH command execution. When you run ssh host 'source venv/bin/activate && python ...', each SSH invocation starts a fresh, non-interactive shell. The source command modifies environment variables (PATH, VIRTUAL_ENV, PS1) in that shell session, but those modifications only persist for the duration of the single command string. If the command string is a compound command (using &&), the environment carries through. But the assistant in [msg 91] had run the activation and pip install as separate SSH commands — or rather, the activation was done in one SSH call and the pip install in another. The environment never persisted across SSH invocations.

The assistant's attempted fix in [msg 92] — bash -l -c "source ~/ml-env/bin/activate && which pip && pip install --upgrade pip" — is a reasonable next step. The -l flag tells bash to act as a login shell, which loads .bash_profile, .bash_login, and .profile. This can sometimes help if the activation script is sourced from one of those files. The -c flag passes a command string. The idea is to force a full login shell environment before activating the venv.

But the result shows /usr/bin/pip — the system pip, not the venv pip. This means the activation still didn't work. Why?

The Hidden Assumption: What Went Wrong

The assistant made a critical assumption: that source ~/ml-env/bin/activate would work inside bash -l -c. The activation script typically modifies PATH by prepending the venv's bin directory. But bash -l -c starts a login shell that reads profile files, and those profile files may override or reset PATH after the activation script runs. The exact order depends on what's in /etc/profile, ~/.profile, and ~/.bashrc.

More fundamentally, the assistant assumed that the venv was properly set up with a working activate script. The venv was created with uv, not with the standard python3 -m venv. While uv creates standard-compliant virtual environments, there can be subtle differences. The uv-created venv at ~/ml-env was confirmed to exist in [msg 94] (the next message), but the bin/ directory listing showed no pip binary — only python3. This is a crucial detail: uv does not install pip into the venv by default. It uses its own resolver and installer. So even if activation had worked, which pip would have found the system pip, not a venv-local pip.

The assistant didn't yet know this at the time of [msg 92]. The discovery that pip was missing from the venv's bin/ directory only came in [msg 94] and [msg 95], after more debugging. So [msg 92] represents an intermediate hypothesis: "activation failed because of SSH shell behavior" — a correct observation but an incomplete diagnosis.

Input Knowledge Required

To fully understand [msg 92], the reader needs several pieces of context:

The PEP 668 error: The previous message ([msg 91]) failed with externally-managed-environment. This is a Python packaging standard (PEP 668) adopted by Debian/Ubuntu that marks system Python installations as "externally managed" to prevent pip from conflicting with apt. The error message explicitly says to use a virtual environment. The assistant knows the venv exists at ~/ml-env (created in segment 0) and that it should be activated before pip commands.
SSH non-interactive shell behavior: SSH executes commands in a non-interactive, non-login shell by default. Environment variables set by source or export in one SSH command do not persist to the next. This is a fundamental Unix SSH behavior that anyone managing remote servers must understand.
The bash -l -c pattern: Using bash -l to force a login shell is a common workaround for SSH environment issues. It loads profile files that may set PATH, LD_LIBRARY_PATH, and other critical variables. However, it can also introduce unexpected overrides.
The venv structure: A Python virtual environment's bin/ directory contains python3, activate scripts, and (if pip was installed) pip. The activate script modifies PATH to point to the venv's bin/ first, so python3 and pip resolve to the venv versions.
The uv package manager: The environment was set up with uv ([msg 80]), which is a fast Python package manager and resolver. uv can create venvs without pip, which is exactly what happened here.## The Thinking Process: A Microcosm of Remote Debugging The reasoning visible in [msg 92] follows a classic debugging pattern:
Observe the symptom: The previous command failed with an externally-managed-environment error, which only happens when using the system pip, not a venv pip.
Form a hypothesis: The source activation didn't work over SSH.
Design an experiment: Use bash -l -c to force a login shell, which should load profile files and potentially fix environment propagation.
Execute and observe: The result shows /usr/bin/pip — the system pip is still being found.
Interpret the result: The activation still failed, but now the assistant has more information (the system pip path is confirmed). The assistant does not, in this message, jump to conclusions. It doesn't immediately blame the venv structure or the uv tool. It tests one hypothesis at a time. The message is a single step in an iterative debugging process — and that's exactly how effective remote debugging works. What's notable is what the assistant doesn't do in this message: it doesn't check whether the venv's bin/ directory contains a pip binary. It doesn't verify the activation script's contents. It doesn't try ~/ml-env/bin/python3 -m pip as an alternative. These checks come in subsequent messages ([msg 94], [msg 95]), showing that debugging is a process of narrowing possibilities, not a single leap to the correct answer.

Mistakes and Incorrect Assumptions

While the assistant's reasoning is sound, there are several implicit assumptions that turned out to be incorrect:

Assumption 1: The venv has pip installed. The assistant assumed that pip would be available inside the activated venv. In a standard python3 -m venv setup, pip is included by default (unless --without-pip is passed). But uv creates venvs differently — it installs only what's needed for its own operations. The pip binary was simply not there.

Assumption 2: bash -l -c would fix the activation. The login shell flag loads profile files, but those files might override PATH in ways that defeat the activation. On Ubuntu 24.04, /etc/profile may source /etc/bash.bashrc or other files that set PATH to system defaults. If the activation script runs before these overrides, the effect is lost.

Assumption 3: The activation script is the only way to use the venv. The assistant could have directly invoked ~/ml-env/bin/python3 -m pip install sglang[all] without any activation. This is actually the most robust approach for non-interactive SSH commands — use absolute paths to the venv binaries. The assistant eventually discovers this in later messages, but [msg 92] still operates within the "activate then use" paradigm.

Assumption 4: The problem is purely about SSH environment propagation. While SSH non-interactive shell behavior is a real issue, it wasn't the root cause here. The root cause was that uv didn't install pip into the venv. The SSH activation issue was a red herring — even if activation had worked perfectly, which pip would still have returned /usr/bin/pip because there was no venv-local pip to find.

This last point is particularly instructive. The assistant correctly identified a real problem (SSH environment isolation) but attributed the wrong symptom to it. The PEP 668 error was not caused by SSH — it was caused by the absence of pip in the venv. The SSH activation issue was a separate, coincidental problem that the assistant happened to notice while debugging.

Output Knowledge Created

Message [msg 92] produces several valuable pieces of knowledge:

Confirmed system pip path: The output /usr/bin/pip confirms that the system Python installation is being used, not the venv. This is evidence that the venv activation is not working as expected.
PEP 668 confirmation: The error message confirms that Ubuntu 24.04 has PEP 668 protections enabled. This means any pip operations must happen inside a virtual environment or use --break-system-packages (which is discouraged).
bash -l -c is not sufficient: The login shell approach did not fix the activation issue. This rules out one class of solutions and narrows the debugging path.
The venv activation script may have issues: The fact that source inside a login shell still doesn't work suggests either the activation script itself is problematic, or the venv structure is non-standard. This knowledge directly drives the next steps: checking the venv's bin/ directory contents ([msg 94]), discovering that pip is missing ([msg 95]), and ultimately switching to uv for package installation or installing pip into the venv.## Broader Lessons for ML Engineering This message, though small, illuminates several enduring truths about deploying ML models on remote infrastructure: Environment management is the hardest part of ML engineering. The glamorous work — loading models, tuning parameters, running benchmarks — depends on a fragile stack of drivers, CUDA toolkits, Python packages, and environment variables. A single missing pip binary can halt an entire deployment. The assistant's struggle here is not unusual; it's the daily reality of anyone working with GPU servers. SSH is a leaky abstraction for remote Python work. The non-interactive shell model means that every SSH command is a fresh environment. Best practices include: using absolute paths to venv binaries (~/ml-env/bin/python3), wrapping multi-step operations in a single script that handles its own activation, or using tools like screen/tmux for persistent sessions. The assistant eventually adopts the absolute-path approach, which is the most robust solution. The uv package manager changes assumptions. uv is increasingly popular for its speed and reliability, but it behaves differently from pip in several ways. One difference is that it doesn't install pip into the venv by default. Anyone migrating from pip to uv needs to be aware of this, or they'll hit exactly the error seen here. Debugging is hypothesis testing, not guessing. The assistant's methodical approach — observe symptom, form hypothesis, test, interpret — is textbook debugging. Each message in the conversation is one iteration of this loop. Message [msg 92] is a single iteration that disproves one hypothesis (that bash -l -c would fix activation) and generates evidence for the next.

Conclusion

Message [msg 92] is a small but revealing moment in a larger deployment story. It captures the instant when a seasoned practitioner realizes that something fundamental is wrong with their environment setup, tests a reasonable fix, and discovers that the problem runs deeper than expected. The message is a testament to the fact that deploying large language models is not just about understanding attention mechanisms and tensor parallelism — it's about knowing how SSH handles environment variables, how uv creates virtual environments, and how PEP 668 protects system Python installations.

The assistant's response to this setback is instructive: it doesn't panic, doesn't repeat the same failed command, and doesn't jump to conclusions. It calmly notes the failure, tries a targeted fix, and uses the output to inform the next step. This is the essence of effective remote debugging, and it's a skill that every ML engineer must cultivate.

In the end, the GLM-5-NVFP4 model was deployed successfully across eight Blackwell GPUs — but not before the assistant navigated a maze of environment issues, CUDA version mismatches, and SSH quirks. Message [msg 92] is a snapshot of one step in that maze, and it reminds us that even the most sophisticated AI systems depend on the mundane, finicky details of operating system configuration.