The sitecustomize Trap: How a Python Environment Hook Nearly Derailed a 4-GPU LLM Deployment

Introduction

In distributed machine learning, the most insidious bugs are often not in the model code or the inference engine, but in the invisible layer of environment configuration that wraps around them. Message [msg 6166] captures a pivotal debugging moment in an opencode session where an AI assistant was deploying a Qwen3.5-122B-A10B BF16 model across 4 NVIDIA RTX PRO 6000 Blackwell GPUs. The message itself is deceptively brief — a single verification command and its output — but it represents a critical breakthrough in a multi-hour debugging saga. In it, the assistant realizes that a Python sitecustomize.py file is silently overriding NCCL environment variables, undoing every attempt to fix a distributed initialization hang from the shell level.

The Message

Here is the complete message as it appeared in the conversation:

Wait, the NCCL env vars are set in sitecustomize.py which runs before any env override. Let me verify: ``bash ssh root@10.1.230.174 '~/ml-env/bin/python3 -c "import os; print(os.environ.get(\"NCCL_P2P_LEVEL\", \"not set\"))"' `` SYS

The output is a single word: SYS. This confirms that the NCCL_P2P_LEVEL environment variable is set to SYS, which is the value assigned by sitecustomize.py — a Python script that runs automatically at interpreter startup and calls os.environ.setdefault() for a suite of NCCL tuning parameters.

The Debugging Context

To understand why this single verification was so consequential, we must trace the debugging path that led to it. The assistant had been trying for over an hour to get SGLang to initialize with tensor parallelism across 4 GPUs (TP=4). Every attempt ended the same way: the server logs would show "Init torch distributed begin" and then hang indefinitely, with GPU memory flatlined at roughly 1 GB per card — the CUDA context allocation, but no model weights loaded.

The assistant had already eliminated several possible causes. A driver version mismatch was discovered and fixed: the container had NVIDIA userspace libraries at version 565.57.01 while the host kernel module was at 590.48.01. Upgrading the container's packages to 590 resolved that discrepancy but did not fix the hang. A TP=1 test confirmed the model itself loaded correctly (it reached MoE weight allocation before OOMing on a single 96 GB GPU, which was expected for a 234 GB model). NCCL debug logging showed that NCCL's own initialization completed successfully — the ring was established, the bootstrap timings looked normal. The hang was therefore in PyTorch's torch.distributed.init_process_group, which wraps NCCL but adds its own rendezvous and rank coordination layer.

In message [msg 6165], the assistant attempted a brute-force override: launching the server with all NCCL environment variables explicitly set to empty strings, hoping to strip away the custom NCCL tuning that had been optimized for an 8-GPU topology and was now causing problems on 4 GPUs. The command was:

NCCL_PROTO="" NCCL_ALGO="" NCCL_P2P_LEVEL="" NCCL_MAX_NCHANNELS="" NCCL_BUFFSIZE="" NCCL_NTHREADS="" NCCL_DEBUG=WARN nohup ...

But this attempt failed silently — the log file was empty because the preceding fuser -k /dev/nvidia* command in the same SSH invocation killed the newly spawned processes before they could write anything. More importantly, even if the command had succeeded, the env var overrides would not have taken effect in the way the assistant assumed.

The sitecustomize Mechanism

This is where message [msg 6166] becomes the turning point. The assistant had earlier discovered a sitecustomize.py file in the Python installation (at /usr/lib/python3.12/sitecustomize.py) that was set up during a previous session to tune NCCL for an 8× RTX PRO 6000 Blackwell configuration on PCIe Gen5. The file contained code like:

import os as _os
_os.environ.setdefault("NCCL_P2P_LEVEL", "SYS")

Python's sitecustomize.py is a standard hook: it is imported automatically at interpreter startup, before any user code runs. The setdefault method on os.environ is a dictionary operation that sets a key only if it is not already present. This is the critical detail: setdefault checks for the existence of the key, not whether its value is meaningful. If the environment variable is set to an empty string, setdefault considers the key present and does nothing. If the variable is unset entirely, setdefault writes the default value.

The assistant's attempted override in message [msg 6165] — setting NCCL_P2P_LEVEL="" in the shell — should have worked in theory. An empty string is still a value; the key exists in the environment, so setdefault should have left it alone. But the verification in message [msg 6166] returned SYS, proving that the override had not taken effect. Why?

The most likely explanation is that the shell-level env var assignments in the SSH command never reached the Python process. The command in message [msg 6165] was a complex multi-step pipeline: kill, fuser, sleep, then nohup ... &. The env var prefixes applied only to the nohup command, but the preceding fuser -k /dev/nvidia* killed all processes using NVIDIA devices — including the one that nohup was about to start. By the time the assistant checked, the process was already dead and the log was empty. The env var assignment was never actually tested.

The Verification That Changed Everything

Message [msg 6166] is the moment the assistant steps back from the cycle of launching and killing processes and asks a more fundamental question: "What is the actual state of the environment when Python starts?" Instead of trying to override the variables and hoping for the best, the assistant runs a minimal probe — a one-liner that prints the value of NCCL_P2P_LEVEL as seen by a freshly launched Python interpreter. The answer is SYS.

This verification reveals two things simultaneously. First, it confirms that sitecustomize.py is active and setting NCCL tuning parameters. Second, it exposes the flaw in the assistant's previous override attempt: either the env vars were never passed to Python (because the process was killed), or the mechanism of passing empty strings through SSH's single-quoted command context was not working as expected. Either way, the assistant now knows that any future attempt to change NCCL behavior must either (a) modify or remove the sitecustomize.py file, (b) set env vars before Python starts in a way that survives the SSH command pipeline, or (c) pass the NCCL parameters as command-line arguments to SGLang if supported.

Assumptions and Mistakes

This message exposes several assumptions that were quietly guiding the debugging effort. The assistant assumed that shell-level env var assignments would propagate cleanly through SSH's single-quoted command execution into the Python process. This is normally true for simple invocations, but the complexity of the command — with backgrounding, redirection, and preceding cleanup commands — introduced failure modes that were not immediately obvious.

The assistant also assumed that setting an env var to an empty string was equivalent to unsetting it. In the context of os.environ.setdefault, these are semantically different: an empty string is a value, and setdefault respects it; a missing key triggers the default. The assistant's mental model was "clear the NCCL tuning so the defaults apply," but the implementation required actual removal of the key, not just emptying its value.

A third assumption was that the NCCL tuning parameters from the 8-GPU configuration were the cause of the 4-GPU hang. This turned out to be correct — NCCL_P2P_LEVEL=SYS forces NCCL to use system memory for peer-to-peer communication, which can cause issues on certain PCIe topologies — but the assistant had not yet proven this causality. The verification in message [msg 6166] was a necessary step toward that proof.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of several interconnected systems. Python's sitecustomize.py mechanism is a relatively obscure feature of the CPython startup sequence, documented in the Python standard library but rarely used in everyday development. The os.environ.setdefault method is a standard dictionary operation, but its interaction with shell environment variables — particularly the distinction between "unset" and "set to empty" — is a subtle point that often surprises developers.

On the ML infrastructure side, understanding NCCL's environment variables — NCCL_P2P_LEVEL, NCCL_PROTO, NCCL_ALGO, and others — requires familiarity with NVIDIA's Collective Communications Library and its tuning parameters. NCCL_P2P_LEVEL=SYS tells NCCL to use system memory (RAM) for GPU-to-GPU transfers when direct peer-to-peer (P2P) access is not available. This is a fallback mode that can work but introduces overhead and can cause deadlocks in certain configurations, especially when combined with IOMMU translation or non-coherent interconnects.

The reader also needs to understand the SSH command execution model: how environment variables are passed through SSH, how shell variable assignments work as command prefixes, and how nohup and backgrounding interact with process groups and signal propagation.

Output Knowledge Created

Message [msg 6166] creates actionable knowledge about the system's state. The assistant now knows with certainty that sitecustomize.py is enforcing NCCL tuning parameters, and that any attempt to change NCCL behavior must account for this hook. The verification output — the single word "SYS" — is a data point that will guide all subsequent debugging steps.

This message also creates meta-knowledge about the debugging process itself. The assistant learns that shell-level env var overrides are not reliable when a sitecustomize.py file is present, especially when the override command is embedded in a complex multi-step SSH invocation. This insight shifts the debugging strategy from "try to override from outside" to "modify the configuration at its source."

The Thinking Process

The reasoning visible in this message is a textbook example of a debugging pivot. The assistant had been operating under the assumption that the NCCL env vars in sitecustomize.py were merely defaults that could be overridden by setting them in the shell. The word "Wait" at the beginning of the message signals a moment of cognitive dissonance — the assistant is reconsidering this assumption in real time.

The structure of the message reveals the assistant's mental model: first, the realization ("Wait, the NCCL env vars are set in sitecustomize.py which runs before any env override"), then the verification step ("Let me verify"), followed by the actual probe and its result. This is the scientific method applied to systems debugging: form a hypothesis about the system's state, design an experiment to test it, execute the experiment, and interpret the result.

The choice of verification command is itself instructive. The assistant could have checked the file contents again, or examined the environment from within a running SGLang process. Instead, the assistant chose the most direct probe possible: start a minimal Python interpreter and print the variable. This eliminates all the confounding factors of the full SGLang launch — the multiprocessing, the NCCL initialization, the model loading — and isolates the single question of what value Python sees for NCCL_P2P_LEVEL at startup.

Broader Implications

This message illustrates a class of bugs that are increasingly common in modern ML infrastructure: configuration that is set in multiple places with different precedence rules. The sitecustomize.py file was created with good intentions — to document a known-working NCCL configuration for a specific hardware topology — but it became a hidden constraint that outlived its original context. When the hardware configuration changed (from 8 GPUs to 4), the tuning parameters persisted silently, blocking the new deployment.

The lesson for ML engineers is twofold. First, sitecustomize.py is a powerful but dangerous tool: it runs before any user code, including framework initialization, and its effects are invisible unless you know to look for them. Second, environment variable debugging requires verifying the actual state at the point of execution, not assuming that shell-level assignments propagate correctly through complex command pipelines.

Conclusion

Message [msg 6166] is a small but pivotal moment in a larger debugging narrative. It is the point where the assistant stops trying to fight the system from the outside and starts examining the system's actual behavior from the inside. The verification that NCCL_P2P_LEVEL=SYS — a single three-letter string — unlocks the next phase of debugging, where the assistant will modify sitecustomize.py, adjust NCCL parameters for the 4-GPU topology, and eventually get the server running at over 2,800 tokens per second. But none of that success would have been possible without this moment of stepping back, questioning assumptions, and verifying the ground truth of the environment.