The Persistent NCCL Hang: Debugging Distributed Init on Blackwell GPUs

Message Overview

In message [msg 6171] of a complex multi-session deployment effort, the assistant issues a single bash command to check the status of an SGLang server that was just launched to serve the Qwen3.5-122B-A10B model across four NVIDIA RTX PRO 6000 Blackwell GPUs. The command and its output are:

[bash] ssh root@10.1.230.174 'sleep 45 && tail -5 /tmp/sglang_122b_v3.log && echo "---GPU---" && nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'
[2026-03-09 12:32:57 TP2] Mamba selective_state_update backend initialized: triton
[2026-03-09 12:32:57 TP2] Init torch distributed begin.
[2026-03-09 12:32:57 TP1] Mamba selective_state_update backend initialized: triton
[2026-03-09 12:32:57 TP1] Init torch distributed begin.
[2026-03-09 12:32:57 TP0] sglang is using nccl==2.29.3
---GPU---
0, 1125 MiB
1, 1133 MiB
2, 1133 MiB
3, 1045 MiB

At first glance, this looks like a routine status check. But the output tells a story of a deeply stuck system. All four GPUs show only ~1 GB of memory used — the baseline allocation before any model weights are loaded. The log tail shows the server processes have reached Init torch distributed begin but never progressed beyond it. And critically, a subtle asymmetry appears: TP0 emits sglang is using nccl==2.29.3 while TP1 and TP2 do not, suggesting the workers are not all executing the same initialization path. This message captures a moment of diagnostic stall — the assistant is watching a process that has already been tried multiple times, with multiple theories tested, and is still failing in the same way.

Why This Message Was Written: The Debugging Context

To understand why this particular bash command was issued, one must appreciate the debugging arc that precedes it. The assistant had been trying to deploy Qwen3.5-122B-A10B (a 122-billion-parameter MoE model with 10 billion active parameters, in BF16 precision requiring approximately 234 GB of memory) across four GPUs using SGLang with tensor parallelism (TP=4). This was a pivot from earlier work deploying much larger models like Qwen3.5-397B-A17B-NVFP4.

The immediate context begins with a driver version mismatch. The Proxmox host had been upgraded to NVIDIA driver 590.48.01 by another agent setting up a confidential computing VM, but the LXC container running SGLang still had userspace libraries from version 565.57.01. The assistant diagnosed this mismatch in [msg 6140]–[msg 6141] by comparing kernel module version (cat /proc/driver/nvidia/version) against installed packages (dpkg -l | grep nvidia). It then fixed it in [msg 6146] by installing the matching 590 packages via apt-get install libnvidia-compute-590 libnvidia-decode-590 nvidia-utils-590.

But the fix didn't work. The server continued to hang at Init torch distributed begin — the point where PyTorch's init_process_group calls into NCCL to establish communication between the four tensor-parallel workers. This led to a series of increasingly sophisticated diagnostic attempts:

  1. Checking NCCL environment variables ([msg 6155]): The assistant examined /usr/lib/python3.12/sitecustomize.py which set NCCL tuning parameters optimized for 8 GPUs (NCCL_P2P_LEVEL=SYS, NCCL_PROTO=LL, etc.). These settings might be inappropriate for the new 4-GPU topology.
  2. Running NCCL_DEBUG=INFO ([msg 6157]): A manual launch with NCCL debug logging showed that NCCL init actually completed successfully — the bootstrap timings were printed and the comm was initialized. This was a critical finding: NCCL itself wasn't hanging, something after NCCL init was.
  3. Testing TP=1 ([msg 6163]–[msg 6164]): The assistant tried loading the model on a single GPU, which confirmed the model code loaded correctly (it OOM'd on weight allocation, as expected for a 234 GB model on a 96 GB GPU). This ruled out model compatibility issues.
  4. Clearing NCCL overrides ([msg 6165]–[msg 6167]): The assistant tried launching with all NCCL environment variables cleared, but discovered that sitecustomize.py uses setdefault, meaning environment variables set before Python starts can override them. Multiple attempts to kill processes and relaunch were complicated by fuser -k /dev/nvidia* accidentally killing the new processes too. Message [msg 6171] is the result of the third attempt (v3) to launch with cleared NCCL overrides. The assistant waited 45 seconds (a deliberately chosen interval — long enough for NCCL init to complete if it was going to, short enough to avoid excessive delay) and checked both the log tail and GPU memory usage. The output confirmed the hang was still happening.

The Thinking Process Visible in the Reasoning

This message is a diagnostic probe, and the reasoning behind it is revealed through the sequence of actions leading up to it. The assistant's thinking, reconstructed from the context, runs approximately as follows:

"The NCCL debug output showed that NCCL init completes successfully. But the server still hangs at 'Init torch distributed begin' — which is the torch distributed wrapper around NCCL. The workers print their initialization messages but never reach weight loading. The GPU memory stays at baseline. Something is blocking the torch distributed rendezvous."

The choice to wait 45 seconds is itself a reasoned decision. NCCL initialization typically takes 5–15 seconds for 4 GPUs. Waiting 45 seconds ensures we're past any transient startup delay. The assistant also checks GPU memory because if weights were being loaded, memory usage would climb from ~1 GB to tens of GB per GPU. The flat memory confirms the model hasn't started loading.

The output reveals a new clue: TP0 shows sglang is using nccl==2.29.3 while TP1 and TP2 show only the Mamba backend initialization and "Init torch distributed begin" lines. This asymmetry is significant. In a properly functioning TP=4 setup, all four workers should follow the same initialization path. The fact that TP0 reaches a different code point suggests either:

Assumptions Made in This Message

The assistant operates under several implicit assumptions:

1. The hang is in NCCL/torch distributed. This assumption is reasonable given that "Init torch distributed begin" is the last log line printed, and the server never reaches weight loading. However, the NCCL_DEBUG=INFO output from [msg 6157] showed NCCL init completing successfully, which should have raised doubt. The assistant appears to treat the NCCL_DEBUG run as a separate attempt that might have succeeded due to different timing or env vars, rather than fully ruling out NCCL as the culprit.

2. A 45-second wait is sufficient. This assumes that if NCCL init were going to succeed, it would do so within 45 seconds. For 4 GPUs on PCIe, this is generally true, but if there's a retry mechanism with a long timeout, it might not be.

3. The GPU memory baseline (~1 GB) indicates no weight loading has occurred. This is correct — the Qwen3.5-122B model's weights would consume ~58 GB per GPU (234 GB / 4), so 1 GB clearly indicates no weights have been loaded.

4. The v3 launch parameters are correct. The assistant launched with NCCL_PROTO=Simple NCCL_ALGO=Ring NCCL_P2P_LEVEL=PHB — these were chosen to override the sitecustomize settings. But the assistant didn't verify that these overrides were actually taking effect (it checked in [msg 6166] that sitecustomize uses setdefault, which respects pre-existing env vars, but didn't confirm the specific values reached the Python process).

5. The server configuration (model path, flags) is correct. The assistant assumes the model at /shared/models/Qwen3.5-122B-A10B is valid and the flags --reasoning-parser qwen3 --tool-call-parser qwen3_coder are appropriate. These had been tested with TP=1 (which got past model loading to OOM), so this assumption is well-founded.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption is that the hang is a software configuration issue (NCCL env vars, driver mismatch) rather than a hardware or virtualization issue. The actual root cause — P2P DMA corruption under SEV-SNP IOMMU translation — is a hardware/IOMMU problem that manifests as NCCL hanging. The assistant is looking in the right place (NCCL init) but for the wrong reasons.

A subtle mistake is in the diagnostic methodology. The NCCL_DEBUG=INFO run in [msg 6157] showed NCCL init completing successfully, but the assistant didn't fully trust that result, perhaps because it was run with a 60-second timeout that killed the process before it could proceed further. The output showed:

llm-two:5948:5948 [0] NCCL INFO [Rank 0] ncclCommInitRank comm 0x47e49e30 rank 0 nranks 4 cudaDev 0 nvmlDev 0 busId 1000 commId 0xb23ca3b0f9a12be8 - Init START
...
llm-two:5948:5948 [0] NCCL INFO Bootstrap timings total 0.004433 (create 0.000043, s...

This shows NCCL comm init starting but the output was truncated by tail -40 and the 60-second timeout. The assistant may have misinterpreted this as NCCL completing, when in fact NCCL might have been hung during comm init (the "Init START" line is printed before the actual all-reduce handshake).

Another mistake is the assumption that the NCCL env var overrides are taking effect. The assistant set NCCL_PROTO=Simple NCCL_ALGO=Ring NCCL_P2P_LEVEL=PHB but sitecustomize.py uses os.environ.setdefault, which only sets the variable if it's not already set. Since the assistant exported these variables in the shell before launching Python, they should take effect. But the assistant didn't verify by checking NCCL_DEBUG output in the v3 run.

Input Knowledge Required

To fully understand this message, one needs:

1. The hardware topology: 4× RTX PRO 6000 Blackwell GPUs (96 GB each, SM120 architecture) connected via PCIe Gen5, hosted in an LXC container on a Proxmox host. The host has SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) enabled, which enables full IOMMU translation.

2. The model characteristics: Qwen3.5-122B-A10B is a Mixture-of-Experts model with 122 billion total parameters and 10 billion active parameters. In BF16 precision, it requires ~234 GB, fitting on 4× 96 GB GPUs with TP=4.

3. SGLang architecture: SGLang uses tensor parallelism (TP), where the model is sharded across GPUs and each GPU runs a worker process. These workers communicate via NCCL for collective operations. The Init torch distributed begin log line is printed by each worker when it enters torch.distributed.init_process_group.

4. NCCL initialization flow: NCCL comm init involves: (a) bootstrap connection between ranks, (b) CUDA IPC handle exchange for P2P, (c) all-reduce handshake to verify connectivity. A hang at "Init torch distributed begin" could be in any of these phases.

5. The driver versioning issue: NVIDIA driver consists of a kernel module (KMD) and userspace libraries (libcuda, libnvidia-ml, etc.). The kernel module version must match the userspace libraries. The container had kernel module 590.48.01 (from the host) but userspace 565.57.01, which was fixed in the previous round.

6. The SEV-SNP context: AMD SEV-SNP is a confidential computing feature that encrypts VM memory and enforces integrity. It requires full IOMMU translation (amd_iommu=on), which intercepts all DMA transfers including GPU P2P. This is the eventual root cause — the IOMMU corrupts P2P DMA data, causing NCCL all-reduce handshakes to fail.

Output Knowledge Created

This message produces several pieces of knowledge:

1. Confirmation of the persistent hang: The v3 attempt with cleared NCCL overrides also hangs. This rules out the specific NCCL tuning parameters (NCCL_PROTO, NCCL_ALGO, NCCL_P2P_LEVEL) as the cause.

2. The NCCL version clue: TP0 prints sglang is using nccl==2.29.3 while TP1 and TP2 do not. This asymmetry is a diagnostic signal — it suggests the workers are not all executing the same initialization path. In a healthy TP=4 setup, all four workers would print this line. The absence on TP1 and TP2 suggests they are blocked before reaching that code point, or there's a race condition where only the first worker to initialize prints the NCCL version.

3. Stable GPU memory baseline: All four GPUs show ~1 GB memory used, unchanged from the idle state. This confirms no model weights have been loaded, narrowing the hang to the distributed initialization phase (before weight loading begins).

4. Timing information: The log timestamps show all three workers printed their initialization messages within the same second (12:32:57), suggesting they started concurrently but then all blocked simultaneously. This is consistent with a collective operation (like NCCL all-reduce) that requires all ranks to participate — if one rank fails to respond, all ranks block.

5. Elimination of driver mismatch as the sole cause: Since the driver was fixed in [msg 6146] and the hang persists, the root cause must be elsewhere. This redirects the diagnostic effort toward NCCL configuration, torch distributed internals, or hardware-level issues.

The Broader Significance

Message [msg 6171] represents a turning point in the debugging process. It's the moment when the assistant has exhausted the obvious software fixes (driver version, NCCL env vars, port conflicts) and must now consider deeper issues. The NCCL version asymmetry on TP0 is a subtle but important clue that points toward a fundamental communication problem rather than a configuration issue.

In the subsequent chunk (not shown in this message but described in the segment summary), the assistant would discover the true root cause: the SEV-SNP IOMMU configuration on the Proxmox host intercepts GPU P2P DMA transfers and corrupts the data, causing NCCL's all-reduce handshake to fail silently. The fix — NCCL_P2P_DISABLE=1 — forces NCCL to use SHM (shared memory) transport instead of P2P DMA, avoiding the IOMMU corruption entirely.

This message is a classic example of debugging at the boundary between software and hardware. The assistant correctly identifies that the hang is in distributed initialization, correctly tests multiple software hypotheses, and correctly narrows the problem. But the root cause lies in a layer the assistant cannot directly observe — the IOMMU translation layer in the Proxmox hypervisor — requiring a different diagnostic approach (checking dmesg for IO_PAGE_FAULTs, running CUDA P2P bandwidth tests) to finally uncover.

The message also illustrates the importance of patience in debugging. The assistant doesn't give up after the first failed attempt, or the second, or the third. Each attempt eliminates a hypothesis and produces new clues. The NCCL version asymmetry, the stable memory baseline, and the concurrent blocking of all workers are all pieces of a puzzle that eventually leads to the IOMMU root cause. This systematic elimination of possibilities is the essence of disciplined debugging.