The Diagnostic Pivot: Isolating the TP=4 Hang in SGLang's Distributed Initialization

Introduction

In the course of deploying the Qwen3.5-122B-A10B BF16 model across four NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant encountered a stubborn and recurring failure: the SGLang server would hang indefinitely at the "Init torch distributed begin" log message, never progressing to weight loading or serving. After multiple rounds of debugging—upgrading NVIDIA userspace libraries, tweaking NCCL environment variables, checking for port conflicts, and verifying NCCL initialization with debug logging—the assistant reached a critical decision point captured in message index 6162. This message represents a classic diagnostic pivot: abandoning the increasingly complex chase of the distributed initialization hang in favor of stripping the problem down to its simplest possible configuration to verify fundamental assumptions.

The Message

The assistant wrote:

No port is being listened on — the rendezvous never got set up. The workers might be deadlocking on each other. Let me kill everything and try with --tp 1 first to see if the model even loads:

>

``bash ssh root@10.1.230.174 'kill -9 $(pgrep -f sglang) 2>/dev/null; fuser -k /dev/nvidia* 2>/dev/null; sleep 2; echo "cleaned"' ``

The Reasoning and Motivation

The message is the culmination of a long debugging session spanning dozens of interactions. To understand why the assistant wrote this, we must trace the chain of reasoning that led to this point.

The assistant had been attempting to start an SGLang server with tensor parallelism of 4 (--tp 4), distributing the 234 GB Qwen3.5-122B-A10B model across four 96 GB Blackwell GPUs. The server consistently hung at the same point: after all four TP workers printed "Init torch distributed begin," no further progress was made. GPU memory remained flat at ~1 GB per card, indicating no weights had been loaded.

The assistant had already exhausted several hypotheses:

  1. Driver version mismatch: The container had NVIDIA userspace libraries at version 565.57.01 while the Proxmox host kernel module was at 590.48.01. The assistant fixed this by installing matching 590 packages, but the hang persisted.
  2. NCCL P2P issues: The assistant suspected GPU-to-GPU P2P DMA corruption under the SEV-SNP IOMMU configuration (a known issue from earlier in the session where NCCL_P2P_DISABLE=1 had been required). However, NCCL debug logging showed successful initialization with P2P/IPC and Ring+LL protocol.
  3. Port conflicts or stale rendezvous files: The assistant checked for listening ports on the distributed communication range and found no conflicts, and verified that no stale torch or NCCL temporary files existed.
  4. NCCL environment variable interference: The sitecustomize.py file in the Python environment set aggressive NCCL tuning variables optimized for the earlier 8-GPU configuration. The assistant tried launching with all NCCL overrides cleared, but the hang remained.
  5. Torch distributed init hanging: The assistant used strace to inspect the blocked TP worker processes and found them all waiting on futex and restart_syscall (socket reads), suggesting a deadlock or barrier that never completes. The key insight in this message is the assistant's observation: "No port is being listened on — the rendezvous never got set up." This refers to the TCP rendezvous port used by PyTorch's distributed process group initialization. The assistant had checked for listening ports on the expected distributed communication endpoints and found none active, confirming that the init_process_group call was indeed the point of failure.

The Diagnostic Strategy

The assistant's decision to test with --tp 1 is a textbook application of the scientific method to systems debugging. By reducing the problem to its simplest form—a single GPU with no distributed communication—the assistant can answer a fundamental question: does the model load at all?

This is important because all prior debugging had been operating under an untested assumption: that the model itself was loadable on this system. The hang at distributed init could have been masking a deeper issue—perhaps the model format was incompatible, the tokenizer was broken, or some other initialization path was failing before distributed setup even completed. By testing TP=1, the assistant isolates the model loading path from the distributed initialization path.

The assistant's reasoning follows a clear logical chain:

  1. If TP=1 works (even if it OOMs on the 234 GB model), then the model is fundamentally loadable and the problem is specifically in the distributed initialization.
  2. If TP=1 also hangs or crashes with a different error, the root cause is elsewhere—perhaps in model compatibility, PyTorch version, or SGLang's model support.
  3. Either outcome dramatically narrows the search space. This is a classic "divide and conquer" debugging strategy. Rather than continuing to probe the distributed initialization with increasingly fine-grained tools (which had already consumed many messages), the assistant steps back to verify the foundational layer.

Assumptions Made

The message reveals several implicit assumptions:

The model should start loading on a single GPU: The assistant assumes that the Qwen3.5-122B-A10B model will at least begin its loading process on one GPU, even though it will eventually fail with an out-of-memory error (the model is 234 GB and each GPU has only 96 GB). This is a reasonable assumption—PyTorch and SGLang typically begin weight allocation before hitting OOM, and the error would manifest as a clean CUDA out of memory exception rather than a hang.

The hang is in torch distributed, not model loading: The assistant assumes that the "Init torch distributed begin" message marks the boundary between distributed setup and model loading. If the hang is actually in model loading code that happens to log after that point, the TP=1 test would reveal it.

The environment is clean after killing: The assistant runs kill -9 and fuser -k /dev/nvidia* to ensure no residual GPU contexts or zombie processes interfere with the new test. This assumes that GPU state is properly reset after forceful termination, which is generally true for NVIDIA drivers but can occasionally leave GPU contexts in a wedged state.

No persistent state corruption: The assistant assumes that the repeated hangs have not left the GPUs or CUDA driver in a corrupted state that would affect a TP=1 test. This is a reasonable assumption given that fuser -k /dev/nvidia* forces all GPU file handles to be released.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

Tensor parallelism (TP): The concept of sharding a large model across multiple GPUs, where each GPU holds a slice of each layer and communicates with peers during forward/backward passes. TP=4 means four GPUs work together as a single logical device.

PyTorch distributed initialization: The init_process_group call that sets up communication between distributed workers using NCCL or Gloo backends. The TCP rendezvous mechanism uses a master worker that listens on a port for connections from other workers.

NCCL (NVIDIA Collective Communications Library): The low-level library for GPU-to-GPU communication that PyTorch uses under the hood. NCCL handles all-reduce, broadcast, and other collective operations.

SGLang's server architecture: SGLang launches multiple worker processes (one per TP rank) that communicate via NCCL. The "Init torch distributed begin" log message marks the start of the distributed setup in the model runner.

The hardware context: Four NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each) connected via PCIe Gen5, running on a Proxmox host with SEV-SNP IOMMU enabled, inside an LXC container.

The model context: Qwen3.5-122B-A10B is a 122-billion parameter Mixture-of-Experts model with ~10 billion active parameters per token, stored in BF16 precision (234 GB total). It was recently released and requires up-to-date software support.

Output Knowledge Created

This message creates several important pieces of knowledge:

A clear diagnostic hypothesis: The assistant articulates that the workers "might be deadlocking on each other," which becomes the working theory to investigate. This hypothesis is testable: if TP=1 works but TP=4 hangs, the deadlock theory is confirmed.

A binary test design: The TP=1 experiment is designed to produce a clear yes/no answer about model loadability. The assistant will either see the model start loading (proving the model is compatible) or see a different failure mode (revealing a deeper issue).

A clean state for the experiment: By forcefully killing all SGLang processes and releasing GPU file handles, the assistant ensures the next test starts from a known clean state, eliminating the possibility that residual state from previous hangs is contaminating the results.

Documentation of the failure mode: The message implicitly documents that the TP=4 hang is not caused by driver mismatch, NCCL configuration, or port conflicts—those hypotheses have been tested and eliminated. The remaining suspects are either a torch distributed deadlock specific to this hardware/software combination or a model compatibility issue.

The Thinking Process

The assistant's thinking is visible in the structure of the message. It begins with an observation ("No port is being listened on"), draws a conclusion from that observation ("the rendezvous never got set up"), proposes a mechanism ("the workers might be deadlocking on each other"), and then designs an experiment to test the hypothesis ("try with --tp 1 first to see if the model even loads").

The phrase "the rendezvous never got set up" is particularly telling. In PyTorch's distributed initialization, the rank 0 worker creates a TCPStore (a key-value store over TCP) that other workers connect to during init_process_group. If no port is listening, it means the rank 0 worker never created the store, which implies it got stuck before or during that step. This is a more precise diagnosis than the earlier "Init torch distributed begin" observation—it pinpoints the failure to the store creation, not just the general distributed setup.

The assistant's decision to test TP=1 rather than, say, debugging the TCPStore creation directly, reflects a pragmatic trade-off. Debugging the TCPStore would require reading PyTorch source code, adding more logging, or using strace/gdb on the hanging processes—all time-consuming and uncertain. The TP=1 test takes 30 seconds and provides immediate, high-value information regardless of outcome.

The Broader Context

This message sits at a turning point in the session. The assistant had been spinning wheels on the TP=4 hang for many messages, trying increasingly fine-grained debugging techniques. The TP=1 test breaks this cycle by providing a clear path forward. If the model loads on TP=1 (even with an OOM error), the assistant can then focus specifically on the distributed initialization path—perhaps by testing TP=2, TP=3, or by examining the torch distributed code more carefully. If the model fails on TP=1, the assistant knows to look at model compatibility, PyTorch version, or SGLang's model support code.

The message also demonstrates a key characteristic of effective debugging: knowing when to stop digging deeper into one hypothesis and instead test a more fundamental assumption. The assistant had accumulated evidence against several hypotheses (driver mismatch, NCCL config, port conflicts) but had never verified the most basic assumption: that the model loads at all. This message corrects that oversight.

Conclusion

Message 6162 is a masterclass in diagnostic reasoning under uncertainty. Faced with a persistent hang that had resisted multiple rounds of debugging, the assistant steps back, identifies the untested foundational assumption, and designs a clean experiment to test it. The decision to try TP=1 is not just a configuration change—it is a deliberate restructuring of the problem space that will either confirm the distributed deadlock theory or reveal a completely different root cause. This pivot, captured in a single message, transforms the debugging trajectory and exemplifies the kind of strategic thinking that separates effective troubleshooting from aimless exploration.