The Silent Hang: Diagnosing NCCL Deadlock in Multi-Node SGLang Deployment

Introduction

In the intricate dance of deploying large language models across multiple machines, few moments are as tense as the wait for distributed initialization to complete. The subject message (index 6631) captures one such moment in a high-stakes deployment: bringing up the Qwen3.5-122B-A10B-FP8 model—a 119-billion-parameter reasoning model—across two NVIDIA DGX Spark systems connected via InfiniBand RoCE. The assistant, having successfully navigated model downloads, memory management, container orchestration, and network configuration, now faces a silent adversary: a hanging NCCL initialization that refuses to complete.

This message is a diagnostic pivot point. It is not a triumphant "it works" nor a frustrated "it failed." It is the quiet, focused moment of a practitioner who sees a symptom and begins the methodical work of tracing its cause. The message contains exactly one tool call—a bash command to check port accessibility—and the reasoning that motivated it. But within this seemingly simple exchange lies a wealth of insight about distributed systems debugging, the assumptions we make about network communication, and the art of reading log output for clues.

The Context: A Multi-Node Deployment in Progress

To understand this message, one must appreciate the journey that led to it. The assistant had been tasked with deploying Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes—compact NVIDIA systems built around the GB10 Grace Blackwell Superchip, each with 120GB of unified memory and an ARM Cortex-X925 CPU. These are not traditional servers; they are purpose-built AI appliances with unified CPU-GPU memory, which means GPU memory is system memory, and competition for that pool is fierce.

The deployment had already overcome significant hurdles. The assistant had freed GPU memory on the second Spark by stopping an existing GLM-4.7-Flash container and a persistent reranker process. It had downloaded the 119GB FP8 model from HuggingFace and rsynced it across nodes at ~640MB/s over the InfiniBand link. It had pivoted from SGLang (whose official Spark image lacked Qwen3.5 support and whose multi-node NCCL initialization hung) to a dedicated vLLM image (hellohal2064/vllm-qwen3.5-gb10). It had configured environment variables like GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME to force distributed communication over the correct RoCE network interface rather than defaulting to loopback or the wrong subnet.

By message 6631, the assistant had launched both nodes and waited through multiple polling cycles. The head node log showed:

[2026-04-08 22:50:17 TP0] CustomAllreduce is disabled because this process group spans across nodes

This was a good sign—it meant both nodes had found each other and formed a process group. But then... silence. No weight loading. No KV cache allocation. No "ready" message. The log simply stopped advancing. The assistant had waited 90 seconds, then 120 seconds, then 180 seconds, checking repeatedly. The processes were alive, using only ~416MiB of GPU memory each—far below what model loading would require. Something was stuck.

The Diagnostic Reasoning

The subject message opens with the assistant articulating its observation:

I see the issue — it says Init torch distributed begin but never Init torch distributed ends. The NCCL initialization might be hanging trying to set up over the network. It could be waiting for NCCL to establish the communication channel.

This is a textbook example of log-driven diagnosis. The assistant has identified a specific log pattern: a "begin" marker without a corresponding "ends" marker. In distributed PyTorch, init_process_group is the function that establishes communication between ranks. It proceeds through several phases:

  1. Rendezvous: Using a TCP store (or other backend) to discover peers and exchange connection information
  2. Backend initialization: Setting up the actual communication backend—in this case, NCCL (NVIDIA Collective Communications Library)
  3. Process group creation: Finalizing the group membership The fact that "CustomAllreduce is disabled because this process group spans across nodes" had already been logged told the assistant that phase 1 (rendezvous) and the initial process group creation had succeeded. The hang was therefore in phase 2—NCCL backend initialization. The assistant's hypothesis is precise: "It could be waiting for NCCL to establish the communication channel." NCCL uses a bootstrap mechanism where each rank needs to exchange network information (IP addresses, ports) to set up direct GPU-to-GPU communication channels. If one rank cannot reach another at the advertised address, NCCL will block indefinitely waiting for the connection to be established.

The Tool Call: Checking Port Accessibility

To test this hypothesis, the assistant runs a network diagnostic:

ssh aurora@10.1.230.180 'ss -tlnp | grep -E "20000|20001|30000" && echo "---head_ports" && ssh aurora@192.168.200.13 "ss -tlnp | grep -E 20000\|20001\|30000" && echo "---worker_ports"'

This command does three things:

  1. On the head node (10.1.230.180), it lists all TCP sockets listening on ports 20000, 20001, or 30000
  2. It prints a separator
  3. It SSHs into the worker node (192.168.200.13) and runs the same port check
  4. It prints another separator The choice of ports is significant: - Port 20000: The --dist-init-addr port, used for the TCP rendezvous store where nodes exchange bootstrap information - Port 20001: The default NCCL communication port, where NCCL expects to establish GPU-to-GPU data channels - Port 30000: The SGLang/vLLM HTTP serving port The result is telling:
LISTEN 0      4096                *:20000            *:*          
---head_ports

Port 20000 is listening on the head node—the rendezvous store is active. But the output ends abruptly after "---head_ports" with no worker port information and no "---worker_ports" separator. This silence is itself a diagnostic signal.

Interpreting the Partial Result

The missing worker output is ambiguous and the assistant must consider multiple possibilities:

  1. The SSH connection to the worker failed: The jump host (10.1.230.180) might not be able to reach the worker (192.168.200.13) at this moment, or the SSH session timed out.
  2. The worker's ss command produced no output: If none of the specified ports are listening on the worker, grep would return nothing, but the echo "---worker_ports" should still execute. The fact that even the separator is missing suggests the SSH command itself may have failed or hung.
  3. The command is still running: The assistant might have received partial output before the connection timed out. The most likely interpretation, given the context of a hanging NCCL initialization, is that the SSH jump to the worker encountered a problem. This could be related to the same underlying network issue causing NCCL to hang—perhaps the InfiniBand interface is not fully routable for TCP connections, or there's a firewall rule blocking inter-node communication on certain ports.

Assumptions Embedded in the Diagnosis

The assistant makes several assumptions in this message, most of which are reasonable but worth examining:

Assumption 1: NCCL is the culprit. The assistant assumes the hang is in NCCL initialization rather than in model weight loading, disk I/O, or memory allocation. This is a well-informed assumption: the log shows "CustomAllreduce is disabled" (a NCCL-related message) and then nothing else. If weight loading were happening, GPU memory usage would increase, but it stayed at ~416MiB. If disk I/O were the bottleneck, we'd see CPU or I/O activity. The static memory usage strongly suggests a communication hang.

Assumption 2: The hang is network-related, not software-bug-related. The assistant assumes NCCL is waiting for a network condition (port unreachable, address mismatch) rather than hitting a deadlock bug in NCCL itself. This is a reasonable first hypothesis—network misconfiguration is far more common in multi-node deployments than NCCL deadlocks—but it's not guaranteed.

Assumption 3: Port accessibility is the right thing to check. The assistant checks whether ports are listening, not whether they're reachable from the other node. A port can be listening on the local machine but blocked by a firewall or not bound to the correct interface for remote access. The ss command with -l (listening) shows only local socket state, not reachability.

Assumption 4: The IP addresses and port numbers in the diagnostic command are correct. The assistant uses 10.1.230.180 for the head node's external interface and 192.168.200.13 for the worker's InfiniBand subnet address. If NCCL was configured to use different addresses (e.g., the head node's IB subnet address 192.168.200.12), the port check on the external interface might not be relevant.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Understanding of distributed PyTorch initialization: Knowledge that init_process_group proceeds through rendezvous, backend setup, and group creation phases, and that NCCL is the GPU communication backend.
  2. Familiarity with NCCL's bootstrap mechanism: NCCL uses a TCP-based bootstrap (via the dist-init-addr parameter) for initial peer discovery, then establishes direct GPU-to-GPU channels. The bootstrap port (20000) and NCCL communication ports (20001+) serve different purposes.
  3. Knowledge of the deployment architecture: Two DGX Spark nodes, each with a single GB10 SoC (120GB unified memory), connected via InfiniBand RoCE. The head node has an external IP (10.1.230.180) and an IB subnet IP (192.168.200.12), while the worker is at 192.168.200.13.
  4. Understanding of SGLang/vLLM server architecture: The model is split across nodes using tensor parallelism (TP=2), meaning each node holds half of each layer. NCCL handles the all-reduce operations needed to synchronize activations during inference.
  5. Familiarity with the ss command: ss -tlnp lists TCP sockets that are listening (-l), showing numeric ports (-n), and the associated process (-p). The -t flag filters to TCP sockets.

Output Knowledge Created

This message produces several pieces of knowledge:

  1. Port 20000 is active on the head node: The TCP rendezvous store is listening and available for connections. This confirms the first phase of distributed initialization completed successfully.
  2. No port information from the worker: The missing worker output is itself a signal. It suggests either a network connectivity issue between the head and worker (separate from the NCCL issue), or that the worker's NCCL-related ports haven't been opened yet.
  3. The hang is specifically in NCCL's post-bootstrap phase: Since the process group was created (evidenced by "CustomAllreduce is disabled") but NCCL communication channels aren't established (port 20001 not listening), the hang is in NCCL's internal channel setup.
  4. The diagnostic approach is validated: The assistant's reasoning that NCCL initialization was the bottleneck is supported by the port check. The rendezvous store is working; the NCCL communication channels are not.

What Happens Next

The assistant's next message (msg 6632) builds on this diagnosis. It notes that port 20001 (the NCCL communication port) is not yet listening, and that both nodes logged "CustomAllreduce is disabled," confirming they passed the initial process group creation. The assistant then decides to enable NCCL_DEBUG=INFO to get more detailed logging from NCCL, which is the natural next step in the diagnostic process.

This progression—from observing a symptom, forming a hypothesis, running a targeted check, interpreting partial results, and planning the next diagnostic step—is the essence of distributed systems debugging. The assistant does not jump to conclusions or make drastic changes. It gathers data methodically.

Broader Significance

This message, while brief, illustrates several important principles for AI infrastructure engineering:

Log analysis is a superpower. The assistant's ability to spot the missing "ends" marker in a sea of log output is what makes this diagnosis possible. In distributed systems, where processes can hang silently for minutes or hours, knowing exactly which log lines to look for is invaluable.

Distributed initialization is fragile. Multi-node tensor parallelism requires multiple network-dependent handshakes: SSH for container orchestration, TCP for the rendezvous store, NCCL for GPU communication, and potentially HTTP for health checks. Each of these can fail independently, and their failure modes interact in complex ways.

The right diagnostic question matters. Rather than restarting the deployment or changing configuration parameters, the assistant asks: "Is the port accessible?" This narrow, targeted question is far more efficient than random experimentation.

Partial information is still information. The missing worker output in the port check is ambiguous, but it's not useless. It narrows the space of possible causes and guides the next diagnostic step.

Conclusion

Message 6631 captures a moment of focused diagnostic work in a complex multi-node AI deployment. The assistant observes a hanging NCCL initialization, hypothesizes a network-level cause, and runs a targeted port accessibility check. The result—port 20000 active on the head, no output from the worker—provides partial confirmation of the hypothesis and guides the next step (enabling NCCL debug logging).

This message is a testament to the value of systematic debugging in distributed systems. It shows that even when a deployment stalls, the path forward is not through random configuration changes but through careful observation, precise hypothesis formation, and targeted data collection. The assistant's approach here—grounded in deep knowledge of the distributed PyTorch initialization sequence and NCCL's communication architecture—is a model for how to diagnose and resolve infrastructure issues in production AI systems.