Debugging Multi-Node Distributed Inference: When --dist-init-addr Isn't Enough

In the course of deploying a large language model across two NVIDIA DGX Spark systems, a single message from the assistant captures a moment of critical diagnostic insight. The message, [msg 6621], is a turning point in a multi-hour debugging session — the moment when a mysterious timeout is correctly attributed to a subtle networking misconfiguration in PyTorch's distributed initialization. This article examines that message in depth: the reasoning behind it, the assumptions it exposes, the knowledge it draws upon, and the insight it produces.

The Context: Deploying Qwen3.5-122B Across Two DGX Sparks

The broader session involved deploying the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter mixture-of-experts model in FP8 precision — across two NVIDIA DGX Spark systems. Each Spark is a compact desktop supercomputer built around the NVIDIA GB10 Blackwell GPU with 120GB of unified memory and an ARM Cortex-X925 CPU. The two units were connected via InfiniBand RoCE (RDMA over Converged Ethernet) on a 192.168.200.x subnet, with a separate management network at 10.1.230.x.

The assistant had already accomplished significant infrastructure work: downloading the 119GB model from HuggingFace, transferring it to the second Spark at ~640MB/s over the InfiniBand link, freeing GPU memory by stopping competing services (an embeddings server and a reranker), and crafting a launch script that used SGLang's native multi-node support via --nnodes, --node-rank, --tp-size, and --dist-init-addr. The architecture was straightforward: two nodes, tensor parallelism (TP) of 2, with the head node at 192.168.200.12 and the worker at 192.168.200.13.

The Message: A Diagnosis of Distributed Timeout

The subject message reads in full:

The worker timed out connecting — it's trying to connect to 127.0.0.1:17795 which means the Gloo transport is using localhost. The --dist-init-addr needs to be the reachable IP. The issue is the worker tried to create a process group but couldn't reach the head. The head is still waiting at "Init torch distributed begin".

>

The problem might be that on the worker, --dist-init-addr 192.168.200.12:20000 is correct but Gloo's TCP transport is trying 127.0.0.1. Let me check if NCCL/Gloo needs explicit interface configuration, and also if the network between nodes is properly set up for the TCP rendezvous:

>

`` [bash] ssh aurora@10.1.230.180 'docker rm -f sglang-qwen35-head 2>/dev/null; ssh aurora@192.168.200.13 "docker rm -f sglang-qwen35-worker 2>/dev/null"' sglang-qwen35-head ``

This message is the culmination of a chain of observations. In the previous message ([msg 6620]), the assistant had checked the logs after waiting 60 seconds. The head node showed repeated "Persistent cache disabled, using in-memory JIT cache" messages, indicating it was stuck before model loading. The worker's log was not shown in that message, but the assistant had seen enough to know the connection had failed.

The Reasoning Process: From Symptom to Root Cause

The assistant's reasoning follows a clear diagnostic chain:

  1. Symptom: The worker timed out connecting. The head is stuck at "Init torch distributed begin" — a log message from PyTorch's distributed initialization indicating it's waiting for all ranks to join the process group.
  2. Evidence: The worker is trying to connect to 127.0.0.1:17795. This is the critical clue. Port 17795 is the default Gloo TCP transport port in PyTorch distributed. The fact that it's using 127.0.0.1 (localhost) rather than the specified --dist-init-addr 192.168.200.12:20000 reveals a fundamental misunderstanding in the launch configuration.
  3. Hypothesis: The --dist-init-addr parameter, which the assistant correctly set to the head node's InfiniBand subnet IP, is not sufficient to control which interface Gloo binds to or connects from. Gloo's TCP transport is independently discovering its network interface and defaulting to localhost, likely because the container's network configuration doesn't expose the InfiniBand interface as the default route.
  4. Action: The assistant formulates two lines of investigation: checking if NCCL/Gloo needs explicit interface environment variables (like GLOO_SOCKET_IFNAME or NCCL_SOCKET_IFNAME), and verifying that the network between nodes is properly set up for TCP rendezvous.

Assumptions and Their Consequences

This message reveals several assumptions the assistant was operating under:

Assumption 1: --dist-init-addr controls the full distributed communication path. The assistant had earlier ([msg 6594]) correctly understood that SGLang's multi-node mode requires matching --dist-init-addr on both nodes. The assumption was that this single parameter would configure both the rendezvous endpoint and the transport-level binding. In reality, --dist-init-addr in SGLang sets the master address for the rendezvous (the TCP store used for rank coordination), but the underlying Gloo transport independently selects its listening interface — and it chose localhost.

Assumption 2: The Docker container would inherit the host's network interfaces correctly. The containers were launched with --network host, which should give them full access to the host's network stack. However, the InfiniBand interface (ib0 or similar) may not have been the default route, and Gloo's interface selection logic picked lo (loopback) instead.

Assumption 3: The IB subnet was properly routable for TCP traffic. The assistant had verified earlier that the InfiniBand link could carry rsync traffic at ~640MB/s, but that used a different protocol. TCP over the same interface might have required additional configuration (IP assignment, routing tables) that wasn't fully in place.

Input Knowledge Required

To understand this message, one needs knowledge of several interconnected domains:

Output Knowledge Created

This message produces several valuable insights:

  1. The root cause is identified: The distributed connection failure is not a firewall issue, a port blockage, or a Docker networking problem — it's a Gloo transport interface selection problem. The worker is trying to reach the head via localhost instead of the InfiniBand IP.
  2. A clear next step is defined: The assistant needs to either set GLOO_SOCKET_IFNAME to the InfiniBand interface name, or configure NCCL_SOCKET_IFNAME if NCCL is the backend. Alternatively, the container's routing table might need adjustment.
  3. The diagnostic pattern is established: The combination of "Init torch distributed begin" on the head and a timeout on the worker, with the worker's connection targeting 127.0.0.1:17795, is a signature of this class of networking misconfiguration.
  4. The cleanup is executed: The assistant immediately cleans up the failed containers (docker rm -f sglang-qwen35-head), preparing for the next attempt with corrected configuration.

The Broader Significance

This message exemplifies a common pattern in distributed ML deployments: the gap between high-level framework parameters and low-level transport behavior. The --dist-init-addr flag is a user-friendly abstraction, but it doesn't fully control the distributed initialization pipeline. The Gloo transport layer operates independently, making its own interface selection decisions based on the operating system's routing table and network interface enumeration.

The assistant's reasoning demonstrates a crucial skill: the ability to map observed behavior (timeout at "Init torch distributed begin") to a specific layer in the distributed communication stack, and to formulate a hypothesis that accounts for the evidence (the 127.0.0.1:17795 connection attempt). This is not guesswork — it's systematic debugging informed by knowledge of how PyTorch's distributed components interact.

The message also reveals the assistant's operational discipline. Even while diagnosing a complex networking issue, it cleans up the failed containers immediately, preventing resource leaks and ensuring a clean state for the next attempt. This attention to operational hygiene is essential in environments with limited GPU memory, where orphaned containers can block subsequent launches.

Conclusion

Message [msg 6621] is a compact but rich example of distributed systems debugging in the context of large language model deployment. It captures the moment when a mysterious timeout is correctly attributed to a subtle interface selection issue in PyTorch's Gloo transport layer. The assistant's reasoning — moving from symptom to evidence to hypothesis to action — follows a textbook diagnostic pattern, while also revealing the assumptions and knowledge that underpin effective troubleshooting in this domain. For anyone deploying multi-node inference systems, this message offers a valuable lesson: high-level distributed configuration parameters are not always sufficient, and understanding the transport layer's behavior is essential for reliable multi-node initialization.