The Pivot Point: Resolving Cross-Node Networking for Multi-Node vLLM on DGX Spark

Introduction

In the complex dance of deploying a 119-billion-parameter language model across two NVIDIA DGX Spark systems, there comes a moment when months of accumulated infrastructure knowledge crystallizes into a single, decisive action. Message <msg id=6712> in this opencode session is that moment. It is the culmination of a multi-hour debugging session spanning networking configuration, container orchestration, and distributed computing internals — all focused on one goal: making two ARM-based Blackwell GPUs talk to each other over InfiniBand to serve the Qwen3.5-122B-A10B-FP8 model.

This message is not flashy. It contains no dramatic discovery, no new algorithm, no breakthrough in model architecture. It is, on its surface, a single bash command that copies a shell script to two remote machines and executes it. But beneath that surface lies a rich story of diagnostic reasoning, iterative hypothesis testing, and the quiet satisfaction of a distributed system finally falling into alignment.

The Message in Full

The assistant executes:

scp /home/theuser/glm-kimi-sm120-rtx6000bw/spark-vllm-qwen35.sh \
    aurora@10.1.230.180:/home/aurora/spark-vllm-qwen35.sh && \
ssh aurora@10.1.230.180 'chmod +x /home/aurora/spark-vllm-qwen35.sh && \
    scp /home/aurora/spark-vllm-qwen35.sh aurora@192.168.200.13:/home/aurora/spark-vllm-qwen35.sh && \
    /home/aurora/spark-vllm-qwen35.sh head && \
    ssh aurora@192.168.200.13 "/home/aurora/spark-vllm-qwen35.sh worker" && \
    sleep 10 && \
    docker exec vllm-qwen35 python3 -c "import ray; ray.init(address=\"auto\"); \
        nodes=ray.nodes(); \
        [print(n[\"NodeManagerAddress\"], dict((k,v) for k,v in n[\"Resources\"].items() \
            if \"node\" in k or \"GPU\" in k)) for n in nodes]"' 2>&1

The output confirms success:

Starting Ray HEAD on 192.168.200.12...
93563c5ad3f98b5dc92c4a5c0a1c038883a785f3035549fb30364930810be137
Ray head started. Container: vllm-qwen35
Starting Ray WORKER on 192.168.200.13, connecting to 192.168.200.12:6379...
4e8fdfbedbbd057e0fdb4a866e5493da601388a7729cc3ddbec5e8ae48d10260
Ray worker started. Container: vllm-qwen35
2026-04-08 23:39:00,558  INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 192.168.200.12:6379...
2026-04-08 23:39:00,566  INFO worker.py:2007 -- Connected...

Why This Message Was Written: The Networking Crisis

To understand why this particular message matters, we must trace the crisis that preceded it. The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model across two DGX Spark nodes for several hours. Each node is an ARM-based NVIDIA GB10 system with 120GB of unified memory and a single Blackwell GPU, connected via a high-speed InfiniBand RoCE (RDMA over Converged Ethernet) interconnect. The model, at 119GB in FP8 precision, barely fits across the two nodes with tensor parallelism.

The initial approach used SGLang, but its official spark image lacked Qwen3.5 support and its multi-node NCCL initialization hung indefinitely. The assistant pivoted to a community-maintained vLLM image (hellohal2064/vllm-qwen3.5-gb10) that specifically targets Qwen3.5 on GB10 hardware. This image uses vLLM 0.17.1rc1 and includes Ray 2.53 for distributed orchestration.

The first attempt at multi-node deployment ([msg 6689] through [msg 6707]) revealed a fundamental networking problem. The DGX Spark nodes have two network interfaces: an external network (10.1.230.x) and an InfiniBand interconnect (192.168.200.x). When Ray started on the head node, it automatically detected the external IP (10.1.230.180) as its node address. The worker node (192.168.200.13) could only reach the head through the IB subnet — it had no route to 10.1.230.180. This caused the PyTorch distributed backend (c10d) on the worker to fail with:

The client socket has failed to connect to any network address of (10.1.230.180, 55053).

This is a classic distributed systems failure mode: an auto-detected address that is correct from the local node's perspective but unreachable from peers. The assistant correctly diagnosed this in <msg id=6708>:

"The core issue: the worker node (192.168.200.13) can't reach the head node at 10.1.230.180:55053. The second spark only has connectivity to the first via the 192.168.200.x network, not via 10.1.230.x."

How Decisions Were Made: The Iterative Fix

The assistant's decision-making process in the messages leading up to <msg id=6712> reveals a methodical, hypothesis-driven approach to debugging distributed systems.

Decision 1: Force Ray to use the IB subnet. The first fix ([msg 6709]) was to add --node-ip-address to the Ray start commands in the launch script, forcing both nodes to use their 192.168.200.x addresses. This ensures Ray's internal registry maps each node to the correct, mutually reachable IP.

Decision 2: Set environment variables for PyTorch distributed. In <msg id=6710>, the assistant recognized that even if Ray uses the right IPs, the PyTorch distributed backend (c10d) and NCCL need their own interface configuration. The fix added VLLM_HOST_IP, TP_SOCKET_IFNAME, and MASTER_ADDR to force torch distributed to use the IB interface.

Decision 3: Per-node VLLM_HOST_IP. In <msg id=6711>, the assistant realized that VLLM_HOST_IP must differ per node — each node needs to advertise its own IB address. This required restructuring the script to accept a node-specific configuration.

Decision 4: Verify before serving. The command in <msg id=6712> includes a verification step using Ray's Python API to inspect the cluster state. This is a deliberate choice to validate the networking fix before proceeding to the actual model serving, which would take 15+ minutes to load. The assistant is being conservative — verify the foundation before building on it.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

Assumption 1: The IB subnet (192.168.200.x) is fully routable between nodes. The assistant assumes that once Ray uses these addresses, all subsequent communication (Ray internal RPC, PyTorch c10d, NCCL) will work. This is a reasonable assumption given the hardware setup (direct InfiniBand link), but it's not tested until this message.

Assumption 2: Ray's --node-ip-address flag overrides all auto-detection. The assistant assumes that setting this flag is sufficient to control Ray's address binding. In practice, Ray's networking stack is complex — there are separate addresses for the Raylet, the GCS (Global Control Store), and object store. The flag controls the primary address but other components may still bind to different interfaces.

Assumption 3: Docker networking is transparent to Ray. The containers run with --network host (as seen in the launch script), so the assistant assumes that Ray inside the container sees the host's network interfaces directly. This is correct for host networking mode.

Assumption 4: The verification script is sufficient to confirm cluster health. The assistant uses a simple Python script that queries Ray for node addresses and GPU resources. This checks that the cluster formed and GPUs are visible, but it doesn't test the PyTorch distributed backend or NCCL connectivity — those are only tested when vLLM actually starts serving.

Mistakes and Incorrect Assumptions

Mistake 1: Underestimating the address propagation problem. In the initial attempt ([msg 6704]), the assistant set VLLM_HOST_IP=192.168.200.12 on the head node, but Ray still registered the node as 10.1.230.180. The assistant assumed that VLLM_HOST_IP would influence Ray's node registration, but Ray uses its own address detection independent of vLLM environment variables. The fix required explicitly passing --node-ip-address to the ray start command.

Mistake 2: Assuming a single VLLM_HOST_IP would work for both nodes. The initial script set a single VLLM_HOST_IP environment variable, but each node needs to advertise its own address. The assistant had to restructure the script in <msg id=6711> to support per-node configuration.

Mistake 3: Not verifying the cluster state before launching the server. In earlier attempts ([msg 6692]), the assistant launched the vLLM server immediately after starting Ray, only to discover the networking failure after a 60-second timeout. The verification step in <msg id=6712> represents a learned lesson — check the foundation before building on it.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of Ray's networking model. Ray uses a distributed scheduler with a Global Control Store (GCS) that maintains cluster state. Each node registers with its IP address, which becomes the key for resource placement. If nodes register with unreachable addresses, the cluster appears healthy but distributed operations fail.

Knowledge of vLLM's distributed architecture. vLLM uses tensor parallelism (TP) across GPUs, which requires a PyTorch distributed process group. This group uses c10d (the PyTorch distributed backend) for collective communication, which in turn relies on NCCL or Gloo for actual data transfer. Each of these layers has its own address resolution mechanism.

Knowledge of DGX Spark networking topology. The DGX Spark has a dual-network setup: an external management network and a dedicated InfiniBand interconnect for GPU-to-GPU communication. Understanding which network to use for which purpose is critical.

Knowledge of Docker container networking. The containers use host networking mode, meaning they share the host's network namespace. This simplifies networking but means container processes must be configured to use the correct interfaces.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

A working Ray cluster across two DGX Spark nodes. The immediate output is a verified two-node Ray cluster with both GPUs visible. This is the foundation for the subsequent vLLM serving deployment.

A validated networking configuration pattern. The combination of --node-ip-address for Ray, per-node VLLM_HOST_IP, and interface-specific environment variables (GLOO_SOCKET_IFNAME, NCCL_SOCKET_IFNAME) forms a reusable pattern for multi-node vLLM deployments on dual-network systems.

A launch script that encapsulates the fix. The spark-vllm-qwen35.sh script, now updated with the networking fixes, becomes a reusable artifact for future deployments.

Confirmation of the diagnostic hypothesis. The assistant's theory — that the root cause was Ray using the wrong IP address — is confirmed correct when the cluster forms successfully with the IB subnet addresses.

The Thinking Process Visible in Reasoning

The assistant's reasoning is most visible in the structure of the command itself. The command chains six operations in a specific order:

  1. Copy script to head node — ensures both nodes get the same updated script
  2. Set permissions and copy to worker — the head node acts as a relay to the worker, since the assistant's local machine may not have direct access to the worker
  3. Start Ray head — must happen first to establish the GCS endpoint
  4. Start Ray worker — connects to the head's GCS at 192.168.200.12:6379
  5. Wait 10 seconds — allows time for cluster formation and heartbeats
  6. Verify cluster state — uses Ray's Python API to inspect nodes and resources The verification step is particularly telling. The assistant could have simply checked ray status from the command line, but instead chose to run a Python script that filters for node addresses and GPU resources. This reveals a specific concern: the assistant wants to confirm that (a) each node registered with the correct IB subnet address, and (b) each node's GPU is visible to Ray. These are the two failure modes from previous attempts. The Python command itself is carefully constructed:
import ray
ray.init(address="auto")
nodes = ray.nodes()
[print(n["NodeManagerAddress"], dict((k,v) for k,v in n["Resources"].items()
    if "node" in k or "GPU" in k)) for n in nodes]

This prints only the relevant fields — NodeManagerAddress (to verify the IP is correct) and resource entries containing "node" or "GPU" (to verify GPU visibility and node labels). The filtering is a deliberate choice to avoid information overload from Ray's verbose resource listing.

Conclusion

Message <msg id=6712> is a quiet triumph in a long debugging session. It doesn't announce itself with fanfare — it's just a bash command followed by confirmation output. But for anyone who has struggled with distributed system networking, it tells a compelling story: the moment when a carefully constructed hypothesis is validated, when the pieces finally click into place, and when the path forward becomes clear.

The message embodies the essence of systems engineering: methodical diagnosis, iterative refinement, and the discipline to verify before proceeding. It's a reminder that in distributed systems, the most critical work often happens not in the code, but in the configuration — in understanding which address to use, which interface to bind to, and which environment variable controls which layer of the stack.

From this point, the assistant can proceed to launch the vLLM server, load the 119GB model, and eventually achieve ~27 tok/s throughput. But that success rests on the foundation laid in this single message — the moment when two Blackwell GPUs, separated by a network topology problem, finally learned to talk to each other.