The Per-Node Insight: How a Single Environment Variable Fixed Multi-Node vLLM Across Two DGX Sparks

Subject Message: [assistant] Now I need per-node VLLM_HOST_IP. Let me restructure: [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/spark-vllm-qwen35.sh Edit applied successfully.

At first glance, message <msg id=6711> appears almost trivial: a two-line utterance followed by an edit command that modifies a shell script. The assistant simply says "Now I need per-node VLLM_HOST_IP. Let me restructure" and applies an edit. There is no fanfare, no explicit debugging narrative, no detailed explanation of why this change matters. Yet this message represents the critical turning point in a multi-hour debugging session spanning two DGX Spark nodes, multiple failed deployment strategies, and a deep entanglement of networking assumptions. It is the moment when the assistant correctly identifies that the VLLM_HOST_IP environment variable — a seemingly simple configuration parameter — must be set differently on each node rather than as a single global value, and that this per-node distinction is the key to making PyTorch's distributed communication (via its c10d library) work across an InfiniBand interconnect.

To understand why this message was written, one must trace the arc of the deployment effort that preceded it. The assistant had been tasked with deploying the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter FP8 quantized reasoning model — across two NVIDIA DGX Spark systems (each with a single NVIDIA GB10 Blackwell GPU, 120GB unified memory, and an InfiniBand RoCE interconnect). This is a multi-node tensor parallelism (TP=2) deployment: the model is too large for a single GPU, so it must be split across both machines, with each GPU holding half the weights and communicating intermediate activations across the InfiniBand link during every forward pass.

The Networking Maze

The assistant had already navigated a series of failed approaches before arriving at message <msg id=6711>. The journey began with an attempt to graft model files from a vLLM 0.17 image onto a vLLM 0.14 base, which failed because the model architectures were incompatible across major versions ([msg 6686]). The assistant then pivoted to using the hellohal2064/vllm-qwen3.5-gb10 Docker image — a community build specifically designed for Qwen3.5 on GB10 — and set up a Ray cluster across both nodes ([msg 6689], [msg 6690]). Ray formed successfully with two active GPUs, but the vLLM serve command crashed.

The crash revealed a subtle networking problem. The worker node (192.168.200.13) could not reach the head node at 10.1.230.180:55053 — the port used by PyTorch's c10d distributed communication backend ([msg 6708]). The issue was that Ray had auto-detected the head node's external IP (10.1.230.180) rather than its InfiniBand subnet IP (192.168.200.12). The two DGX Sparks are connected via a dedicated 192.168.200.x InfiniBand network; the 10.1.230.x external network is only reachable from the first Spark, not the second. So when PyTorch's distributed library tried to connect to 10.1.230.180:55053 from the worker, the connection simply hung.

The assistant's initial fix was to force Ray to use the 192.168.200.x subnet by passing --node-ip-address to the Ray start commands ([msg 6709]). This ensured Ray registered both nodes with their correct IB subnet IPs. But this alone was insufficient because vLLM and PyTorch also need to know which IP to bind to for their own distributed communication channels. The assistant then attempted to set VLLM_HOST_IP globally to 192.168.200.12 ([msg 6700]), but this was wrong because it made both nodes try to use the same IP.

The Per-Node Realization

This is where message <msg id=6711> enters. The assistant had just finished adding VLLM_HOST_IP, TP_SOCKET_IFNAME, and MASTER_ADDR to the script ([msg 6710]), but then realized a fundamental flaw: these environment variables cannot be set to a single value for both nodes. The head node (192.168.200.12) and the worker node (192.168.200.13) each need their own VLLM_HOST_IP. The MASTER_ADDR should point to the head node, but VLLM_HOST_IP on each node must be that node's own IP address so that PyTorch's c10d library knows which interface to bind to and which address to advertise to peers.

The reasoning process visible in the message is compressed but clear. The assistant says "Now I need per-node VLLM_HOST_IP" — the word "per-node" is the critical insight. Earlier attempts had treated VLLM_HOST_IP as a global configuration parameter, something you set once and forget. But distributed systems are not that simple: each process in a tensor-parallel group needs to know its own address so it can participate in the collective communication (all-reduce, etc.) that synchronizes activations and gradients across GPUs. The VLLM_HOST_IP variable tells vLLM which IP address to use when setting up the PyTorch distributed process group. If both nodes use the same IP, the group cannot form — each rank must have a unique address.

The assistant's decision to "restructure" the script reflects an understanding that the launch script's architecture needs to change. Instead of a single flat set of environment variables applied to both nodes, the script must now have conditional logic: the head node section sets VLLM_HOST_IP=192.168.200.12, the worker node section sets VLLM_HOST_IP=192.168.200.13, and both use MASTER_ADDR=192.168.200.12 (the head node's address) as the coordination point for the distributed group.

Assumptions Made and Corrected

This message implicitly corrects several assumptions that had been operating in earlier steps. The first assumption was that Ray's node registration and PyTorch's distributed communication would naturally use the same network interface. In reality, Ray and PyTorch's c10d library have independent networking stacks: Ray was successfully configured to use the IB subnet via --node-ip-address, but PyTorch's distributed backend was still trying to use the external IP because VLLM_HOST_IP wasn't set correctly.

The second assumption was that a single VLLM_HOST_IP value could serve both nodes. This assumption likely stemmed from single-node deployments where there is only one host, so the variable is naturally unambiguous. In multi-node deployments, each rank in the tensor-parallel group must have a distinct address, and the environment must be configured per-process.

The third assumption — visible in the earlier attempt at <msg id=6700> — was that using the external IP (10.1.230.180) would work because it's the head node's "real" address. But this ignored the network topology: the worker node has no route to the external network. The assistant had to learn that the correct address is not the one that is "most public" but the one that is mutually reachable by all participating nodes. In this case, that is the 192.168.200.x InfiniBand subnet.

Input and Output Knowledge

The input knowledge required to understand this message includes: the topology of the two DGX Spark nodes (each with one GPU, connected via InfiniBand on 192.168.200.x); the role of VLLM_HOST_IP in vLLM's distributed initialization (it sets the IP that PyTorch's c10d library binds to); the distinction between Ray's node registration and PyTorch's process group formation; and the history of failed attempts documented in the preceding messages.

The output knowledge created by this message is the corrected launch script with per-node VLLM_HOST_IP values. The immediate consequence is visible in the very next message ([msg 6712]), where the assistant deploys the updated script, starts the Ray head and worker, and confirms that both nodes are now registered with their 192.168.200.x IPs. In the following messages ([msg 6713], [msg 6714], [msg 6715]), the assistant launches the serve command and observes the model loading successfully — shards being streamed across the InfiniBand link, NCCL connected, distributed initialization working correctly for the first time.

The Broader Significance

What makes this message noteworthy is not its verbosity — it is remarkably terse — but its placement in the debugging narrative. It is the moment of synthesis where multiple threads of investigation converge: the network topology constraint (worker can't reach external IP), the Ray configuration fix (--node-ip-address), the distributed communication requirement (each rank needs its own address), and the practical implementation (restructuring the launch script). The assistant does not belabor the point; it simply states the conclusion and acts on it.

This pattern is characteristic of expert debugging in complex distributed systems. The surface symptom (vLLM crash with "client socket failed to connect") could have many root causes. The assistant systematically eliminated possibilities: model incompatibility, Ray cluster formation, GPU visibility, placement group constraints, and network reachability. Each elimination narrowed the search space until the remaining cause became clear: the VLLM_HOST_IP variable needed per-node values. The message <msg id=6711> is where that narrowing reaches its conclusion.

The edit itself — restructuring a shell script to use different environment variables on different nodes — is a mundane operation. But the reasoning behind it is not. It reflects an understanding that in distributed inference, the network configuration is not a property of the cluster but of each individual process's relationship to the cluster. The head node and worker node have different roles, different IPs, and different environment requirements. Treating them identically was the bug; recognizing their asymmetry was the fix.

In the end, this single message, barely two lines long, marks the transition from failure to success in a complex multi-node deployment. It is a testament to the fact that in distributed systems engineering, the most impactful insights often arrive not as lengthy analyses but as concise realizations — "Now I need per-node VLLM_HOST_IP" — followed by immediate action.