The Orchestration Moment: Multi-Node Ray Cluster Formation Across Two DGX Spark Nodes

Introduction

In the sprawling narrative of deploying a large language model across distributed hardware, there are moments of quiet triumph that rarely make the headlines. Message <msg id=6704> in this opencode session is one such moment — a single, dense bash command that represents the successful culmination of a long debugging chain. The assistant executes a multi-step orchestration: copying an updated shell script to two DGX Spark nodes, launching a Ray cluster across them, and verifying the cluster formation. On the surface, it is a routine infrastructure operation. Beneath the surface, it embodies hours of failed attempts, networking diagnostics, and architectural pivots that preceded this clean execution.

This article examines that message in depth: why it was written, the reasoning encoded in its structure, the assumptions it carries, and the knowledge it both consumes and produces. It is a case study in how distributed AI serving infrastructure is debugged, rebuilt, and finally made to work.

The Message: A Single Shot at Multi-Node Deployment

The subject message is a bash command issued by the assistant, executed on the Proxmox host (the machine orchestrating the DGX Spark nodes). Here is the exact command and its output:

[assistant] [bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/spark-vllm-qwen35.sh aurora@10.1.230.180:/home/aurora/spark-vllm-qwen35.sh && ssh aurora@10.1.230.180 'chmod +x /home/aurora/spark-vllm-qwen35.sh && scp /home/aurora/spark-vllm-qwen35.sh aurora@192.168.200.13:/home/aurora/spark-vllm-qwen35.sh && /home/aurora/spark-vllm-qwen35.sh head && ssh aurora@192.168.200.13 "/home/aurora/spark-vllm-qwen35.sh worker" && sleep 10 && docker exec vllm-qwen35 python3 -c "import ray; ray.init(address=\"auto\"); nodes=ray.nodes(); [print(n[\"NodeManagerAddress\"], list(n[\"Resources\"].keys())) for n in nodes]"' 2>&1
Starting Ray HEAD on 192.168.200.12...
653c6568f7aebfd4896e5a4d2e951b9f3bfb80ec467b37f97d089eec421d11a4
Ray head started. Container: vllm-qwen35
Starting Ray WORKER, connecting to 192.168.200.12:6379...
e1b66d43df8fb8cd6104873b8302831ce9d012c41a6aaf536a244df77c7ddd4f
Ray worker started. Container: vllm-qwen35
2026-04-08 23:35:52,025	INFO worker.py:1821 -- Connecting to existing Ray cluster at address: 10.1.230.180:6379...
2026-04-08 23:35:52,033	INFO worker.py:2007 -- Connected to Ray cluster.
...

The command chains six operations together using && operators, meaning each must succeed before the next proceeds. This is not a cautious, exploratory command — it is a confident re-launch after debugging, encoding the full corrected deployment sequence in a single atomic action.

Why This Message Was Written: The Context and Motivation

To understand why this message exists, one must trace the tortured path that led to it. The assistant was attempting to deploy Qwen3.5-122B-A10B-FP8, a 119-billion-parameter model, across two NVIDIA DGX Spark nodes (each with a single GB10 Blackwell GPU and 120GB unified memory). The model is too large for a single GPU, so tensor parallelism across two nodes is required — a demanding configuration that requires low-latency inter-node communication and a distributed execution framework.

The assistant had already tried and failed at several approaches:

Grafting model files between vLLM versions (messages <msg id=6679> through <msg id=6686>): The assistant attempted to copy Qwen3.5 model files from a working vLLM 0.17 image onto a vLLM 0.14 image that had multi-node support. This failed because the model code depended on base classes and APIs that changed between major versions — a classic version incompatibility trap.
Using the existing cluster launch script (messages <msg id=6684>–<msg id=6686>): The hellohal2064/vllm-qwen3.5-gb10 image (vLLM 0.17, built for Qwen3.5 on GB10) worked for single-node but its entrypoint didn't match the existing launch-cluster.sh script, causing container startup failures.
Writing a custom launch script (messages <msg id=6687>–<msg id=6703>): The assistant wrote spark-vllm-qwen35.sh to handle Ray head, worker, and serve lifecycle. Initial launches failed because VLLM_HOST_IP was set to the InfiniBand subnet IP (192.168.200.x), but Ray auto-detected the external IP (10.1.230.180) for the head node, causing a placement group mismatch where vLLM couldn't find the GPU resources it needed. The message at <msg id=6704> is the corrected re-launch after fixing the VLLM_HOST_IP issue. The assistant had edited the script in messages <msg id=6702> and <msg id=6703> to remove the per-node IP forcing, then stopped the old containers in <msg id=6701>. This message is the "apply and verify" step — deploying the fix and confirming it works.

How Decisions Were Made: The Structure of the Command

The command's structure reveals the assistant's decision-making process. Each &&-chained step reflects a deliberate ordering:

Step 1: scp the script to the head node. The script was edited on the Proxmox host (the assistant's workspace). It must be copied to the DGX Spark before execution. This is the deployment step.

Step 2: chmod +x and scp to the worker. The assistant makes the script executable on the head node, then copies it to the second Spark node over the InfiniBand link (192.168.200.13). This ensures both nodes have identical infrastructure code.

Step 3: Run ./spark-vllm-qwen35.sh head. This starts a Docker container on the head node with Ray head process inside. The container is named vllm-qwen35 and mounts the model weights from the host filesystem.

Step 4: Run ./spark-vllm-qwen35.sh worker on the worker node. Via SSH to 192.168.200.13, the assistant launches the worker container, which connects to the Ray head at 192.168.200.12:6379.

Step 5: Sleep 10 seconds. A brief wait for Ray to fully initialize and register both nodes.

Step 6: Verify cluster formation. The assistant runs a Python script inside the head container that connects to Ray, lists nodes, and prints each node's address and available resources. This is the validation step — without it, the assistant would be blindly proceeding to launch vLLM serve.

The output confirms success: two Ray containers started, and the Python verification connected to the cluster at 10.1.230.180:6379 (the external IP, confirming the earlier fix was correct). The output is truncated with ..., but the absence of error messages strongly suggests the verification passed.

Assumptions Made

This message encodes several assumptions, most of which were validated by the preceding debugging:

The script is correct. The assistant assumes that the edits made in <msg id=6702> and <msg id=6703> (removing VLLM_HOST_IP forcing) are sufficient to fix the placement group mismatch. This assumption is validated by the successful cluster formation.
The Docker images are present on both nodes. The hellohal2064/vllm-qwen3.5-gb10 image must already exist on both Spark nodes. The assistant had previously transferred it via docker save/docker load over SSH in <msg id=6683>.
SSH connectivity works. The assistant assumes SSH to both 10.1.230.180 (external) and 192.168.200.13 (IB subnet) is functional. This was verified in earlier messages.
The model weights are mounted. The script mounts /home/aurora/models/Qwen3.5-122B-A10B-FP8 into the container. The assistant assumes this path exists and contains the downloaded model files (previously rsynced at ~640MB/s over the IB link).
Ray will auto-detect the correct IP. The key fix: instead of forcing VLLM_HOST_IP to the IB subnet, the assistant now lets Ray use whatever IP it detects. The assumption is that Ray's auto-detection (which picked 10.1.230.180 for the head) will be consistent and that vLLM's placement group logic will match.
The InfiniBand interface is correctly configured. The script sets GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME to the RoCE interface, and NCCL_IB_GID_INDEX for InfiniBand. The assistant assumes these environment variables will enable correct inter-node GPU communication.

Mistakes and Incorrect Assumptions

While this message itself is successful, it is built on the ashes of earlier incorrect assumptions:

The most significant mistake was assuming VLLM_HOST_IP should match the InfiniBand subnet. In messages <msg id=6700>–<msg id=6701>, the assistant set VLLM_HOST_IP=192.168.200.12 on the head node, expecting Ray to register the node under that IP. But Ray auto-detected the external IP (10.1.230.180) instead. This caused vLLM's placement group to request node:192.168.200.12 which didn't exist in Ray's registry, leading to the error "No available node types can fulfill resource request." The fix was to remove the forced IP and let Ray use its auto-detected address.

A subtler incorrect assumption was that grafting model files between vLLM versions would work. The assistant assumed that model code is self-contained and portable across vLLM 0.14 and 0.17. In reality, the model classes depend on base classes, tensor parallel utilities, and configuration registries that differ between versions. This is a common trap in AI infrastructure — model code is deeply coupled to the framework version.

Another assumption that nearly failed: that the hellohal2064/vllm-qwen3.5-gb10 image had Ray. The assistant checked this in <msg id=6686> and confirmed Ray 2.53 was present. If it hadn't been, the entire multi-node approach would have required a different distributed executor backend.

Input Knowledge Required

To understand this message, one needs:

Network topology knowledge: The DGX Spark nodes have two network interfaces — an external IP (10.1.230.180) for management and an InfiniBand/RoCE subnet (192.168.200.x) for high-speed inter-node communication. The assistant uses the IB subnet for SSH between nodes and for NCCL communication.
Ray cluster architecture: Ray uses a head node (with a GCS server at port 6379) and worker nodes that connect to it. Placement groups allow scheduling tasks across specific nodes. The ray.init(address="auto") call connects to an existing cluster.
vLLM distributed execution: vLLM uses --distributed-executor-backend ray for multi-node tensor parallelism. It creates placement groups that pin GPU workers to specific nodes. The VLLM_HOST_IP environment variable tells vLLM which IP to use for inter-node communication.
Docker container management: The script uses docker run with GPU mounts, volume mounts for model weights, and network configuration. The assistant manages container lifecycle (stop, start, exec).
The Qwen3.5 model specifics: The model is a 122B-parameter MoE architecture with FP8 quantization, requiring ~119GB of storage. It uses a custom reasoning parser and tool-call parser.
The preceding debugging chain: The message is meaningless without knowing about the failed VLLM_HOST_IP attempt, the model grafting failure, and the script edits. The assistant's reasoning is encoded in the sequence of fixes that led to this clean execution.

Output Knowledge Created

This message produces several concrete outcomes:

A verified two-node Ray cluster with two active GPUs (one per DGX Spark). The cluster is ready to accept distributed workloads.
A validated deployment script (spark-vllm-qwen35.sh) that now correctly handles multi-node Ray startup. The script encapsulates the full lifecycle: head start, worker start, serve launch, and stop.
A confirmed networking configuration where Ray auto-detects the external IP (10.1.230.180) for the head node, and the InfiniBand subnet (192.168.200.x) is used for inter-node SSH and NCCL communication. This dual-network setup is now proven to work.
A reproducible deployment sequence that can be re-run after node reboots or failures. The assistant later demonstrates this by successfully launching the vLLM serve process and serving the model at ~27 tok/s single-request throughput.
Documentation of the VLLM_HOST_IP pitfall — the lesson that forcing a specific IP can conflict with Ray's auto-detection, causing placement group failures. This is a non-obvious debugging insight for anyone deploying multi-node vLLM with Ray.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, while not explicitly stated in this message, is visible in the surrounding context and in the structure of the command itself:

The assistant recognized that the previous VLLM_HOST_IP approach was fundamentally wrong. In messages <msg id=6699>–<msg id=6700>, the assistant inspected Ray's node list and discovered that the head node was registered as node:10.1.230.180 (the external IP) rather than node:192.168.200.12 (the IB IP that VLLM_HOST_IP was set to). The assistant correctly deduced that vLLM's placement group was requesting a node that didn't exist in Ray's registry.

The fix was to remove the IP forcing entirely. Instead of trying to make Ray use the IB subnet IP (which would require changing Ray's startup configuration), the assistant let Ray use its default auto-detection and removed the VLLM_HOST_IP overrides. This is a pragmatic decision — it accepts Ray's networking behavior and adapts the deployment to match reality rather than fighting the framework.

The command structure shows systematic thinking. The assistant doesn't just launch the cluster and hope — it chains a verification step (docker exec ... python3 -c "import ray; ...") that confirms the cluster formed correctly before proceeding. This is the mark of an engineer who has been burned by silent failures and has learned to validate each step.

The use of && chaining is deliberate. Each step depends on the previous one. If scp fails (network issue), the script won't try to launch. If the head fails to start, the worker won't attempt to connect. This atomicity ensures the cluster is either fully formed or not started at all — no half-initialized state.

The sleep 10 is a pragmatic heuristic. Rather than implementing a proper readiness check (which would require polling Ray's status in a loop), the assistant uses a fixed 10-second delay. This is a reasonable trade-off: it's simple, it works for the typical startup time, and the verification step afterward catches any failures.

Conclusion

Message <msg id=6704> is a quiet victory lap after a long debugging session. It is the moment when the networking puzzle finally clicks into place, when the Ray cluster forms correctly on the first try after multiple failed attempts. The command is dense with meaning: it encodes the corrected deployment script, the verified networking configuration, and the systematic validation approach that characterizes mature infrastructure engineering.

For the reader, this message illustrates a crucial lesson about distributed AI serving: the hardest problems are often not about the model itself, but about the networking and orchestration layers beneath it. Getting two GPUs in different physical machines to cooperate on a single inference request requires solving IP routing, Ray placement groups, NCCL socket interfaces, and container networking — all before a single token is generated. The assistant's methodical debugging, from inspecting Ray's node registry to removing the conflicting VLLM_HOST_IP, is a textbook example of how to approach these challenges.

The successful cluster formation in this message directly enables the subsequent deployment of Qwen3.5-122B-A10B-FP8 at ~27 tok/s, proving that the infrastructure foundation is sound. It is a reminder that in the world of large model deployment, the quiet orchestration moments are often the most important ones.