The Orchestrator's Pivot: Re-establishing a Multi-Node Ray Cluster Across Rebooted DGX Sparks

Message at a Glance

Both GPUs clear. Now launch Qwen3.5:

>

``bash ssh aurora@10.1.230.180 '/home/aurora/spark-vllm-qwen35.sh head && ssh aurora@192.168.200.13 "/home/aurora/spark-vllm-qwen35.sh worker" && sleep 8 && docker exec vllm-qwen35 ray status 2>&1 | grep -E "Active|GPU"' ``

>

Output: `` Starting Ray HEAD on 192.168.200.12... vllm-qwen35 f9962e45fafa9a03aabf9f4d05048e1eed33d7c98b8ce306f61595bfb44fb3eb Ray head started. Container: vllm-qwen35 Starting Ray WORKER on 192.168.200.13, connecting to 192.168.200.12:6379... vllm-qwen35 d62eeb6c733697e7bc7a4189a0fd93d59f27db87dfe8459d0fbe731f77ee7b5a Ray worker started. Container: vllm-qwen35 Active: 0.0/2.0 GPU ``

This message, appearing at index 6738 in a sprawling multi-session conversation, is a masterclass in operational discipline. On the surface, it is a simple sequence: clean up old services, start a Ray head node, start a Ray worker node, verify the cluster formed. But beneath this terse shell pipeline lies the culmination of dozens of preceding messages, hours of debugging, and a deep understanding of distributed inference infrastructure. The message is not merely doing something—it is re-establishing a fragile distributed system after an uncontrolled reboot, and doing so with the quiet confidence of someone who has already paid the tuition of failure.

The Context: Why This Message Was Written

To understand this message, one must understand what came before it. The user's request was simple: "relaunch the models" ([msg 6731]). But the assistant had to first assess the state of two entirely separate clusters—a Proxmox-hosted LXC with RTX PRO 6000 GPUs running SGLang, and a pair of DGX Spark nodes running vLLM with Ray. The Proxmox host was unreachable ("No route to host"), and the Sparks had been rebooted, meaning their old GLM-based services had auto-started, occupying the GPUs with stale containers.

The assistant's response in [msg 6737] was the cleanup: stopping systemd services (vllm-cluster.service, vllm-proxy.service, vllm-embeddings.service, vllm-reranker.service), force-removing Docker containers, and verifying via nvidia-smi that both GPUs on both nodes were back to idle (~18 MiB consumed by Xorg/gnome-shell, essentially free). This was the prerequisite—the surgical removal of the old deployment to make room for the new one.

Message 6738 is therefore the pivot point: the moment the assistant transitions from cleanup to construction. It is the first step in re-deploying the Qwen3.5-122B-A10B-FP8 model after a full system reboot, and it must work correctly because every subsequent step—model loading, CUDA graph capture, serving, benchmarking—depends on the Ray cluster being stable.

The How: A Deliberate Sequence of Operations

The command pipeline is worth dissecting in detail. It chains four operations on a single SSH invocation to the head Spark (10.1.230.180):

1. /home/aurora/spark-vllm-qwen35.sh head — Starts the Ray head node inside the vllm-qwen35 Docker container, binding to 192.168.200.12 (the InfiniBand subnet IP, not the external 10.1.230.180 address). This was a hard-won fix from earlier messages (<msg id=6709-6712>), where the assistant discovered that Ray's default IP detection picked the external interface, making the worker unreachable.
2. ssh aurora@192.168.200.13 "/home/aurora/spark-vllm-qwen35.sh worker" — From the head node, SSH into the second Spark over the IB network and start the Ray worker, connecting back to the head at 192.168.200.12:6379.
3. sleep 8 — A brief pause to allow Ray to fully initialize and register the worker's resources.
4. docker exec vllm-qwen35 ray status 2>&1 | grep -E "Active|GPU" — Query the Ray cluster status from inside the head container, filtering for active nodes and GPU resources. The output confirms success: two Docker container IDs printed (one per node), both Ray processes started cleanly, and the status line 0.0/2.0 GPU indicates that the cluster sees 2 GPUs across 2 nodes, with 0 currently allocated to any Ray task. This is exactly the expected state—the GPUs are registered as available resources but not yet claimed by the vLLM engine.

Assumptions Embedded in This Message

Every line of this command carries implicit assumptions, some of which were validated only through prior failure:

• The shell script persists across reboots: The assistant assumes that /home/aurora/spark-vllm-qwen35.sh survived the Spark reboot. This is reasonable for a home directory on a local drive, but not guaranteed—the script could have been on a tmpfs or ephemeral volume.
• The Docker image is still present: The hellohal2064/vllm-qwen3.5-gb10 image (vLLM 0.17.1rc1, specifically built for Qwen3.5 on GB10) must still be cached locally. If the reboot triggered a Docker garbage collection or the image was pulled to a tmpfs-backed store, it would need to be re-downloaded.
• The IB network configuration persists: The 192.168.200.x subnet, the enp1s0f1np1 interface, and the NCCL socket configuration are all assumed to survive the reboot. In practice, network interface naming and IP assignment can change after reboot, especially on systems with multiple NICs.
• The model weights are still cached: The Qwen3.5-122B-A10B-FP8 model (119 GB) was previously downloaded to /models/ on the head Spark and rsynced to the worker. The assistant assumes this directory survived the reboot.
• No port conflicts: Port 6379 (Ray), 30000 (vLLM API), and the NCCL communication ports are assumed free after the old containers were removed.

Mistakes and Incorrect Assumptions

The most notable output is 0.0/2.0 GPU. A less experienced operator might see this and worry—"zero GPUs active, something is wrong!" But the assistant correctly interprets this as the expected pre-allocation state. The GPUs are registered as Ray resources but not yet assigned to any task; they will be claimed when the vLLM serve command (issued in the very next message, [msg 6739]) starts the engine workers. This is not a mistake, but it is a potential point of confusion for readers unfamiliar with Ray's resource accounting.

A more subtle assumption: the assistant does not explicitly verify that the Docker containers from the old GLM deployment are fully removed before starting the new ones. The cleanup in [msg 6737] used docker rm -f with explicit container names (vllm_node reranker vllm_embeddings), but the new deployment uses a container named vllm-qwen35. If a stale vllm-qwen35 container from a previous session still existed (perhaps started automatically after reboot), the docker run inside the shell script would fail with a name conflict. The assistant implicitly trusts that no such container exists—a reasonable assumption given the reboot, but not verified.

Input Knowledge Required

To fully understand this message, a reader needs:

• Ray architecture: The distinction between head node (cluster management, GCS, dashboard) and worker node (task execution, GPU resources). The ray status command and its output format.
• DGX Spark (GB10) hardware characteristics: ARM Cortex-X925 CPU, 120 GB unified LPDDR5x memory shared between CPU and GPU, single GPU per node, InfiniBand RoCE (RDMA over Converged Ethernet) for inter-node communication.
• The history of networking fixes: Why 192.168.200.12 is used instead of 10.1.230.180—the worker node cannot reach the head on the external IP, so all distributed communication must be forced onto the IB subnet.
• vLLM's Ray integration: vLLM uses Ray for multi-node tensor parallelism (TP=2 in this deployment), where each GPU holds half the model weights and communicates activations over NCCL.
• The shell script's structure: The spark-vllm-qwen35.sh script encapsulates all the environment variables (VLLM_HOST_IP, NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME, RAY_memory_monitor_refresh_ms=0) that were painstakingly debugged in earlier messages.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

1. Ray cluster is healthy: Two nodes, two GPUs, all registered. The cluster survived the reboot and the networking configuration (forced to IB subnet) is intact.
2. The shell script is reboot-resilient: The fact that spark-vllm-qwen35.sh head and spark-vllm-qwen35.sh worker both succeed confirms that the script, the Docker image, and the network configuration all persisted through the reboot. This is a significant validation of the deployment's robustness.
3. GPU state is clean: The 0.0/2.0 GPU line, combined with the prior nvidia-smi output showing ~18 MiB per GPU, confirms that no memory fragmentation or GPU process leaks survived the reboot.
4. The deployment pipeline is reproducible: The assistant has now demonstrated the full lifecycle—initial deployment, cleanup after reboot, and re-deployment—using the same script and process. This is the difference between a one-shot hack and a reproducible deployment.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible in the surrounding messages, follows a clear pattern: assess, clean, build, verify.

In [msg 6735], after discovering both the Proxmox host and the LXC are unreachable, the assistant explicitly states: "Let me focus on what I can reach: the Sparks." This is a pragmatic triage decision—don't chase unreachable systems, work with what's available.

In [msg 6737], the assistant methodically stops systemd services, removes Docker containers, and checks GPU memory on both nodes. The command structure shows careful thought: sudo systemctl stop for managed services, docker rm -f for containers, then nvidia-smi for verification. The worker check is done via SSH from the head, showing awareness that the assistant's direct SSH access is only to the head node.

Then in [msg 6738], the assistant proceeds to launch. The command is structured as a single SSH invocation that chains multiple operations, minimizing latency and ensuring atomicity—if any step fails, the whole command fails and the assistant will see the error in the output.

The sleep 8 between starting Ray and checking status is a deliberate choice. Ray's resource registration is asynchronous; the worker needs time to connect to the head, register its resources, and have them reflected in the cluster state. Eight seconds is a heuristic—long enough for a fast connection over IB, short enough to avoid excessive waiting if something is broken.

The grep -E "Active|GPU" filter on ray status is a signal-extraction technique: ray status prints a verbose output with object store memory, task queues, and other details. The assistant knows that the two critical lines are the active node count and the GPU resource line, and filters accordingly.

The Broader Significance

This message, for all its brevity, represents a critical inflection point in the deployment lifecycle. The assistant has navigated a multi-node distributed system through a reboot—one of the hardest tests of infrastructure robustness. The fact that the Ray cluster forms cleanly, with the correct networking configuration, on the first attempt after reboot, validates every fix applied in the preceding 30+ messages.

The 0.0/2.0 GPU output is a quiet triumph. It says: the cluster is ready, the GPUs are waiting, the infrastructure is sound. The model loading and serving that follow (in <msg id=6739-6740>) will build on this foundation, culminating in a fully operational Qwen3.5-122B-A10B-FP8 service serving 27 tok/s on two ARM-based DGX Sparks—a deployment that, just hours earlier, was failing with Ray OOM kills, NCCL connection timeouts, and SGLang multi-node hangs.

In distributed systems engineering, the most important work is often invisible. This message is the visible tip of an iceberg of debugging, the moment when all the fixes click into place and the system behaves as designed.