The Critical Join: Starting the Ray Worker on Spark-2

[assistant] [bash] # Start Ray worker on spark-2
ssh aurora@10.1.230.180 'ssh aurora@192.168.200.13 "/home/aurora/spark-vllm-qwen35.sh worker"' 2>&1
Starting Ray WORKER, connecting to 192.168.200.12:6379...
15858321465bef0abdd929fb3097e46435a8e8d67e08d24449d98a7532d1e247
Ray worker started. Container: vllm-qwen35

At first glance, message [msg 6690] appears to be a routine operation: start a Ray worker process on a second machine. The command is straightforward — SSH into the head node, then SSH from there into the second DGX Spark, and run a script. The output confirms success with a container ID and a status line. But this message is anything but routine. It represents the culmination of a long and arduous debugging journey, the moment when a carefully crafted multi-node strategy finally snaps into place.

The Long Road to This Moment

To understand why this message was written, one must trace the path that led here. The assistant had been tasked with deploying the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter FP8 quantized reasoning model — across two NVIDIA DGX Spark systems (GB10, SM121 Blackwell, ARM Cortex-X925 CPUs, 120GB unified memory each). These two machines were connected via InfiniBand RoCE, and the goal was to split the model across both GPUs using tensor parallelism.

The journey had been a graveyard of failed approaches. The assistant first tried SGLang's official spark image, but it lacked Qwen3.5 support entirely. When the assistant attempted to use SGLang's multi-node NCCL initialization, it hung indefinitely — a dead end. Then the assistant pivoted to the hellohal2064/vllm-qwen3.5-gb10 Docker image, which was specifically built for Qwen3.5 on GB10 and had been verified to load the model correctly. But integrating this image with the existing launch-cluster.sh script (designed for a different vllm-node image) proved impossible: the entrypoint conflicted, and overriding it caused the container to fail because it couldn't find ./run-cluster-node.sh ([msg 6675]).

Undeterred, the assistant attempted a surgical approach: graft the Qwen3.5 model files from the vLLM 0.17-based hellohal2064 image onto the vLLM 0.14-based vllm-node image ([msg 6673]). This required patching the model registry, copying configuration files, and upgrading transformers. The image built successfully, but when launched, the model class failed to be inspected ([msg 6685]). The root cause was clear: vLLM 0.17's model implementation depended on base classes and APIs that simply didn't exist in vLLM 0.14. Grafting files between major versions was fundamentally unsound.

The Pivot That Made This Message Possible

The breakthrough came when the assistant asked a simple question: does the hellohal2064 image have Ray installed? The answer was yes — Ray 2.53.0 ([msg 6686]). This changed everything. The assistant realized it didn't need to fight with the launch-cluster.sh script or attempt cross-version file grafting. It could use the hellohal2064 image directly, start Ray manually on both nodes, and then launch vLLM within the Ray cluster. This is the standard multi-node pattern for vLLM 0.17, and the image already had all the pieces.

The assistant wrote a custom launch script, spark-vllm-qwen35.sh, that encapsulated three operations: head (start Ray head node), worker (start Ray worker and connect to head), and serve (launch vLLM within the Ray cluster). The script was copied to both Spark nodes ([msg 6688]). In the previous message ([msg 6689]), the assistant started the Ray head on spark-1 (192.168.200.12) and confirmed it was running.

Now, in [msg 6690], the assistant executes the second step: joining spark-2 to the cluster.

The Architecture of the Command

The command itself reveals several design decisions. The double-SSH pattern — ssh aurora@10.1.230.180 'ssh aurora@192.168.200.13 ...' — is notable. The assistant does not SSH directly to 192.168.200.13 from its own environment. Instead, it proxies through the head node (10.1.230.180). This is likely because the assistant's own machine (the Proxmox host running the RTX PRO 6000 setup) is on a different network segment and may not have direct connectivity to the 192.168.200.x InfiniBand subnet. The head node acts as a jump host, and the worker script is invoked from there.

The worker script connects to 192.168.200.12:6379 — the default Ray port on the head node. This IP is on the InfiniBand subnet, meaning inter-node Ray communication will use the high-speed RoCE link rather than the slower external network. This was a deliberate choice: earlier in the deployment, the assistant discovered that Ray's auto-detection picked up the external IP (10.1.230.180), which was unreachable from the second Spark. Forcing the node IPs to the IB subnet was one of the key networking fixes ([chunk 42.0]).

The output confirms the worker started successfully inside a Docker container named vllm-qwen35. The container ID 15858321465bef... is returned, and the status line confirms the container is running. This is the same container name used for the head node, meaning both nodes run identically named containers — a common pattern in Docker-based Ray deployments where the container name is consistent across nodes but the hostnames differ.

Assumptions and Their Validation

This message rests on several assumptions, some explicit and some implicit. The most critical assumption was that the hellohal2064 image contained a functional Ray installation. The assistant verified this in [msg 6686] by running a quick inspection inside the image. Without Ray, the entire multi-node strategy would have collapsed, and the assistant would have needed yet another approach.

Another assumption was that the custom launch script, written moments earlier, would work correctly on both nodes. The script was untested at this point — the assistant had written it, copied it to both machines, and was now running it in production. The successful output validates this assumption, but it was a leap of faith.

The assistant also assumed that the networking configuration was correct: that the Ray head was listening on 192.168.200.12:6379, that the worker could reach it over the IB subnet, and that Docker networking on spark-2 was properly configured to allow the container to connect. These assumptions were grounded in earlier work: the assistant had already verified the IB interfaces (rocep1s0f1, roceP2p1s0f1) and set GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME to the correct RoCE interface.

One subtle assumption is that the worker script would correctly detect the node IP. The spark-vllm-qwen35.sh script likely uses --node-ip-address to force the IB subnet IP, preventing Ray from auto-detecting the external IP. This was a lesson learned from earlier failures.

Input Knowledge Required

To understand this message fully, one needs knowledge of several domains. Ray's multi-node architecture is central: the head node runs a Ray GCS (Global Control Store) that workers connect to via a Redis-compatible protocol on port 6379. The concept of a "Ray cluster" as a distributed scheduler that manages resources across nodes is essential.

Docker networking knowledge is required to understand how containers on different hosts communicate. The containers are started with host networking or specific network configurations that allow them to see each other's Ray ports across the IB link.

The InfiniBand/RoCE topology matters: 192.168.200.x is the IB subnet, and the fact that the worker connects to the head on this subnet means inter-node communication will use the high-speed RDMA-capable link rather than the 1GbE management network.

Finally, understanding the vLLM deployment model is crucial: vLLM 0.17 uses Ray as its distributed runtime for tensor parallelism across nodes. The Ray cluster must be formed before vLLM can start, and vLLM's --tensor-parallel-size 2 flag tells it to split the model across both GPUs in the cluster.

Output Knowledge Created

This message creates several pieces of knowledge. Most immediately, it confirms that the Ray worker on spark-2 has successfully joined the cluster. The container ID provides a handle for future Docker operations (logs, inspection, cleanup). The status line confirms the container is running and healthy.

More broadly, this message validates the entire multi-node strategy. The assistant had tried and failed with SGLang, with the launch-cluster.sh script, and with the hybrid image approach. Each failure eliminated a path and narrowed the solution space. The success of this message confirms that the custom Ray-based approach is viable — that the hellohal2064 image can be used for multi-node deployment with manual Ray orchestration.

The next message ([msg 6691]) will confirm the cluster state: ray status shows 2 active nodes with 2 GPUs. The assistant will then launch vLLM serve ([msg 6692]), and after a ~15-minute model load, the service will be operational, achieving ~27 tok/s single-request throughput.

The Thinking Process Visible

The reasoning in this message is largely implicit — the command itself is terse, and the thinking happened in the messages that preceded it. But the thought process is visible in the sequence of actions. The assistant recognized that the hellohal2064 image contained Ray (a discovery that required proactive investigation), understood that Ray could be used for multi-node orchestration independently of the launch-cluster.sh script, and designed a three-step deployment (head, worker, serve) that mirrors the standard Ray workflow.

The double-SSH pattern reveals an awareness of network topology constraints. The assistant understood that its own environment might not have direct access to the IB subnet and chose to proxy through the head node rather than fight with routing tables or VPN configurations. This is a pragmatic decision that prioritizes reliability over elegance.

The assistant also demonstrated an understanding of failure modes. Earlier attempts failed because of entrypoint conflicts, API incompatibilities, and networking issues. Each failure was diagnosed and the approach was adjusted. The custom script approach was designed to avoid all the known pitfalls: it doesn't rely on the image's entrypoint, it doesn't require cross-version compatibility, and it explicitly controls the node IP addresses.

Conclusion

Message [msg 6690] is the quiet pivot point of a complex multi-node deployment. On its surface, it is a single bash command — start a Ray worker on a second machine. But in context, it represents the successful resolution of a long debugging session, the moment when a carefully constructed strategy pays off. The assistant navigated through failed approaches, discovered the key insight (Ray is available in the image), designed a custom solution, and executed it cleanly. The worker joins the cluster, the foundation is laid, and the model deployment can proceed.