The Pivot Point: Launching the Ray Head Node for Multi-Node Qwen3.5 Deployment

Introduction

In the sprawling narrative of deploying a 119-billion-parameter Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark systems, one message stands out as a quiet turning point. It is deceptively brief — a single bash command that starts a Ray head node on the first Spark machine. But this message, <msg id=6689>, represents the culmination of a long and winding debugging journey, the abandonment of several failed approaches, and the first successful step in a new strategy that would ultimately bring the multi-node inference service to life.

The message reads in full:

[assistant] Now launch: Ray head first, then worker, then serve: [bash] # Start Ray head on spark-1 ssh aurora@10.1.230.180 '/home/aurora/spark-vllm-qwen35.sh head' 2>&1 Starting Ray HEAD on 192.168.200.12... 1ce7d8e6fccd678e44b336422befce27c7fa4c9a467a0180b9acbe20b5cece05 Ray head started. Container: vllm-qwen35

To an outside observer, this looks like a routine infrastructure command — SSH into a remote machine, execute a shell script, confirm the container started. But to understand why this message matters, one must understand the labyrinth of failed attempts that preceded it, the architectural decisions crystallized in the spark-vllm-qwen35.sh script, and the precise technical context that made this simple command the correct one at last.

The Road to This Message: A Cascade of Failed Approaches

The assistant had been attempting to deploy Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes for several rounds. The DGX Spark is NVIDIA's compact desktop AI supercomputer, built around the GB10 Grace Blackwell Superchip (SM121 Blackwell GPU, ARM Cortex-X925 CPU, 120GB unified memory). The two nodes were connected via InfiniBand RoCE (RDMA over Converged Ethernet), providing a high-speed interconnect essential for tensor parallelism across machines.

The first attempt used SGLang's official Spark image, but it lacked Qwen3.5 model support and its multi-node NCCL initialization hung indefinitely. The assistant then pivoted to a community image, hellohal2064/vllm-qwen3.5-gb10, which contained vLLM 0.17.1rc1 with proper Qwen3.5 support. However, this image was designed as a standalone container, not as part of the existing spark-vllm-docker cluster infrastructure that used a launch-cluster.sh script based on Ray.

The assistant tried several integration strategies:

  1. Direct script integration (<msg id=6670>): Using launch-cluster.sh with the -t hellohal2064/vllm-qwen3.5-gb10 flag to specify the custom image. This failed because the image's entrypoint conflicted with the script's own run-cluster-node.sh entrypoint.
  2. Entrypoint override (<msg id=6674>): Setting --entrypoint="" via VLLM_SPARK_EXTRA_DOCKER_ARGS. This caused the container to fail with exec: "./run-cluster-node.sh": stat ./run-cluster-node.sh: no such file or directory — the script's own bootstrap file was missing because the image didn't contain it.
  3. Hybrid image grafting (<msg id=6673, 6679, 6682>): Building a custom Docker image that copied Qwen3.5 model files from the vLLM 0.17 image onto the vLLM 0.14 base used by the cluster infrastructure. This failed because the model files depended on APIs and base classes that differed between major versions, producing the error Model architectures ['Qwen3_5MoeForConditionalGeneration'] failed to be inspected. Each failure taught the assistant something important. The grafting approach revealed that vLLM's internal APIs are not stable across major versions — model files from 0.17 cannot simply be copied onto 0.14. The entrypoint override revealed that the cluster script and the image had incompatible expectations about container initialization. The direct integration attempt showed that the community image was not designed to work with the existing Ray-based cluster launcher.

The Critical Insight: Ray Is Already There

The breakthrough came in <msg id=6686>, when the assistant checked whether the hellohal2064/vllm-qwen3.5-gb10 image contained Ray at all. The result was positive: Ray 2.53.0 was installed. This was the key insight that unlocked the correct approach.

Rather than trying to force the community image into the existing cluster script's mold, the assistant recognized that it could use Ray directly — the standard, documented way to build multi-node vLLM deployments. The image already had everything needed; what was missing was a launch script tailored to this image's specific setup.

The assistant then wrote spark-vllm-qwen35.sh (<msg id=6687>), a custom launch script that would:

The Subject Message: Why It Matters

Message <msg id=6689> is the execution of the first step in this new three-phase plan: "Ray head first, then worker, then serve." The assistant explicitly lays out the sequence in the message's opening line, demonstrating a clear, methodical approach after the chaos of the previous failed attempts.

The command itself is straightforward: SSH into the first Spark node (reachable via its external IP 10.1.230.180) and execute spark-vllm-qwen35.sh head. The script runs inside a Docker container based on the hellohal2064/vllm-qwen3.5-gb10 image, starting the Ray head process and binding it to the InfiniBand subnet address.

The output confirms success: a container ID is printed (1ce7d8e6fccd678e44b336422befce27c7fa4c9a467a0180b9acbe20b5cece05), and the message "Ray head started. Container: vllm-qwen35" indicates the head node is alive.

Assumptions and Decisions

This message embodies several assumptions, both correct and incorrect, that shaped the deployment.

Correct assumptions:

Input Knowledge Required

To understand this message fully, one needs knowledge spanning several domains:

  1. Ray architecture: Understanding that Ray uses a head node (which runs the GCS — Global Control Store) and worker nodes that connect to it. The head node must be started first, and its address must be known to workers.
  2. DGX Spark hardware: The GB10 system-on-chip combines a Grace ARM CPU with a Blackwell GPU in a unified memory architecture. The two Sparks are connected via InfiniBand RoCE, which appears as network interfaces rocep1s0f1 and roceP2p1s0f1.
  3. vLLM multi-node deployment: vLLM uses Ray for multi-node tensor parallelism. The --tensor-parallel-size 2 flag distributes the model across two GPUs (one per Spark), requiring NCCL communication over the InfiniBand interconnect.
  4. Docker networking: The script runs inside Docker containers, and the Ray ports (6379 for GCS, 8265 for dashboard) must be exposed or accessible between containers on different hosts.
  5. The Qwen3.5 model family: Qwen3.5-122B-A10B-FP8 is a Mixture-of-Experts model with 122B total parameters but only ~10B active per token (hence "A10B"). It uses FP8 quantization and requires specific model registry entries in vLLM.

Output Knowledge Created

This message produced several concrete outcomes:

  1. A running Ray head container on the first DGX Spark, named vllm-qwen35, with the head node bound to 192.168.200.12. This container would serve as the coordination point for the entire Ray cluster.
  2. Validation of the custom script: The spark-vllm-qwen35.sh head command executed successfully, confirming that the script's Docker invocation, network configuration, and Ray initialization were correct. This was the first time any approach had produced a clean success.
  3. A foundation for the next steps: With the head node running, the assistant could proceed to start the worker on the second Spark (which happened in <msg id=6690>), verify the cluster formation (<msg id=6691>), and finally launch vLLM serve (<msg id=6692>).
  4. Confidence in the approach: After multiple failed attempts spanning image grafting, entrypoint conflicts, and script incompatibilities, this clean success signaled that the direct Ray approach was the correct path forward.

The Thinking Process Visible in the Message

The message reveals the assistant's reasoning in its opening line: "Now launch: Ray head first, then worker, then serve." This is not just a plan — it's a deliberate sequencing decision. The assistant could have tried to start everything at once (as the earlier launch-cluster.sh exec attempts had done), but instead chose a phased approach where each component is verified before proceeding.

The use of the word "Now" is significant. It signals a transition from the planning and preparation phase (writing the script, copying it to both nodes) into the execution phase. The assistant has learned from the previous failures that careful orchestration matters — the head node must be fully initialized before the worker can connect, and both must be ready before vLLM can start.

The comment # Start Ray head on spark-1 in the bash command shows the assistant thinking aloud, documenting the purpose of each step for the human reader. This is characteristic of the assistant's style throughout the conversation: it narrates its reasoning as it goes, making the decision process transparent.

The Broader Significance

Message <msg id=6689> is a microcosm of the entire deployment effort. It represents the moment when complexity collapses into simplicity — after trying to graft, override, and retrofit existing infrastructure, the assistant finally takes the direct approach: use the tool as intended, with a custom script tailored to the specific environment.

This pattern recurs throughout infrastructure engineering: the most elegant solution is often not the one that reuses existing abstractions, but the one that understands the fundamental primitives and composes them directly. The assistant could have continued debugging the launch-cluster.sh integration, but recognizing that Ray was already available and could be used directly was the insight that broke the logjam.

The subsequent messages confirm the success: the worker joined the cluster (<msg id=6690>), Ray reported two active nodes (<msg id=6691>), and vLLM serve launched and loaded the model (<msg id=6692>). The deployment that had eluded the assistant for over a dozen messages was finally in place, all because of the simple, correct decision embodied in this one command.

Conclusion

At first glance, message <msg id=6689> is unremarkable — a single SSH command, a container ID, a confirmation message. But in the context of the broader conversation, it is a watershed moment. It marks the transition from failed experimentation to successful execution, from complex workarounds to direct solutions, from uncertainty to confidence. The assistant's decision to write a custom Ray launch script rather than continue fighting existing infrastructure abstractions was the key insight that made the multi-node Qwen3.5 deployment possible. This message, in its brevity and clarity, encapsulates the essence of effective infrastructure engineering: understand the tools, recognize when existing abstractions don't fit, and build the right abstraction for the task at hand.