The Pivot Point: Launching Multi-Node vLLM Across Two DGX Spark Nodes

Introduction

In the sprawling narrative of deploying large language models across distributed GPU infrastructure, some messages are dense with technical complexity while others are deceptively simple. Message <msg id=6692> belongs to the latter category. On its surface, it is a single bash command—a one-liner that launches a vLLM server across a two-node Ray cluster. But this message is not merely an execution; it is a culmination. It represents the moment when a long sequence of failed approaches, dead ends, and architectural pivots finally converges into a working deployment strategy. Understanding why this message was written, what assumptions underpin it, and what knowledge it both consumes and produces reveals the deep structure of distributed ML serving.

The Message

The assistant writes:

2 nodes active in the Ray cluster. Now launch vLLM serve:

>

``bash ssh aurora@10.1.230.180 'nohup /home/aurora/spark-vllm-qwen35.sh serve > /home/aurora/vllm-qwen35-serve.log 2>&1 & echo "Serve started, PID: $!"' 2>&1 ``

>

Serve started, PID: 2438758

The message is composed of two parts: a status confirmation ("2 nodes active in the Ray cluster") and the launch command. The brevity is itself meaningful—it signals that the assistant has reached a point of confidence where no further diagnostics are needed before proceeding.

The Reasoning and Motivation

To understand why this message was written, one must trace the path that led to it. The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark systems (each with a single GB10 Blackwell GPU, 120GB unified memory, connected via InfiniBand RoCE). This is a 119-billion-parameter FP8 model that requires tensor parallelism across both nodes—meaning the model is too large for a single GPU and must be split across the two machines.

The journey to this point was arduous. In the preceding messages, the assistant tried and abandoned multiple strategies:

1. Direct use of launch-cluster.sh with the hellohal2064/vllm-qwen3.5-gb10 image (messages [msg 6670]–[msg 6676]): This failed because the image's entrypoint conflicted with the cluster script's exec action, and overriding the entrypoint caused the script's own run-cluster-node.sh to be unfindable.
2. Building a hybrid Docker image (messages [msg 6679]–[msg 6686]): The assistant attempted to graft Qwen3.5 model files from vLLM 0.17 onto the vllm-node base image (vLLM 0.14). This failed because the model implementation depended on APIs that changed between versions—the registry entries were found, but the model class failed to be inspected due to incompatible base classes.
3. Patching the registry and config files (messages [msg 6679]–[msg 6682]): Even with careful Python-based registry patching and transformers upgrades, the fundamental API incompatibility between vLLM 0.14 and 0.17 could not be papered over. Each failure taught the assistant something important. The first attempt revealed that the hellohal2064 image had its own entrypoint that was incompatible with the existing cluster infrastructure. The second attempt revealed that model files are not portable across vLLM versions—they depend on internal APIs that change between releases. The third attempt confirmed that the only clean path was to use vLLM 0.17 in its entirety. The critical insight came in message [msg 6687]: the assistant checked whether the hellohal2064 image contained Ray, and discovered it had Ray 2.53. This was the key realization—the image did support multi-node deployment, just not through the existing launch-cluster.sh script. The assistant pivoted from trying to force the image into an existing infrastructure to building custom infrastructure around the image.

The Decision-Making Process

Message [msg 6692] represents the execution of a decision that was made implicitly in the preceding messages. The assistant wrote a custom script spark-vllm-qwen35.sh (message [msg 6687]), copied it to both nodes (message [msg 6688]), started the Ray head on the first node (message [msg 6689]), started the Ray worker on the second node (message [msg 6690]), and verified the cluster state (message [msg 6691]). Only after confirming that both nodes were active did the assistant proceed to launch the vLLM serve command.

The decision to write a custom script rather than continuing to adapt the existing launch-cluster.sh is itself significant. It represents a judgment that the existing infrastructure was too tightly coupled to the vllm-node image and could not be cleanly adapted to the hellohal2064 image. The assistant chose to build a lightweight wrapper—starting Ray containers manually, then running vLLM serve inside the head container—rather than fighting the existing tooling.

Several design decisions are embedded in this approach:

• Using nohup and backgrounding: The serve command is run with nohup and & so it survives the SSH session and runs as a daemon. This is essential for a production service.
• Logging to a file: Output is redirected to /home/aurora/vllm-qwen35-serve.log, allowing the assistant to monitor progress and diagnose failures in subsequent messages.
• Running serve on the head node only: The script presumably starts vLLM serve only on the Ray head node, relying on Ray's distributed execution to handle the worker node. This is the standard pattern for Ray-based vLLM deployments.

Assumptions Made

This message rests on several critical assumptions:

That the Ray cluster is correctly configured for GPU work. The assistant had verified that Ray saw 2 GPUs (1 per node) in message [msg 6691], but the ray status output shown only included CPU resources. The assistant assumed the GPUs were visible to Ray's resource scheduler, which later turned out to be only partially correct—Ray saw the GPUs, but vLLM's placement group resolution would fail because of a host IP mismatch (as revealed in messages [msg 6697]–[msg 6700]).

That the spark-vllm-qwen35.sh serve command includes all necessary arguments. The script was written to wrap the vLLM serve command with the correct model path, tensor parallel size, and other parameters. The assistant assumed the script was complete and correct.

That the model files are accessible from within the Docker container. The assistant had mounted the model directory in earlier attempts, but the serve command in the custom script needed to reference the correct path inside the container.

That the NCCL/InfiniBand interconnect would work transparently. The assistant had set NCCL_SOCKET_IFNAME and related environment variables in the script, but the actual NCCL initialization across the two nodes would prove to be a source of further issues.

Mistakes and Incorrect Assumptions

The most significant incorrect assumption—revealed in the very next message ([msg 6693])—was that the serve command would work without the --distributed-executor-backend ray flag. When using tensor parallelism across multiple nodes (TP=2 with 1 GPU per node), vLLM requires explicit specification of the distributed backend. The initial launch failed because vLLM defaulted to a different backend that couldn't coordinate across nodes.

Additionally, the assistant assumed that the Ray node IPs (192.168.200.x from the InfiniBand subnet) would match what vLLM used for placement group resolution. As messages [msg 6699]–[msg 6700] revealed, the Ray head node was registered with its external IP (10.1.230.180), not the IB subnet IP, causing a mismatch in vLLM's placement group constraints. This required a fix to VLLM_HOST_IP in the script.

The assistant also assumed that ray status showing 2 nodes was sufficient evidence that the cluster was ready for vLLM. In reality, the GPU resources were visible to Ray but the placement group scheduling had additional constraints that weren't apparent from the status output alone.

Input Knowledge Required

To understand this message, one needs knowledge of:

• Ray's architecture: How Ray clusters work with head and worker nodes, how resources are tracked, and how placement groups are used for scheduling.
• vLLM's distributed serving model: How tensor parallelism works across nodes, the role of --distributed-executor-backend, and the VLLM_HOST_IP configuration.
• Docker networking: How containers on different hosts communicate, the role of entrypoints, and volume mounts.
• The DGX Spark platform: Understanding that each Spark has a single GB10 GPU with 120GB unified memory, and that the two nodes are connected via InfiniBand RoCE.
• The Qwen3.5 model architecture: The 122B-A10B-FP8 variant uses FP8 quantization and a Mixture-of-Experts architecture, requiring specific model support in vLLM.
• Shell scripting and SSH: The use of nohup, background processes, and remote command execution.

Output Knowledge Created

This message creates several forms of knowledge:

A verified deployment pattern: The combination of the hellohal2064/vllm-qwen3.5-gb10 image with manual Ray cluster setup and the custom spark-vllm-qwen35.sh script becomes a known-working approach for multi-node Qwen3.5 deployment on DGX Spark.

A diagnostic baseline: The serve log file (/home/aurora/vllm-qwen35-serve.log) begins accumulating output that will be used in subsequent messages to diagnose and fix the distributed backend and host IP issues.

A process checkpoint: The assistant now has a running Ray cluster with both nodes active and a vLLM serve process starting. This is the foundation for all subsequent debugging and optimization.

Documentation of failure modes: The sequence of failed approaches leading to this message implicitly documents that (a) vLLM model files are not portable across versions, (b) the launch-cluster.sh script is tightly coupled to the vllm-node image, and (c) the hellohal2064 image requires a custom launch approach.

The Thinking Process Visible in Reasoning

The assistant's reasoning is most visible in the progression of messages leading to [msg 6692]. Each failure is met with a diagnostic step that reveals the root cause:

1. Entrypoint conflict → check what the image's entrypoint does
2. Model file incompatibility → check the registry format and API differences
3. Registry patching works but model inspection fails → recognize that the API surface is too different between vLLM 0.14 and 0.17 The critical reasoning leap comes in message [msg 6687]: "It has Ray 2.53! So we can do the multi-node setup the standard way." This reframes the problem. Previously, the assistant was trying to make the image work with the existing cluster script. Now, the assistant recognizes that the image already supports multi-node through Ray—it just needs a different launch procedure. The decision to write a custom script rather than modify the existing one shows systems thinking: rather than continuing to patch a fundamentally incompatible approach, the assistant chose to build the minimal infrastructure needed to leverage the image's native capabilities.

Conclusion

Message [msg 6692] is a turning point in the deployment of Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes. It represents the transition from infrastructure wrestling to active serving—from setting up the stage to starting the performance. The message is short, but it carries the weight of several failed approaches, a critical insight about Ray compatibility, and the beginning of a new set of challenges around distributed backend configuration and network addressing. In the broader narrative of the coding session, this is the moment where the assistant stops fighting the tools and starts using them as intended.