The Pivot: How Discovering Ray 2.53 in a Docker Image Unlocked Multi-Node vLLM Deployment

"It has Ray 2.53! So we can do the multi-node setup the standard way."

This single sentence, uttered in message [msg 6687] of an intense coding session, marks a critical inflection point in a multi-hour struggle to deploy the Qwen3.5-122B-A10B-FP8 model across two DGX Spark nodes. The message is deceptively brief — a mere 34 words followed by a file write — but it encapsulates the culmination of a long chain of failed attempts, a moment of discovery, and a strategic pivot that would ultimately lead to success.

The Full Message

[assistant] It has Ray 2.53! So we can do the multi-node setup the standard way: start Ray on both nodes, then run vLLM. Let me write a proper multi-node launch script for this image:
[write] /home/theuser/glm-kimi-sm120-rtx6000bw/spark-vllm-qwen35.sh
Wrote file successfully.

The Context of Exhaustion

To understand why this message matters, one must appreciate the gauntlet of failures that preceded it. The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter FP8 quantized reasoning model — across two NVIDIA DGX Spark systems connected via InfiniBand RoCE. The deployment environment was a Docker-based multi-node setup using a custom launch-cluster.sh script that handled Ray orchestration, NCCL configuration, and container lifecycle management.

The first attempt used the hellohal2064/vllm-qwen3.5-gb10 image directly, but the cluster script's IMAGE_NAME override didn't take effect ([msg 6668]). A second attempt with the -t flag worked but timed out ([msg 6670]). The assistant then pursued an increasingly complex series of workarounds: building a hybrid Docker image that grafted model files from vLLM 0.17 onto a vLLM 0.14 base ([msg 6673]), patching the model registry with Python scripts ([msg 6679]), and overriding container entrypoints ([msg 6674]). Each attempt failed for a different reason — incompatible base classes, shell escaping issues in zsh, missing entrypoint scripts, and model inspection failures.

By message [msg 6686], the assistant had reached a clear conclusion: "The model files from v0.17 aren't compatible with v0.14's base classes. This approach of grafting files between versions won't work." This realization was the necessary precondition for the pivot that follows in [msg 6687].

The Discovery

The critical insight came at the end of [msg 6686]. After acknowledging that grafting wouldn't work, the assistant asked a simple question: does the hellohal2064/vllm-qwen3.5-gb10 image have Ray? The answer, obtained via a docker run command that piped pip show ray through a Python import check, was emphatically yes: Ray 2.53.0.

This discovery was transformative. Until this moment, the assistant had been operating under an implicit assumption: that the hellohal2064 image used a different, non-Ray multi-node mechanism (perhaps vLLM's built-in distributed runtime or a custom entrypoint). The launch-cluster.sh script, which the assistant had been trying to force-fit, was designed for the vllm-node image that explicitly used Ray. The incompatibility between the two images had been the source of all the entrypoint conflicts and startup failures.

The discovery that the hellohal2064 image already contained Ray 2.53 meant that the entire grafting approach was unnecessary. The assistant didn't need to copy model files, patch registries, or override entrypoints. It could simply use the image as-is, with standard Ray multi-node orchestration.

The Reasoning Behind the Pivot

The message in [msg 6687] reveals several layers of reasoning:

Strategic simplification. The assistant recognizes that "the standard way" — starting Ray on both nodes, then running vLLM — is the most maintainable and least error-prone approach. This is a classic engineering tradeoff: earlier attempts tried to minimize change by reusing existing infrastructure (the launch-cluster.sh script), but each workaround introduced new failure modes. The "standard way" requires more upfront setup (writing a new script) but eliminates the compatibility risks.

Proactive automation. Rather than manually running Ray start commands on both nodes, the assistant immediately decides to write a script. The file path — /home/theuser/glm-kimi-sm120-rtx6000bw/spark-vllm-qwen35.sh — shows this is intended as a reusable artifact, not a one-off hack. The script will encapsulate the Ray head/worker startup, NCCL configuration, and vLLM launch parameters, making the deployment repeatable and auditable.

Learning from past failures. The earlier networking issues (Ray auto-detecting the wrong external IP in [msg 6670]) inform the assistant's approach. The "standard way" will need to handle the IB subnet addressing, NCCL interface selection, and OOM killer mitigation that plagued earlier attempts. The script isn't just a convenience — it's a record of the operational knowledge accumulated through failure.

Assumptions Embedded in the Message

The message rests on several assumptions, some explicit and some implicit:

Ray 2.53 is compatible with vLLM 0.17. The assistant assumes that the Ray version bundled in the hellohal2064 image will work correctly with vLLM's multi-node tensor parallelism. This is a reasonable assumption given that the image was specifically built for Qwen3.5 on DGX Spark, but it's not verified until the cluster actually forms.
Standard Ray multi-node will handle the IB networking. The earlier attempts revealed that Ray's auto-detection picked up the wrong IP (10.1.230.180 instead of 192.168.200.x). The assistant assumes that explicitly specifying node IPs and NCCL socket interfaces will resolve this — an assumption that proved correct in subsequent messages.
The model will load correctly in multi-node mode. The assistant had previously verified that the model loads and serves correctly in single-node mode using the hellohal2064 image. The assumption is that TP=2 across two nodes will work identically, which depends on NCCL being able to communicate across the InfiniBand link.
Writing a script is the right level of abstraction. The assistant chooses to write a shell script rather than modifying the existing launch-cluster.sh or using Docker Compose. This assumes that the deployment complexity is manageable in a single script and that future iterations won't need more sophisticated orchestration.

Input Knowledge Required

To understand this message, one needs:

The history of failed approaches. Without knowing about the grafting attempts, the registry patching, and the entrypoint overrides, the pivot seems like an overreaction to a simple problem. The message's significance is entirely contextual.
The Ray multi-node architecture. The phrase "start Ray on both nodes, then run vLLM" implies knowledge of Ray's head/worker topology, where one node runs the Ray head (with the GCS/object store) and worker nodes connect to it. The assistant knows that vLLM can use Ray for distributed tensor parallelism.
The DGX Spark networking topology. The IB subnet (192.168.200.x), the RoCE interfaces (rocep1s0f1), and the NCCL socket interface configuration are all prerequisites for making the "standard way" work.
The model's requirements. The Qwen3.5-122B-A10B-FP8 model requires specific vLLM version support (0.17+), the qwen3_5 model class, the qwen3_5_moe architecture, and specific serving parameters (reasoning parser, tool call parser, etc.).

Output Knowledge Created

This message creates:

The script file spark-vllm-qwen35.sh, which will contain the Ray multi-node launch logic. This is the primary artifact — a reusable deployment tool that encapsulates the operational knowledge gained from the entire session.
A validated deployment strategy. The message establishes that using the hellohal2064 image with standard Ray multi-node is the correct approach, replacing the failed grafting strategy. This strategic knowledge guides all subsequent actions.
A template for future multi-node deployments. The pattern established here — use a pre-built image with Ray, write a launch script that handles node IPs and NCCL configuration, then run vLLM with TP=N — can be applied to other models and node configurations.

The Thinking Process

The message's thinking is visible in its structure. The exclamation "It has Ray 2.53!" conveys genuine discovery — this was not an obvious property of the image, and the assistant seems surprised. The phrase "So we can do the multi-node setup the standard way" is the logical conclusion being drawn in real-time. The decision to write a script rather than run commands interactively shows forward-thinking: the assistant anticipates needing to repeat this process, possibly multiple times as configuration issues are resolved.

The message is also notable for what it doesn't say. There's no lament about the wasted effort on the grafting approach, no self-criticism for not checking for Ray earlier. The assistant simply absorbs the new information and pivots. This is characteristic of effective debugging: the goal is not to assign blame but to find the shortest path to a working system.

Aftermath

The subsequent messages confirm the pivot was correct. The script was deployed to both nodes ([msg 6688]), Ray head was started successfully ([msg 6689]), and within a few rounds the full multi-node deployment was serving the model at ~27 tok/s with correct reasoning output. The "standard way" worked.

In retrospect, message [msg 6687] is the turning point of the entire segment. Before it: a cascade of increasingly desperate workarounds. After it: a clean, working deployment. The message is a testament to the power of stepping back, re-examining assumptions, and choosing the simplest path forward — even when that path means abandoning significant prior investment.