The Pivot Point: Reconciling Two Docker Worlds to Deploy Qwen3.5 Across Dual DGX Spark Nodes

Introduction

In the course of deploying the massive Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark nodes, the assistant reaches a critical inflection point in message [msg 6676]. This message is not about a triumphant breakthrough—it is about the moment when a series of failed integration attempts forces a fundamental rethinking of strategy. It is a message of diagnostic clarity, where the assistant steps back from the rubble of three failed approaches, identifies the root incompatibility at each level of abstraction, and commits to a clean, proper solution: building a custom Docker image that combines the best of two worlds.

To understand why this message matters, one must appreciate the technical landscape. The DGX Spark is an ARM-based NVIDIA GB10 system (SM121 Blackwell GPU, Cortex-X925 CPU, 120GB unified memory) connected via InfiniBand RoCE to a second identical node. The model being deployed—Qwen3.5-122B-A10B-FP8—is a 122-billion-parameter FP8 quantized MoE model weighing approximately 119GB. It cannot fit on a single node's 120GB of unified memory (leaving room for OS, drivers, and overhead), so tensor parallelism across both nodes is mandatory. This requires a multi-node inference engine with NCCL-based inter-node communication, which in turn demands Ray or a similar distributed runtime.

The assistant has two Docker images available, each containing half of the solution—and neither containing the whole.

The Two Images and Their Incompatible Virtues

The hellohal2064/vllm-qwen3.5-gb10 image contains vLLM 0.17.1rc1.dev25, a bleeding-edge build that includes native support for the Qwen3.5 model architecture. This image successfully loads and begins serving the model on a single node (as demonstrated in [msg 6664]), but it lacks the Ray-based multi-node orchestration infrastructure needed for tensor parallelism across two DGX Sparks. Its entrypoint is a custom wrapper script that sets default arguments and launches vllm serve directly—it does not participate in any cluster formation protocol.

The vllm-node image, on the other hand, is the existing production base for the DGX Spark cluster. It contains vLLM 0.14.0rc2, Ray, NCCL configuration, and the launch-cluster.sh orchestration script that handles multi-node Docker container lifecycle, Ray head/worker setup, and NCCL interface detection. But vLLM 0.14 predates Qwen3.5 support—the model architecture is simply not in its model registry.

The assistant's task is to get Qwen3.5-122B-A10B-FP8 running across both nodes. The obvious path—use hellohal2064's image with launch-cluster.sh—has already been tried and failed in the messages leading up to [msg 6676].

The Three Failed Approaches

The message opens with the assistant analyzing the most recent failure. In [msg 6675], the assistant attempted to override the Docker entrypoint of the hellohal2064 image using --entrypoint="" and passing it through the launch-cluster.sh script via the VLLM_SPARK_EXTRA_DOCKER_ARGS environment variable. The result was a runtime error:

docker: Error response from daemon: failed to create task for container: 
failed to create shim task: OCI runtime create failed: runc create failed: 
unable to start container process: error during container init: 
exec: "./run-cluster-node.sh": stat ./run-cluster-node.sh: no such file or directory: unknown

The assistant correctly diagnoses the problem: "The launch-cluster.sh tries to run ./run-cluster-node.sh (which is its own script for Ray setup), but with --entrypoint="" the image can't find it." The empty entrypoint causes Docker to fall back to the image's default CMD, but launch-cluster.sh explicitly passes ./run-cluster-node.sh as the command to execute. That script exists inside the vllm-node image but not inside the hellohal2064 image. The fundamental assumption—that overriding the entrypoint would let the cluster script take control—was wrong because the cluster script's own binary was missing from the foreign image.

The assistant then proposes a second approach: "directly use the existing vllm-node image with the launch-cluster.sh script, but mount the qwen3_5 model files from the hellohal image as volume overlays." This is a clever idea—bind-mount the Python model files over the top of the vllm-node image's filesystem at runtime, giving vLLM 0.14 the Qwen3.5 model classes it needs. But the assistant immediately identifies the fatal flaw: "the real issue is vLLM 0.14 doesn't have the Qwen3.5 model class in its registry. Even if I copy the files in, the __init__.py model registry won't know about them." The model registry is not a simple file listing—it is a Python dictionary mapping architecture strings to module paths, defined in registry.py and __init__.py. Dropping a .py file into the models directory does not register it. The registry must be patched at the code level.

This insight reveals a deep understanding of vLLM's internal architecture. The assistant knows that model loading in vLLM is not a simple filesystem scan; it is a registry-based dispatch system where the HuggingFace model configuration's architectures field (e.g., ["Qwen3_5MoeForConditionalGeneration"]) is matched against entries in a Python dictionary. Without the registry entry, vLLM will refuse to load the model even if all the implementation files are present.

The third approach—building a hybrid Docker image—was attempted in [msg 6673] and failed due to a transformers version conflict. The vllm-node image (vLLM 0.14) requires transformers<5,>=4.56.0, but the Qwen3.5 model files from vLLM 0.17 require transformers>=5.0 (the newer model architecture is only registered in transformers 5.x). Building FROM vllm-node and COPY --from=hellohal2064 the model files, then upgrading transformers, produced an incompatible dependency graph.

The Decision: Build a Proper Combined Image

At this point in the message, the assistant makes a strategic decision. Instead of continuing to hack around the edges—overriding entrypoints, mounting overlays, or hybrid-building with incompatible dependencies—the assistant commits to doing it properly:

"The cleanest approach is to build a proper Docker image that combines: 1. The vllm-node base (with Ray, multi-node support, correct entrypoint) 2. The qwen3_5 model files + registry updates from hellohal2064"

This is the pivotal insight: the problem is not about file placement or entrypoint configuration. It is about model registry integration. The assistant needs to understand exactly what registry entries the hellohal2064 image has added to support Qwen3.5, so those same entries can be applied to a vLLM 0.17 build that sits on top of the vllm-node Ray infrastructure.

The assistant immediately executes a reconnaissance command to inspect the registry:

docker run --rm --entrypoint bash hellohal2064/vllm-qwen3.5-gb10 -c \
  "grep -n qwen3_5 /opt/vllm/vllm/model_executor/models/registry.py 2>/dev/null || \
   grep -rn Qwen3_5 /opt/vllm/vllm/model_executor/models/__init__.py 2>/dev/null | head -5"

The output reveals the precise registry entries that need to be replicated:

514:        "qwen3_5",
518:        "qwen3_5",
566:    "Qwen3_5MTP": ("qwen3_5_mtp", "Qwen3_5MTP"),
567:    "Qwen3_5MoeMTP": ("qwen3_5_mtp", "Qwen3_5MoeMTP"),

Lines 514 and 518 appear to be entries in the architecture string list (the __init__.py or registry that maps HuggingFace architecture names to vLLM model classes). Lines 566-567 are the MTP (Multi-Token Prediction) variants—the speculative decoding heads that Qwen3.5 uses for its internal MTP-based inference optimization.

What This Message Reveals About the Assistant's Thinking

The reasoning process visible in [msg 6676] is methodical and self-correcting. The assistant does not simply try one thing after another; it analyzes each failure, extracts the root cause, and uses that knowledge to eliminate entire categories of solutions.

The first category eliminated: entrypoint manipulation. The assistant realizes that launch-cluster.sh is tightly coupled to the vllm-node image's filesystem layout. The script expects ./run-cluster-node.sh to exist inside the container. Any image that lacks this file—regardless of entrypoint configuration—cannot be used with the script's exec action. This rules out using the hellohal2064 image directly with the existing orchestration infrastructure.

The second category eliminated: runtime file overlays. The assistant understands that vLLM's model discovery is registry-based, not filesystem-based. Mounting .py files over the existing models directory does not register the new architecture. This rules out any approach that relies on Docker volume mounts or COPY --from without also patching the registry code.

The third category eliminated: quick hybrid builds. The transformers version conflict between vLLM 0.14 (requires transformers<5) and the Qwen3.5 model code (requires transformers>=5) means that simply copying files from one image to another creates an unresolvable dependency conflict. The two vLLM versions are built against incompatible transformers ecosystems.

What remains is the clean solution: build from the hellohal2064 image (which has vLLM 0.17 and the correct transformers version) but add the Ray multi-node infrastructure from vllm-node. This is the path the assistant commits to at the end of the message.

Assumptions and Their Validity

The message operates under several assumptions, most of which are validated by the evidence:

  1. The registry entries are the key missing piece. This assumption is correct. The output confirms that the hellohal2064 image has explicit registry entries for qwen3_5 and its MTP variants. Without these, vLLM cannot map the HuggingFace model configuration to a Python class.
  2. The vllm-node image's Ray infrastructure works correctly. This is a reasonable assumption given that the existing GLM deployment on the same cluster uses this infrastructure successfully.
  3. Building a combined image is feasible. The assistant assumes that the hellohal2064 image can serve as a base, with Ray and cluster scripts added on top. This is a standard Docker layering operation—install Ray, copy the cluster scripts, and ensure the entrypoint is set to launch-cluster.sh's expected script. One potential incorrect assumption is that the registry entries alone are sufficient. The Qwen3.5 model may depend on other changes in vLLM 0.17 beyond the registry—changes to the model executor, attention backend, or quantization kernels. If the hellohal2064 image's vLLM 0.17 has other architectural differences from the vllm-node image's vLLM 0.14, simply adding the registry entries to a vLLM 0.14 build might not work. However, the assistant's plan is to use the hellohal2064 image as the base (keeping vLLM 0.17) and add Ray on top, which avoids this risk entirely.

Input Knowledge Required

To fully understand this message, one needs knowledge of:

Output Knowledge Created

This message produces concrete, actionable knowledge:

  1. The exact registry entries needed for Qwen3.5 support: Lines 514, 518, 566, and 567 of the registry file, showing both the architecture name strings and the MTP variant mappings.
  2. The infeasibility of three integration strategies: Entrypoint override, runtime file overlay, and quick hybrid build are all ruled out with specific technical justifications.
  3. A clear path forward: Build a combined Docker image using the hellohal2064 image as the base (preserving vLLM 0.17 and Qwen3.5 support) and adding Ray multi-node infrastructure on top.

Conclusion

Message [msg 6676] is the turning point in a complex multi-node deployment. It is the moment when the assistant stops fighting the incompatibilities between two Docker images and instead decides to build a unified solution. The reasoning is transparent, self-correcting, and grounded in deep technical knowledge of vLLM's internal architecture, Docker's container execution model, and the constraints of the DGX Spark hardware platform. By identifying that the model registry—not file placement or entrypoint configuration—is the core issue, the assistant demonstrates the kind of systems-level thinking that distinguishes effective infrastructure engineering from trial-and-error hacking. The message is a case study in diagnostic reasoning: each failure narrows the space of viable solutions until only the correct one remains.