The Moment of Launch: Deploying Qwen3.5-122B-A10B-FP8 Across Dual DGX Spark Nodes

Introduction

In the sprawling, multi-session journey of deploying large language models across heterogeneous GPU infrastructure, few moments carry the weight of the one captured in message 6684. After an extended sequence of failed attempts, shell escaping errors, Docker entrypoint conflicts, and version incompatibilities, the assistant finally issues the command that brings everything together: a launch of the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark nodes using a carefully constructed custom Docker image. This message is the climax of a multi-hour debugging saga — the point at which all the pieces click into place and the actual deployment begins.

The Context: A Long Road to Readiness

To understand why message 6684 matters, one must appreciate the context that precedes it. The session began with the user's directive to deploy Qwen3.5-122B-A10B-FP8 — a massive 125-billion-parameter Mixture-of-Experts model in FP8 quantization, weighing approximately 119GB on disk — across two DGX Spark systems. These are not ordinary servers; the DGX Spark is an ARM-based NVIDIA system (SM121 Blackwell, Cortex-X925 CPU) with 120GB of unified memory and InfiniBand-over-RoCE networking. The deployment target was a pair of these nodes connected via a 192.168.200.x InfiniBand subnet.

The assistant's first discovery, back in message 6560, was sobering: the existing Docker-based vLLM infrastructure (v0.14.0rc2) did not support the Qwen3.5 architecture at all. The model registry only contained entries like Qwen3ForCausalLM and Qwen3MoeForCausalLM, but the new model required Qwen3_5MoeForConditionalGeneration. The model files simply did not exist in the installed vLLM version.

This set off a chain of increasingly sophisticated attempts to bridge the gap between what was available and what was needed.

The Failed Approaches

The assistant tried multiple strategies before arriving at message 6684. First, it attempted to use the hellohal2064/vllm-qwen3.5-gb10 image directly — a community-built Docker image specifically designed for Qwen3.5 on GB10 hardware. The launch-cluster.sh script was invoked with this image, but the image's custom entrypoint conflicted with the cluster script's expectation of a Ray-based setup (message 6673). The containers started but never became ready.

Next, the assistant tried overriding the entrypoint with --entrypoint="" (message 6674). This caused the cluster script to fail because its own run-cluster-node.sh script was no longer found inside the container (message 6675). The two systems — the community image and the existing cluster infrastructure — were fundamentally incompatible.

The assistant then pivoted to a hybrid approach: copy the Qwen3.5 model files from the community image into the existing vllm-node base image (message 6673). This failed because the vllm-node image's vLLM 0.14 required transformers<5, while the Qwen3.5 model files from vLLM 0.17 required transformers>=5.0. The version conflict was unresolvable with a simple file copy.

The Breakthrough: Building a Proper Hybrid Image

The turning point came when the assistant recognized that a proper hybrid image was needed — one that not only copied the model files but also patched the model registry and configuration initialization. This required understanding exactly what registry entries the new model needed.

Through a series of Docker inspection commands (messages 6676-6678), the assistant extracted the exact registry entries from the community image:

"Qwen3_5ForConditionalGeneration": ("qwen3_5", "Qwen3_5ForConditionalGeneration"),
"Qwen3_5MoeForConditionalGeneration": ("qwen3_5", "Qwen3_5MoeForConditionalGeneration"),

These entries needed to be injected into the vllm-node base image's registry.py, alongside the corresponding config classes in transformers_utils/configs/__init__.py.

The first attempt to build this patched image failed due to shell escaping issues with the remote host's zsh shell (message 6679). The assistant adapted by writing a standalone Python patch script (patch_vllm_qwen35.py) and copying it to the remote machine via scp (messages 6680-6681). The Dockerfile then used COPY to bring the script into the build context and executed it as a RUN step.

The build succeeded (message 6682), producing the vllm-node-qwen35 image. The assistant then transferred it to the second Spark node using Docker save/load over SSH (message 6683), achieving a transfer rate of approximately 640MB/s over the InfiniBand link.

The Launch Command (Message 6684)

With the custom image built and distributed, the assistant issued the launch command that is the subject of this article:

Now launch using the existing cluster infrastructure: ``bash ssh aurora@10.1.230.180 'cd /home/aurora/spark-vllm-docker && \ VLLM_SPARK_EXTRA_DOCKER_ARGS="-v /home/aurora/models/Qwen3.5-122B-A10B-FP8:/root/.cache/huggingface/hub/Qwen3.5-122B-A10B-FP8:ro" \ nohup ./launch-cluster.sh \ -t vllm-node-qwen35 \ -n 192.168.200.12,192.168.200.13 \ --name vllm_qwen35 \ --nccl-debug WARN \ exec vllm serve /root/.cache/huggingface/hub/Qwen3.5-122B-A10B-FP8 \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.90 \ --max-model-len 32768 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8001 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ > /home/aurora/vllm-qwen35-cluster.log 2>&1 & echo "Cluster launch PID: $!"' 2>&1 Cluster launch PID: 2436585 ``

Analysis of the Command

Every element of this command reflects lessons learned from the preceding failures.

The image tag -t vllm-node-qwen35 is the most critical choice. After trying and failing to use the community image directly, the assistant is now using the custom-built hybrid that combines the Ray-based cluster infrastructure of vllm-node with the Qwen3.5 model support extracted from the community image. This is the synthesis that makes multi-node deployment possible.

The volume mount VLLM_SPARK_EXTRA_DOCKER_ARGS="-v /home/aurora/models/Qwen3.5-122B-A10B-FP8:/root/.cache/huggingface/hub/Qwen3.5-122B-A10B-FP8:ro" maps the model weights from the host filesystem into the container at the HuggingFace cache path. This is a deliberate design choice: rather than downloading the 119GB model inside each container, the assistant mounts it read-only from the pre-downloaded location. The path /root/.cache/huggingface/hub/... is significant — it tells us the model was downloaded using HuggingFace's snapshot_download or similar tool, which organizes models under the hub directory structure.

The node specification -n 192.168.200.12,192.168.200.13 uses the InfiniBand subnet addresses rather than the external network (10.1.230.180). This is a crucial networking decision. Earlier in the session, the assistant discovered that Ray's auto-detection used the external IP which was unreachable from the second Spark. Forcing the InfiniBand subnet addresses ensures that the Ray cluster and NCCL communication use the high-speed RoCE interconnect.

The vLLM arguments reveal the deployment's production intent:

• --tensor-parallel-size 2 splits the model across both GPUs (one per node), which is necessary because the 119GB model exceeds a single DGX Spark's 120GB unified memory.
• --gpu-memory-utilization 0.90 reserves 10% of GPU memory for the framework and KV cache overhead, a lesson likely learned from earlier OOM failures.
• --max-model-len 32768 sets a generous context window, appropriate for a reasoning model.
• --reasoning-parser qwen3 and --tool-call-parser qwen3_coder enable the model's unique output format, where reasoning content is separated from the final answer.
• --enable-auto-tool-choice allows the model to autonomously decide when to invoke tools. The port choice --port 8001 is notable. The standard vLLM port is 8000, but the assistant is using 8001 — likely because port 8000 is already occupied by the proxy service (vllm-proxy.service) that was stopped earlier but whose port may still be reserved.

Assumptions and Decisions

The assistant makes several assumptions in this message. The most significant is that the vllm-node-qwen35 image will work correctly in the multi-node setup. The image was built by patching model registry files, but the underlying vLLM version (0.14.0rc2) is still the same as the base vllm-node image. The assistant is assuming that the model files from vLLM 0.17 are compatible with the vLLM 0.14 runtime. This is a gamble — the model architecture code might depend on internal APIs that changed between versions.

Another assumption is that the launch-cluster.sh script's exec action will properly handle the custom image. The script was designed for the standard vllm-node image, and the assistant is now using it with a derivative. The earlier failures with the community image showed that entrypoint conflicts can break the script's assumptions.

The assistant also assumes that the model weights are correctly organized under the HuggingFace hub directory structure. The volume mount maps the host path directly to the expected container path, but if the model was downloaded with a different tool or the directory structure differs, vLLM may fail to find the configuration files.

The Thinking Process

The assistant's reasoning is visible in the structure of the command itself. The use of nohup and background execution (&) with log redirection shows an understanding that model loading will take significant time (the 119GB model requires approximately 15 minutes to load over the InfiniBand link). The Cluster launch PID: 2436585 output confirms the process is running in the background.

The choice to use --nccl-debug WARN rather than INFO or TRACE reflects a balance between diagnosability and log noise. Earlier attempts used WARN level, and the assistant is maintaining consistency.

The decision to mount the model read-only (:ro) is a defensive measure — it prevents accidental modification of the model weights and signals that the deployment is immutable.

What Follows

Message 6684 is the launch point. In the subsequent messages (6685 onward), the assistant monitors the deployment, discovers that the model loads successfully, and verifies that it produces correct reasoning output. The initial response shows content: None, which the assistant correctly interprets not as an error but as the model properly routing its reasoning into the separate reasoning field — confirming that the reasoning parser is active and working.

This message, then, is the fulcrum on which the entire deployment turns. It represents the successful resolution of every obstacle encountered in the preceding hours: the missing model registry entries, the Docker entrypoint conflicts, the shell escaping issues, the version incompatibilities, and the networking configuration. All of these are addressed in the single command that launches the model into production.