The Pivot: How Discovering vLLM 0.17's Qwen3.5 Support Saved a Multi-Node Deployment

Introduction

In any complex infrastructure deployment, there comes a moment when a single observation changes the entire trajectory of the work. Message [msg 6656] in this opencode session represents precisely such a turning point. After spending over a dozen messages wrestling with a deadlocked SGLang multi-node deployment across two NVIDIA DGX Spark systems, the assistant makes a critical discovery: the community-built hellohal2064/vllm-qwen3.5-gb10 Docker image contains native Qwen3.5 support in vLLM 0.17. This single message captures the moment of recognition, the reasoning that validated the find, and the decisive action to transfer the image to the second node — a pivot that would ultimately lead to a successful deployment.

The Context: A Multi-Node Nightmare

To understand the weight of this message, one must appreciate the struggle that preceded it. The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion parameter MoE (Mixture-of-Experts) model in FP8 precision — across two DGX Spark nodes. These are NVIDIA's compact ARM-based desktop supercomputers featuring the GB10 processor (SM121 Blackwell GPU, ARM Cortex-X925 CPU, 120GB unified memory) connected via InfiniBand-over-RoCE.

The initial approach used SGLang, a popular inference engine. But the deployment hit a wall: SGLang's multi-node NCCL initialization would hang indefinitely. Message after message documented the same pattern — NCCL channels would be set up ("Channel 00/0 : 1[0] -> 0[0] [send] via NET/IBext_v11/0"), but the process would never progress past CustomAllreduce is disabled. GPU memory usage remained at a mere 416MB per GPU — the model weights had never even begun loading. The assistant tried waiting longer (7+ minutes), added --disable-custom-all-reduce, and even examined NCCL debug output, but nothing broke the deadlock.

Then came the web search. In [msg 6649], the assistant queried for DGX Spark vLLM images and discovered hellohal2064/vllm-qwen3.5-gb10 — a Docker image specifically built for Qwen3.5 on GB10 hardware, updated just 20 days prior. This was the first hint of an alternative path.

The Discovery: Validating the Image

The assistant pulled the image in [msg 6650] and immediately began probing its contents. But this proved unexpectedly difficult. The vLLM image had an auto-import mechanism that crashed when run without a GPU — every attempt to inspect the model registry or check the version triggered a cascade of pydantic validation errors. The assistant had to work around this by using --entrypoint bash to bypass the auto-import ([msg 6654]).

The initial inspection was ambiguous. The file listing showed qwen3_next_mtp.py, qwen3_omni_moe_thinker.py, and other Qwen3 files — but no files with "3_5" in the name. This was concerning: did the image actually support Qwen3.5, or was the name misleading? The assistant persisted, refining the search to look specifically for qwen3_5 and qwen3.5 patterns across the entire /opt directory.

The breakthrough came in [msg 6655]. The refined search revealed:

/opt/vllm/vllm/model_executor/models/qwen3_5_mtp.py
/opt/vllm/vllm/model_executor/models/qwen3_5.py
/opt/vllm/vllm/transformers_utils/configs/qwen3_5_moe.py
/opt/vllm/vllm/transformers_utils/configs/qwen3_5.py

The image had the goods: model implementation files, MTP (Multi-Token Prediction) support, and custom config classes for the Qwen3.5 MoE architecture. Transformers was at version 4.57.3.

The Subject Message: Analysis and Decision

Message [msg 6656] is where the assistant synthesizes these findings and acts on them. The message opens with a concise technical assessment:

vLLM 0.17 has Qwen3.5 support files (qwen3_5.py, qwen3_5_mtp.py) and also custom config classes (qwen3_5_moe.py, qwen3_5.py in transformers_utils). It's using its own config parser, not relying on transformers for the model class. Transformers 4.57.3 might be OK since vLLM has its own config shim.

This analysis reveals several layers of understanding. First, the assistant recognizes that vLLM 0.17 ships with its own model implementations for Qwen3.5 — these are not relying on the HuggingFace Transformers library to provide the model class. This is significant because it means the image can support Qwen3.5 even if Transformers hasn't yet added the model to its registry. The custom config classes (qwen3_5_moe.py, qwen3_5.py) in transformers_utils serve as a bridge between vLLM's internal model code and the HuggingFace config format.

Second, the assistant notes that "Transformers 4.57.3 might be OK since vLLM has its own config shim." This is a nuanced observation. Transformers 4.57.3 was released before Qwen3.5, so it wouldn't natively understand the model's configuration. But vLLM's custom config parser acts as a compatibility layer, interpreting the model's config.json without needing upstream Transformers support. This is a common pattern in the vLLM ecosystem, where model support often lands in vLLM before it reaches Transformers proper.

The message then states the strategic conclusion:

This is a much better path — vLLM is proven to work multi-node on DGX Spark (the existing setup used it).

This reasoning is critical. The assistant had already observed that the DGX Spark nodes had a pre-existing vLLM-based setup (the vllm-node image used for GLM-4.7-Flash). That setup demonstrably worked across nodes. The SGLang path had failed completely. The vLLM path, by contrast, had a proven track record on this exact hardware. The combination of proven multi-node support plus native Qwen3.5 model code made this the clearly superior choice.

The Action: Image Transfer

The message concludes with the decisive action:

ssh aurora@10.1.230.180 'docker save hellohal2064/vllm-qwen3.5-gb10 | ssh aurora@192.168.200.13 "docker load" 2>&1 | tail -5'

This is a clever one-liner that pipes the Docker image directly from one node to another over SSH, avoiding the need to pull from Docker Hub separately on each machine. The docker save command outputs the image as a tar archive to stdout, which is piped through SSH to the second node where docker load reads it from stdin. This is bandwidth-efficient (the image is already on the head node) and avoids potential rate limits or network issues with Docker Hub. The result confirms success: "Loaded image: hellohal2064/vllm-qwen3.5-gb10:latest."

Assumptions and Knowledge

The assistant makes several assumptions in this message, all of which turn out to be correct:

  1. vLLM 0.17's Qwen3.5 support is functional on GB10 hardware. The image was specifically built for this platform, and the model files are present, but the assistant hasn't actually run the model yet. This is a reasonable assumption given the image's stated purpose.
  2. vLLM's multi-node support will work where SGLang's failed. The existing vLLM setup on these nodes worked for GLM-4.7-Flash, but Qwen3.5 is a much larger model with different architecture (MoE, FP8). The assistant is extrapolating from past success.
  3. The custom config shim will handle Qwen3.5's configuration correctly. This is a technical bet on vLLM's architecture — that its model code can interpret the HuggingFace config.json without Transformers-level support. The input knowledge required to understand this message is substantial. One needs to understand: - The difference between SGLang and vLLM as inference engines - How model support is implemented in vLLM (model registry, custom configs) - The relationship between vLLM and HuggingFace Transformers - The DGX Spark hardware architecture (ARM64, Blackwell SM121 GPU, unified memory) - Docker image transfer techniques (docker save | ssh docker load) - The Qwen3.5 model family (MoE architecture, FP8 quantization) The output knowledge created by this message is equally significant: - vLLM 0.17.1rc1 supports Qwen3.5 with custom model implementations - The hellohal2064/vllm-qwen3.5-gb10 image is viable for this deployment - The image has been successfully transferred to both nodes - A clear path forward exists: abandon SGLang, use vLLM with the existing cluster infrastructure

The Thinking Process

The assistant's reasoning in this message reflects a systematic diagnostic approach. Having exhausted the SGLang path (multiple launch attempts, NCCL debugging, configuration variations), it recognized the need for a fundamentally different approach. The web search was the turning point — it revealed a purpose-built image that addressed both the hardware (GB10) and model (Qwen3.5) requirements simultaneously.

The validation process shows careful skepticism. The assistant didn't just trust the image name — it probed the filesystem, checked for model files, verified the config classes, and assessed the Transformers version. When the initial file listing was ambiguous, it refined the search. This thoroughness prevented a costly mistake: deploying an image that claimed Qwen3.5 support but didn't actually have it.

The decision to transfer the image immediately, rather than waiting to test it first, reflects confidence in the analysis and an understanding that the image transfer was a prerequisite step that could proceed in parallel with further planning. The docker save | ssh technique also shows operational sophistication — minimizing network transfers and avoiding Docker Hub dependency.

Conclusion

Message [msg 6656] is a classic example of a "pivot point" in infrastructure work. It captures the moment when a failing approach is abandoned for a promising alternative, based on careful evidence gathering and technical reasoning. The assistant's method — discover, validate, decide, act — is a model for troubleshooting complex deployments. The subsequent messages in the session would prove this pivot correct, as the vLLM-based deployment would successfully serve the Qwen3.5-122B-A10B-FP8 model across both DGX Spark nodes, achieving approximately 27 tok/s single-request throughput with correct reasoning output. But that success story begins here, with a single message that recognized the right tool for the job.