The Pivot Point: Discovering Qwen3.5 Support in vLLM 0.17 on DGX Spark

Introduction

In the complex landscape of deploying large language models across multi-node GPU clusters, few moments are as decisive as the discovery that a critical piece of software actually supports the hardware and model you need. Message [msg 6655] in this opencode session captures precisely such a moment — a turning point where the assistant, after hours of wrestling with SGLang's multi-node NCCL deadlock on two NVIDIA DGX Spark systems, validates that a community-built vLLM Docker image contains the model files necessary to serve Qwen3.5-122B-A10B-FP8 across two nodes. This message is deceptively simple: a single bash command and its output. But within that output lies the resolution of a fundamental uncertainty that had been blocking all progress.

The Message Content

The assistant writes:

This image has vLLM 0.17.1rc1.dev25 — very recent! But no qwen3_5* files. Let me check if there's a qwen3_5_moe.py:

>

[bash] ssh aurora@10.1.230.180 'docker run --rm --entrypoint bash hellohal2064/vllm-qwen3.5-gb10 -c "ls /opt/vllm/vllm/model_executor/models/ | grep -i qwen && echo --- && python3 -c \"import transformers; print(transformers.__version__)\" && find /opt -name \"qwen3_5\" -o -name \"qwen3.5\" 2>/dev/null"' 2>&1 | tail -15

>

qwen3_asr.py qwen3_asr_realtime.py qwen3_moe.py qwen3_next.py qwen3_next_mtp.py qwen3_omni_moe_thinker.py qwen3_vl.py qwen3_vl_moe.py qwen_vl.py --- 4.57.3 /opt/vllm/vllm/model_executor/models/qwen3_5_mtp.py /opt/vllm/vllm/model_executor/models/qwen3_5.py /opt/vllm/vllm/transformers_utils/configs/qwen3_5_moe.py /opt/vllm/vllm/transformers_utils/configs/qwen3_5.py

The output confirms that the image, despite its name suggesting Qwen3.5 support, does indeed contain the critical model implementation files: qwen3_5.py (the main model), qwen3_5_mtp.py (Multi-Token Prediction support), and custom config classes in transformers_utils. This is the validation the assistant needed to commit to the vLLM path.

Why This Message Was Written: The Reasoning and Motivation

The motivation behind this message can only be understood by examining the preceding 25+ messages in this segment. The assistant had been engaged in a protracted battle to deploy Qwen3.5-122B-A10B-FP8 across two DGX Spark nodes — ARM-based NVIDIA GB10 systems with unified memory and InfiniBand RoCE interconnect.

The initial approach used SGLang, which the assistant had successfully deployed on single-node setups earlier in the session. However, multi-node SGLang proved intractable. Across messages [msg 6628] through [msg 6647], the assistant repeatedly launched SGLang containers on both nodes, only to find them stuck at NCCL initialization — the logs showed CustomAllreduce is disabled but never progressed to weight loading. The assistant tried waiting (up to 7+ minutes), adding --disable-custom-all-reduce, and checking NCCL debug output, but the deadlock persisted.

The critical insight came in [msg 6649] when the assistant searched for alternative Docker images and discovered hellohal2064/vllm-qwen3.5-gb10 — a community image specifically built for Qwen3.5 on DGX Spark. In [msg 6650], the assistant pulled the image, and in [msg 6651]-[msg 6654], attempted to verify its contents. However, those earlier verification attempts failed because the vLLM Python import crashed when no GPU was available inside the container. The assistant had to work around this by using --entrypoint bash and running shell commands directly.

By the time we reach [msg 6655], the assistant has already confirmed the image has vLLM 0.17.1rc1.dev25 and a rich set of Qwen3 model files. But the critical question remains: does it support Qwen3.5 specifically? The model name Qwen3.5-122B-A10B-FP8 is distinct from Qwen3, and the model architecture (a Mixture-of-Experts with 122B total parameters, 10B active) requires specific model code. The assistant's earlier command in [msg 6654] listed qwen3_*.py files but didn't show any qwen3_5* files — the find command was cut off mid-output. This ambiguity is what drives the current message: the assistant needs a definitive answer before committing to this deployment path.

How Decisions Were Made

This message represents a verification step rather than a decision point per se, but the decision it enables is monumental. The assistant is executing a systematic validation protocol:

  1. Hypothesis: The hellohal2064/vllm-qwen3.5-gb10 image, despite its name, might only support Qwen3, not Qwen3.5. The earlier truncated output from [msg 6654] showed qwen3_*.py files but no qwen3_5* files, raising doubt.
  2. Test design: The assistant constructs a compound command that (a) lists all Qwen model files in the vLLM models directory, (b) checks the transformers version, and (c) does a broad find for any file with qwen3_5 or qwen3.5 in its name anywhere under /opt.
  3. Execution: The command is run via SSH on the head Spark node, using docker run --rm --entrypoint bash to bypass the GPU-dependent Python auto-import that had crashed earlier.
  4. Interpretation: The output reveals four Qwen3.5-specific files — two model implementations and two config utilities. This is unambiguous evidence that the image supports Qwen3.5 natively. The decision that follows from this verification (visible in [msg 6656]) is immediate: the assistant declares "This is a much better path" and proceeds to transfer the Docker image to the second Spark node via docker save | ssh ... docker load, setting up the vLLM multi-node deployment.

Assumptions Made by the Assistant

Several assumptions underpin this message:

  1. The image name is meaningful: The assistant assumes that an image named vllm-qwen3.5-gb10 is likely to support Qwen3.5, but treats this as a hypothesis to be verified rather than a fact.
  2. Model files imply model support: The assistant assumes that the presence of qwen3_5.py and qwen3_5_mtp.py in the vLLM model registry means the model can actually be loaded and served. This is a reasonable assumption — vLLM's model registry is the authoritative source of supported architectures.
  3. Transformers 4.57.3 is sufficient: The assistant notes the transformers version (4.57.3) and later in [msg 6656] comments that "vLLM has its own config shim," implying that the transformers version doesn't need to natively support Qwen3.5 because vLLM handles configuration parsing internally.
  4. The image will work on both Spark nodes: The assistant assumes that since both DGX Spark nodes have identical hardware (SM121 Blackwell, ARM Cortex-X925), the image will function identically on both.
  5. vLLM multi-node will work where SGLang failed: This is perhaps the most critical assumption. The assistant's reasoning in [msg 6656] states "vLLM is proven to work multi-node on DGX Spark (the existing setup used it)." This is based on the observation that the Spark nodes already had a working vLLM-based GLM deployment.

Mistakes and Incorrect Assumptions

The most notable near-mistake is the assistant's initial statement: "But no qwen3_5* files." This conclusion was drawn from the truncated output of the previous command in [msg 6654], where the find results were cut off. The assistant was about to give up on this image when it decided to run a more targeted check — a decision that proved critical.

This highlights an important pattern: the assistant's initial grep for qwen3 files returned a list that included qwen3_*.py files but not qwen3_5* files, because the ls command only listed files in one directory, while the Qwen3.5 files were found by the broader find command. The assistant correctly identified that the earlier check was incomplete and designed a more thorough verification.

Another subtle assumption worth examining: the assistant assumes that the presence of model files in /opt/vllm/vllm/model_executor/models/ guarantees the model can be served. In practice, model serving also depends on correct weight loading, tensor parallelism support, and memory management — none of which are verified by this file check. However, for the purpose of this message, the file presence is a necessary precondition, and the assistant is proceeding methodically by verifying prerequisites before attempting the actual deployment.

Input Knowledge Required

To fully understand this message, one needs:

  1. The deployment context: The assistant is deploying Qwen3.5-122B-A10B-FP8, a 122B-parameter Mixture-of-Experts model with 10B active parameters, quantized to FP8, across two DGX Spark nodes. Each Spark has a single NVIDIA GB10 GPU (SM121 Blackwell architecture) with 120GB unified memory.
  2. The SGLang failure history: The preceding ~30 messages document the assistant's failed attempts to use SGLang for multi-node serving, culminating in a NCCL deadlock that could not be resolved.
  3. The vLLM/SGLang landscape: Understanding why the assistant pivots from SGLang to vLLM requires knowing that vLLM has more mature multi-node support, particularly on DGX Spark where community images like hellohal2064/vllm-qwen3.5-gb10 have been specifically built and tested.
  4. Model registry mechanics: The significance of files in /opt/vllm/vllm/model_executor/models/ — this is vLLM's internal registry where model architectures are defined. A model cannot be served unless its architecture class exists in this directory.
  5. The hardware architecture: DGX Spark uses ARM Cortex-X925 CPUs, which means Docker images must be ARM64-compatible. The hellohal2064/vllm-qwen3.5-gb10 image is specifically built for ARM64, which is why it works on these systems.

Output Knowledge Created

This message produces several pieces of critical knowledge:

  1. vLLM 0.17.1rc1.dev25 supports Qwen3.5 natively: The model files qwen3_5.py and qwen3_5_mtp.py confirm that vLLM's latest release candidate includes support for the Qwen3.5 architecture, including Multi-Token Prediction.
  2. Custom config parsers exist: The files qwen3_5_moe.py and qwen3_5.py in transformers_utils/configs/ indicate that vLLM has implemented custom configuration parsing for Qwen3.5, rather than relying on HuggingFace transformers to provide it. This is important because transformers 4.57.3 may not natively support Qwen3.5's MoE configuration.
  3. The image is ARM64-native and functional: The successful execution of the docker run command confirms the image works on the DGX Spark's ARM architecture.
  4. A viable deployment path exists: This is the most consequential output. The assistant now has a clear path forward: use the hellohal2064/vllm-qwen3.5-gb10 image with vLLM's proven multi-node infrastructure (Ray-based cluster management) to deploy the model.
  5. The transformers version is 4.57.3: This provides a baseline for compatibility. If issues arise, the assistant knows which transformers version is in play.

The Thinking Process Visible in Reasoning

The assistant's reasoning in this message reveals a methodical, hypothesis-driven approach to troubleshooting:

Step 1 — State the current understanding: "This image has vLLM 0.17.1rc1.dev25 — very recent!" The assistant acknowledges the positive finding (recent vLLM version) but immediately flags the concern: "But no qwen3_5* files."

Step 2 — Formulate a specific hypothesis: "Let me check if there's a qwen3_5_moe.py." The assistant is specifically looking for the MoE variant of Qwen3.5, which is relevant because the model being deployed (Qwen3.5-122B-A10B) uses a Mixture-of-Experts architecture.

Step 3 — Design a comprehensive test: The command is carefully constructed to (a) list all Qwen model files in the vLLM models directory, (b) print a separator, (c) check the transformers version, and (d) do a broad recursive search for any file with qwen3_5 or qwen3.5 in its name. The use of find with -o (OR) ensures both naming conventions are covered.

Step 4 — Interpret results: The output shows no qwen3_5_moe.py in the models directory listing, but the find command reveals four Qwen3.5-specific files. The assistant correctly interprets this: the model files exist but are named with underscores (qwen3_5.py) rather than dots (qwen3.5), which is why the earlier ls-based check missed them.

Step 5 — Draw conclusions and act: In the follow-up message ([msg 6656]), the assistant synthesizes the findings: "vLLM 0.17 has Qwen3.5 support files... This is a much better path." The assistant then immediately proceeds to transfer the image to the second Spark node.

The thinking process demonstrates a key engineering skill: not trusting incomplete information. The assistant's earlier check showed no qwen3_5* files, but rather than accepting this negative result, the assistant designed a more thorough verification. This persistence — the willingness to question one's own data — is what ultimately unlocked the deployment.

Broader Significance

This message, while brief, represents a classic inflection point in complex systems engineering. The assistant had been pursuing a SGLang-based approach for hours, investing significant effort in diagnosing NCCL deadlocks, adjusting launch parameters, and waiting for initialization. The pivot to vLLM was not a random switch but a calculated decision based on evidence: the existing GLM deployment on these Spark nodes used vLLM successfully, and a community image specifically targeting Qwen3.5 on GB10 existed.

The verification in this message is the moment where uncertainty collapses into certainty. Before this message, the vLLM path was a hypothesis — the image might not support Qwen3.5, and the assistant would have been back to debugging SGLang or building vLLM from source. After this message, the path is clear: transfer the image, set up the Ray cluster, and launch the model.

In the broader narrative of this opencode session, [msg 6655] is the message that enables everything that follows: the successful multi-node vLLM deployment, the ~27 tok/s throughput, and the correct reasoning output verification. It is a reminder that in complex deployments, the most critical work is often not the final launch command, but the diagnostic and verification steps that confirm the chosen path is viable.