The Compatibility Check That Defined a Deployment Strategy

In the sprawling, multi-threaded narrative of deploying large language models across heterogeneous hardware, there are moments that pivot on a single question: will this software serve this model? Message [msg 6560] in this opencode session captures exactly such a moment. It is a brief but consequential exchange — two bash commands, a summary of findings, and a critical negative result — that determines the entire trajectory of deploying Qwen3.5-122B-A10B-FP8 across two NVIDIA DGX Spark systems. This message is the hinge point where the assistant transitions from exploration to decision, and where a seemingly promising path is closed off, forcing a strategic pivot.

The Context: Two Sparks, One Model, Many Unknowns

To understand why this message was written, one must appreciate the situation that preceded it. The user had tasked the assistant with deploying the Qwen3.5-122B-A10B-FP8 model — a 125-billion-parameter mixture-of-experts architecture with FP8 quantization — across two NVIDIA GB10 DGX Spark systems. These are unusual machines: ARM-based (Cortex-X925), each with 120GB of unified memory shared between CPU and GPU, connected via InfiniBand (RoCE) interconnect. They are not standard server hardware, and inference frameworks have only recently begun supporting them.

The assistant had spent several messages exploring both nodes ([msg 6555], [msg 6556], [msg 6557], [msg 6558]). It discovered that the head Spark (10.1.230.180) was running a Docker-based vLLM setup serving GLM-4.7-Flash-NVFP4, with a custom vllm-node Docker image. The second Spark (192.168.200.13) was reachable through a jump host and had its own vLLM instance. The assistant had also found a spark-vllm-docker/launch-cluster.sh script suggesting prior multi-node deployment work. The natural inclination was to reuse this existing infrastructure — after all, it was already working, already configured for multi-node, and already had the right CUDA and PyTorch versions.

But a critical question remained unanswered: does this vLLM version support the Qwen3.5 architecture? The existing Docker image contained vLLM 0.14.0rc2.dev262 — a relatively recent nightly build. But Qwen3.5 is a new model family, and support for it may require bleeding-edge code. The assistant needed to verify compatibility before committing to this deployment path.

The Message: A Systematic Compatibility Probe

Message [msg 6560] opens with a structured summary of the assistant's current knowledge:

Model: Qwen/Qwen3.5-122B-A10B-FP8 — 125B total params, FP8 = ~125GB on disk. safetensors shows F8_E4M3: 123B params = ~123GB Docker vLLM: v0.14.0rc2.dev262 with transformers 5.0.0rc3, torch 2.9.1+cu130 — this is quite recent Qwen3.5 requires: vLLM from main branch or recent nightly. v0.14 might work but let me check

This summary is itself a product of earlier investigation. The assistant had already fetched the HuggingFace model page ([msg 6559]) and inspected the Docker image's Python environment. The model size estimate of ~125GB for FP8 is crucial: with 120GB unified memory per node and tensor parallelism across two nodes, each node would need to hold ~62.5GB of parameters, leaving ~57.5GB for KV cache and overhead. This is tight but feasible — provided the software stack supports the model.

The assistant then executes two bash commands in parallel. The first attempts to find Qwen3-related model classes in vLLM's model registry:

docker exec vllm_node python3 -c "from vllm.model_executor.models import _MODELS; print([k for k in _MODELS if \"qwen3\" in k.lower() or \"Qwen3\" in k])"

This command returns no output — a null result that is itself informative. The second command is more thorough, trying three different approaches to find Qwen3.5 support:

  1. It first tries vllm.model_executor.models.registry._VLLM_MODELS (the newer registry)
  2. Then falls back to vllm.model_executor.models._MODELS (the older registry)
  3. Finally attempts a direct import of vllm.model_executor.models.qwen3_5_moe The output is definitive:
_VLLM_MODELS: ['Qwen3NextForCausalLM', 'Qwen3ForCausalLM', 'Qwen3MoeForCausalLM', 'Qwen3OmniMoeForConditionalGeneration', 'Qwen3VLForConditionalGeneration', 'Qwen3VLMoeForConditionalGeneration', 'Eagle3Qwen3vlForCausalLM', 'Qwen3NextMTP']
qwen3_5_moe: No module named 'vllm.model_executor.models.qwen3_5_moe'

This is the critical finding. The vLLM 0.14 nightly supports many Qwen3 variants — Qwen3ForCausalLM, Qwen3MoeForCausalLM, Qwen3VLForConditionalGeneration, etc. — but not Qwen3_5MoeForConditionalGeneration, which is the specific architecture needed for the Qwen3.5-122B-A10B model. The direct import confirms this: the module simply does not exist in this version.

The Reasoning: What This Means and What Comes Next

The assistant's reasoning, visible in the message's structure, reveals a careful, methodical approach. The summary line "Qwen3.5 requires: vLLM from main branch or recent nightly. v0.14 might work but let me check" shows an awareness that model support is a moving target. The assistant is not assuming compatibility based on version numbers alone — it is empirically verifying.

The decision to check both the registry and attempt a direct import is a good example of defensive probing. The registry query might miss models registered under non-standard names, while the direct import catches any module that exists but isn't properly registered. By trying both approaches, the assistant minimizes the chance of a false negative.

The negative result has immediate implications. The existing Docker-based vLLM setup cannot serve Qwen3.5-122B-A10B-FP8. This means the assistant cannot simply reuse the spark-vllm-docker infrastructure. It must either:

  1. Build a newer vLLM that includes Qwen3.5 support, potentially from source
  2. Switch to SGLang, which the NVIDIA Spark guide mentions and which may have more recent model support
  3. Find a pre-built Docker image specifically for Qwen3.5 on GB10 The assistant's next moves (visible in subsequent messages) show it pursuing option 3 — discovering the hellohal2064/vllm-qwen3.5-gb10 image on Docker Hub, which is a vLLM 0.17.1rc1 build specifically compiled for Qwen3.5 on GB10. This pivot is a direct consequence of the negative result in this message.

Assumptions and Their Validity

The message operates on several assumptions, most of which are sound:

Assumption 1: The model registry is authoritative. The assistant assumes that if a model class is not in _VLLM_MODELS, the framework cannot serve it. This is generally correct for vLLM, though edge cases exist where models are loaded via custom configurations. The direct import test mitigates this risk.

Assumption 2: The Docker image is the only vLLM installation. The assistant checks only the Docker container's vLLM, not any host-side installation. This is reasonable because the existing deployment uses Docker, and the launch-cluster.sh script references the Docker image. However, it does mean the assistant might miss a host-side vLLM that could be used instead.

Assumption 3: Qwen3.5 requires a specific model class. The assistant searches for Qwen3_5MoeForConditionalGeneration specifically. This is correct — the Qwen3.5-122B-A10B is a MoE architecture that requires its own model implementation. It cannot be served using Qwen3MoeForCausalLM because the architecture differs (e.g., hybrid GDN attention mechanisms).

Assumption 4: The model size estimate is accurate. The assistant estimates ~125GB for FP8 based on 125B parameters at 1 byte per parameter (FP8). This is a reasonable approximation, though real-world model sizes can vary due to embedding tables, norm parameters, and other non-weight components. The safetensors metadata showing F8_E4M3: 123B params confirms this is close.

One subtle assumption worth examining: the assistant assumes that the absence of qwen3_5_moe in vLLM 0.14 means the model cannot be served at all. In theory, one could attempt to load the model using a generic MoE configuration or with custom model code. In practice, this is unlikely to work for a complex architecture like Qwen3.5, so the assumption is pragmatic.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. Knowledge of vLLM's model registration system: The distinction between _VLLM_MODELS, _MODELS, and direct module imports. The assistant's multi-pronged approach reflects an understanding that model registration can happen at multiple levels.
  2. Awareness of the Qwen model family hierarchy: Qwen3, Qwen3.5, and their variants (MoE, VL, Omni, etc.). The assistant knows that Qwen3_5MoeForConditionalGeneration is a distinct architecture from Qwen3MoeForCausalLM, even though both are MoE models.
  3. Understanding of FP8 memory requirements: The ability to estimate model size from parameter count and quantization format. 125B parameters × 1 byte/parameter (FP8) = 125GB.
  4. Context about the DGX Spark hardware: 120GB unified memory, ARM architecture, SM121 compute capability. This context explains why the assistant is concerned about memory margins.
  5. Knowledge of the existing infrastructure: The Docker-based vLLM setup, the spark-vllm-docker directory, and the multi-node launch script. This context was built over the preceding messages.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Definitive incompatibility: vLLM 0.14.0rc2.dev262 does not support Qwen3.5-122B-A10B-FP8. This is the primary output and the most actionable finding.
  2. Supported model list: The existing vLLM supports Qwen3NextForCausalLM, Qwen3ForCausalLM, Qwen3MoeForCausalLM, Qwen3OmniMoeForConditionalGeneration, Qwen3VLForConditionalGeneration, Qwen3VLMoeForConditionalGeneration, Eagle3Qwen3vlForCausalLM, and Qwen3NextMTP. This is useful context — if the user later wants to deploy a different Qwen3 variant, the existing infrastructure would work.
  3. Model size validation: The FP8 model is confirmed at ~123B parameters (from safetensors metadata), close to the 125B estimate. This validates the memory planning.
  4. Decision point established: The assistant now has the information needed to choose a deployment strategy. The existing path is blocked; alternatives must be explored.

The Thinking Process: A Window into Methodical Debugging

The reasoning in this message reveals a structured, hypothesis-driven approach. The assistant forms a hypothesis ("v0.14 might work") and then designs experiments to test it. The experiments are not random — they are targeted at the specific mechanism by which vLLM discovers and loads model architectures.

The first bash command is a quick probe: check the older _MODELS registry. When this returns nothing (not even an error message — the output is simply absent), the assistant doesn't assume the model isn't there. Instead, it runs a more comprehensive second command that checks three separate paths. This is a good example of increasingly specific probing: start broad, then narrow down.

The choice to check _VLLM_MODELS (the newer registry) before _MODELS (the older one) shows awareness of vLLM's code evolution. The assistant knows that newer versions may have moved to a different registration system. The fallback to the older registry and then to direct import creates a complete coverage of possible model discovery mechanisms.

The direct import test is particularly insightful. Even if a model isn't in the registry, it might still be importable — perhaps it was added but not registered, or registered under a different name. The ImportError confirms the module truly doesn't exist in this version.

Mistakes and Incorrect Assumptions

The message is largely correct, but there are minor points worth examining:

The model size estimate of ~125GB is slightly high. The safetensors metadata shows F8_E4M3: 123B params, meaning the actual parameter count is 123B, not 125B. The assistant's summary says "125B total params" but the safetensors data says 123B. This 2B discrepancy is minor but could affect memory planning at the margins.

The assistant assumes the Docker container's vLLM is the only option. It doesn't check whether a host-side vLLM installation exists or whether the Docker image could be extended. In fairness, building a new Docker image on an ARM system with limited internet access (the second Spark has no direct internet) would be challenging, so the assumption is pragmatic.

The message doesn't check vLLM's model configuration system. vLLM can sometimes load models using a generic configuration if the model's architecture is close enough to a supported one. The assistant doesn't attempt to load the model with --model to see what error occurs. This would be a more direct test but would require downloading the model first, which is expensive (125GB download). The registry check is a reasonable proxy.

Conclusion: A Small Message with Large Consequences

Message [msg 6560] is a textbook example of a compatibility gate — a check that determines whether an entire deployment strategy is viable. In just two bash commands and a structured summary, the assistant establishes that the existing vLLM infrastructure cannot serve the target model, forcing a pivot to alternative approaches. The message is methodical, defensive, and empirically grounded. It avoids the common pitfall of assuming compatibility based on version numbers or superficial similarities, instead probing the actual model registration mechanisms.

This message also demonstrates the value of negative results in engineering work. The finding that Qwen3.5 is not supported is just as valuable as finding that it is — it prevents wasted effort trying to make an incompatible setup work, and it redirects effort toward viable alternatives. In the broader narrative of this deployment, message [msg 6560] is the moment when one path closes and another opens, setting the stage for the successful deployment using the specialized hellohal2064/vllm-qwen3.5-gb10 image that follows.