The Elegant Workaround: Verifying Qwen3.5 Support in a GPU-Less Docker Container

Introduction

In the middle of a complex multi-node deployment spanning two NVIDIA DGX Spark systems, the assistant encountered a frustrating but common problem: a Docker image designed for GPU-accelerated inference refused to run any Python commands without a GPU present. The image's entrypoint automatically imported vLLM, which crashed immediately when no NVIDIA driver was detected. Yet the assistant needed to verify that this image—hellohal2064/vllm-qwen3.5-gb10—actually contained the model files necessary to serve Qwen3.5-122B-A10B-FP8 across two nodes. Message <msg id=6654> captures the moment the assistant pivoted from a failing approach to an elegant workaround, using --entrypoint bash to bypass the auto-import and inspect the container's filesystem directly.

This message, while brief, represents a critical inflection point in a much larger deployment saga. It demonstrates the kind of practical debugging skill that separates a working deployment from a stalled one: recognizing when a tool is getting in its own way, and finding a path around the obstacle rather than continuing to fight it.

The Broader Context: A Multi-Node Deployment Under Pressure

To understand why this message matters, we need to step back and appreciate the larger picture. The assistant was in the middle of deploying Qwen3.5-122B-A10B-FP8, a 119-billion-parameter Mixture-of-Experts model, across two DGX Spark nodes connected via InfiniBand RoCE. This is no small feat. The DGX Spark is an ARM-based (Cortex-X925) system with 120GB of unified memory and a single Blackwell GB10 GPU (SM121). To run a 119B model, the assistant needed both nodes working together, splitting the model across two GPUs with tensor parallelism.

The previous attempts using SGLang had failed catastrophically. After multiple rounds of debugging spanning messages <msg id=6627> through <msg id=6647>, the assistant had identified a deadlock in SGLang's NCCL initialization. Both the head and worker nodes would log CustomAllreduce is disabled and then hang indefinitely, never progressing to weight loading. The NCCL channels would be set up over NET/IBext_v11 (InfiniBand), but the process would never complete the communicator initialization. GPU memory usage stayed at a paltry 416MB—far below what would be needed for a 119B model.

After exhausting SGLang options (including trying --disable-custom-all-reduce), the assistant made the strategic decision to pivot to vLLM, which had a proven track record for multi-node inference on DGX Spark systems. A web search in <msg id=6649> revealed the hellohal2064/vllm-qwen3.5-gb10 image, specifically built for Qwen3.5 on GB10 hardware. The assistant pulled this image in <msg id=6650> and then faced the verification problem.

The Verification Problem: A Catch-22

The assistant needed to answer a simple question: does this Docker image actually support the Qwen3.5 model architecture? The image was advertised as "vLLM optimized for NVIDIA DGX Spark GB10 (ARM64, SM120/121, CUDA 13)" and named vllm-qwen3.5-gb10, but the assistant needed to confirm that the vLLM model registry inside the container included Qwen3.5 model classes.

The obvious approach—running a Python command inside the container—failed repeatedly. Messages <msg id=6651> through <msg id=6653> show the assistant trying various incantations:

docker run --rm hellohal2064/vllm-qwen3.5-gb10 python3 -c "
import vllm; print('vLLM:', vllm.__version__)
from vllm.model_executor.models.registry import _VLLM_MODELS
q35 = [k for k in _VLLM_MODELS if '3_5' in k or 'qwen3.5' in k.lower()]
print('Qwen3.5 models:', q35)
"

Each attempt crashed with the same error: the container's default entrypoint (or some auto-import mechanism) would try to initialize vLLM, which would fail because no GPU was available. The error trace showed a cascade through _compute_kwargs, pydantic validation, and ultimately a crash. The assistant was stuck in a catch-22: to verify the image could run on GPUs, it needed to run the image, but running the image required a GPU.

The Insight: Bypassing the Entrypoint

Message <msg id=6654> captures the moment of insight. The assistant wrote:

The vLLM image has some auto-import that's crashing without GPU. Let me use a simpler approach.

The key phrase is "simpler approach." The assistant recognized that the problem was the default entrypoint of the Docker image, which presumably ran vllm serve or some initialization script that imported vLLM. By overriding the entrypoint with --entrypoint bash, the assistant could bypass this entirely and run arbitrary shell commands inside the container without triggering the GPU-dependent import.

The command executed was:

ssh aurora@10.1.230.180 'docker run --rm --entrypoint bash hellohal2064/vllm-qwen3.5-gb10 -c "ls /opt/vllm/vllm/model_executor/models/ 2>/dev/null | grep qwen3 ; find / -path \"*/models/qwen3*\" -name \"*.py\" 2>/dev/null | head -10 ; pip show vllm 2>/dev/null | grep -i version"' 2>&1 | tail -15

This command does three things:

  1. Lists the vLLM model executor directory and filters for qwen3 files
  2. Searches the entire filesystem for Python files matching qwen3* in any models directory
  3. Shows the vLLM version The results were encouraging:
qwen3_next_mtp.py
qwen3_omni_moe_thinker.py
qwen3_vl.py
qwen3_vl_moe.py
/opt/vllm-env/lib/python3.12/site-packages/lmcache/v1/compute/models/qwen3.py
/opt/vllm-env/lib/python3.12/site-packages/transformers/models/qwen3_vl/__init__.py
/opt/vllm-env/lib/python3.12/site-packages/transformers/models/qwen3_vl/processing_qwen3_vl.py
/opt/vllm-env/lib/python3.12/site-packages/transformers/models/qwen3_vl/modeling_qwen3_vl.py
/opt/vllm-env/lib/python3.12/site-packages/transformers/models/qwen3_vl/modula...

The presence of qwen3_next_mtp.py and qwen3_omni_moe_thinker.py in the vLLM model executor directory strongly suggested that this image did include Qwen3.5 support (the "next_mtp" variant being the Multi-Token Prediction version of Qwen3.5). The transformers files confirmed that the HuggingFace transformers library also had Qwen3.5 model classes.

The Thinking Process: A Study in Diagnostic Reasoning

The assistant's thinking process in this message reveals several important cognitive patterns:

Pattern 1: Recognizing failure modes. The assistant immediately identified that the auto-import was the problem, not a deeper issue with the image or the container runtime. This distinction is crucial—it prevented the assistant from going down rabbit holes like "maybe the image is corrupted" or "maybe Docker is broken."

Pattern 2: Knowing Docker internals. The assistant knew about --entrypoint as a way to override the default command. This is a relatively advanced Docker feature that many users don't know about. The assistant understood that the entrypoint was the mechanism causing the crash, and that bypassing it would allow inspection of the container's filesystem without triggering the problematic initialization.

Pattern 3: Multi-tool composition. The assistant composed several diagnostic techniques into a single command: listing directories, searching with find, and checking package versions. This shows an understanding that verification requires multiple angles—a single check might miss something.

Pattern 4: Progressive simplification. Earlier attempts tried to run Python code with GPU-dependent imports. When that failed, the assistant tried running just ls and pip show without Python imports. When even that failed (because the entrypoint ran before any command), the assistant went one level deeper and bypassed the entrypoint entirely. Each step removed one layer of abstraction until reaching the bare minimum needed to get information.

Assumptions Made

The assistant made several assumptions in this message:

  1. The auto-import was in the entrypoint, not in bashrc or profile scripts. If the crash had been triggered by a .bashrc or profile script that ran even with --entrypoint bash, this approach would have failed too. The assistant assumed the default entrypoint was the culprit, which turned out to be correct.
  2. The model files would be in predictable locations. The assistant assumed vLLM's model executor models were under /opt/vllm/vllm/model_executor/models/ and that transformers models would be in the standard site-packages path. These assumptions were based on knowledge of vLLM's codebase structure.
  3. File presence implies functionality. Finding qwen3_next_mtp.py doesn't guarantee that the model loads correctly or that all dependencies are met. The assistant implicitly assumed that if the model class file existed, the image was likely suitable for deployment. This was a reasonable heuristic for a quick check, but not a definitive validation.
  4. The image name matched the content. The image was named vllm-qwen3.5-gb10, and the assistant assumed this was truthful advertising. The file inspection was a sanity check rather than a deep audit.

Input Knowledge Required

To understand and execute this message, the assistant needed:

  1. Docker fundamentals: How --entrypoint works, how container entrypoints differ from commands, and how to override them.
  2. vLLM architecture: Knowledge that vLLM stores model implementations in vllm/model_executor/models/ and that the model registry (_VLLM_MODELS) is populated from these files.
  3. Qwen3.5 model naming conventions: Understanding that Qwen3.5 uses class names like Qwen3_5MoeForConditionalGeneration and that the file names would follow patterns like qwen3_next_mtp.py.
  4. DGX Spark hardware constraints: Understanding that the container would crash without a GPU because vLLM's initialization probes for CUDA devices.
  5. SSH and remote execution: The ability to compose complex commands over SSH, including quoting and escaping for nested shells.
  6. The deployment context: Knowledge that SGLang had failed, that vLLM was the fallback, and that this specific image was the best candidate for multi-node Qwen3.5 inference.

Output Knowledge Created

This message produced several important pieces of knowledge:

  1. Confirmed Qwen3.5 support in the vLLM image. The presence of qwen3_next_mtp.py and qwen3_omni_moe_thinker.py confirmed that this image could serve the target model. This justified the pivot from SGLang to vLLM.
  2. Validated the image as a viable fallback. After SGLang's multi-node NCCL deadlock, this verification gave the assistant confidence to proceed with vLLM deployment.
  3. Established a reusable diagnostic technique. The --entrypoint bash trick for inspecting GPU-dependent containers is a general technique applicable to many similar situations.
  4. Identified the vLLM version indirectly. While the pip show output was truncated in the message, the assistant could infer the vLLM version from the file structure and image metadata.
  5. Mapped the model file locations. The find command revealed where Qwen3.5 model files lived in both vLLM and transformers, which would be useful for any future debugging or patching.

Mistakes and Potential Issues

While the approach was successful, there are some potential issues worth noting:

  1. Incomplete verification. Finding model files doesn't guarantee the model loads correctly. The assistant didn't verify that the model class was actually registered in _VLLM_MODELS or that it could be instantiated. A more thorough check would have required running the container with GPU passthrough.
  2. Truncated output. The tail -15 at the end of the command might have cut off important information, particularly the vLLM version from pip show. The output shown ends with modula... suggesting truncation.
  3. No check for CUDA compatibility. The assistant didn't verify that the vLLM build in the image was compiled for the correct CUDA version (CUDA 13, matching the DGX Spark's Blackwell GPU).
  4. Single-node verification. The assistant verified the image on one node but didn't check that the same image would work identically on the second Spark. In practice, both nodes ran the same image, but subtle differences in driver versions or kernel modules could cause issues.

The Broader Significance

This message is a textbook example of diagnostic minimalism—the art of finding the simplest possible check that answers your question. The assistant could have spent hours trying to fix the GPU-less import crash, patching the entrypoint, or building a custom test harness. Instead, it recognized that the question "does this image support Qwen3.5?" could be answered by looking at files on disk, without ever running the application.

This approach is particularly valuable in production debugging scenarios where:

Conclusion

Message <msg id=6654> is a small but pivotal moment in a complex multi-node deployment. Faced with a Docker image that refused to cooperate without a GPU, the assistant used --entrypoint bash to bypass the problematic auto-import and inspect the container's filesystem directly. The verification succeeded, confirming that the hellohal2064/vllm-qwen3.5-gb10 image contained the necessary Qwen3.5 model files and paving the way for a successful vLLM-based deployment across two DGX Spark nodes.

The message exemplifies several hallmarks of expert debugging: recognizing when a tool is the obstacle rather than the solution, knowing the right workaround for the specific failure mode, and composing multiple diagnostic techniques into a single efficient command. It's a reminder that in complex systems engineering, the most valuable skill is often not knowing the answer, but knowing how to ask the question in a way that the system can answer.