The Breakthrough Moment: Validating Qwen3.5-122B-A10B-FP8 on a Single DGX Spark

Introduction

In the complex world of multi-node large language model deployment, success often arrives not with a bang but with a quiet log line confirming that the model architecture has been correctly identified. Message <msg id=6665> captures exactly such a moment — a turning point in a grueling session where the assistant had been fighting deadlocked NCCL initialization, incompatible Docker images, and networking configuration issues while attempting to deploy the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark systems. This message is the assistant's report that a single-node validation test has succeeded, proving that the chosen vLLM Docker image can correctly load and begin serving this large MoE model on the DGX Spark's GB10 (SM121 Blackwell) hardware.

The Message: A Victory Lap and a Cautionary Note

The message opens with an emphatic declaration:

It's working! vLLM 0.17 resolved the architecture to Qwen3_5MoeForConditionalGeneration and is now loading the model on single-node! It detected FP8 quantization and is using TRITON for MoE, FLASHINFER for attention.

This is the culmination of a long debugging chain. The assistant had spent multiple previous rounds trying to get SGLang working in a multi-node configuration, only to find it deadlocked repeatedly at the NCCL channel initialization phase. The pivot to vLLM — specifically to the hellohal2064/vllm-qwen3.5-gb10 Docker image, which was built specifically for Qwen3.5 on DGX Spark hardware — was a strategic decision that paid off immediately.

The message then provides a critical piece of technical context about the image's default configuration:

The image uses --max-model-len 8192 by default (its own entrypoint). This single-node test will likely OOM since the FP8 model is ~119GB and the node only has 120GB — but it proves the model/image combo works.

This sentence reveals the assistant's clear-headed understanding of the constraints. The DGX Spark has 120GB of unified memory (shared between CPU and GPU). The FP8 quantized model is approximately 119GB. With a default context length of 8192 tokens and GPU memory utilization at 85%, an Out Of Memory (OOM) error is almost certain on a single node. But that is not the point of this test. The point is to validate that the model architecture is correctly recognized, that the quantization format is properly detected, and that the engine initializes without crashing. These are necessary conditions for the multi-node deployment that will follow.

The message concludes with a bash command that sleeps for 60 seconds and then tails the Docker logs, and the output confirms that the engine has indeed started initializing successfully:

(APIServer pid=1) INFO 04-08 23:16:38 [compilation.py:289] Enabled custom fusions: norm_quant, act_quant
(EngineCore_DP0 pid=96) INFO 04-08 23:16:45 [core.py:101] Initializing a V1 LLM engine (v0.17.1rc1.dev25+g1a629ce0d) with config: model='/models/model', speculative_config=None, tokenizer='/models/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, ...

These log lines are deeply meaningful to someone familiar with the vLLM inference stack. The Enabled custom fusions: norm_quant, act_quant line confirms that the FP8 quantization-aware kernel fusions are active. The engine initialization line confirms the version (v0.17.1rc1, a very recent release candidate), the data type (bfloat16 for the non-quantized parts), and the maximum sequence length (8192). The model path /models/model shows that the Docker image's entrypoint script has mapped the model correctly.

The Strategic Pivot: Why This Message Matters

To understand why this message is significant, one must appreciate the context that preceded it. The assistant had been attempting to deploy this model using SGLang, a popular inference engine that had been used successfully in earlier parts of the session for other models. However, SGLang's multi-node NCCL initialization kept hanging at the same point — after channel setup and the "CustomAllreduce is disabled" message, the process would stall indefinitely. Multiple attempts with different configurations (including --disable-custom-all-reduce) failed to resolve the issue.

The decision to pivot to vLLM was not made lightly. The assistant had to:

  1. Discover the right image: Through web search, the assistant found hellohal2064/vllm-qwen3.5-gb10, a community-maintained Docker image specifically built for Qwen3.5 on DGX Spark hardware.
  2. Verify the image contents: The assistant inspected the image to confirm it contained Qwen3.5 model support files (qwen3_5.py, qwen3_5_mtp.py, qwen3_5_moe.py in the transformers_utils configs).
  3. Transfer the image to the second node: Using docker save piped over SSH to docker load, the assistant ensured both nodes had the image available.
  4. Test single-node first: Rather than jumping straight to the multi-node configuration, the assistant wisely chose to validate the basic model loading on a single node first. This last decision — test single-node before multi-node — is a textbook debugging strategy. By reducing the problem space to a single GPU, the assistant eliminated the entire Ray cluster coordination, NCCL inter-node communication, and networking complexity that had plagued the SGLang attempts. If the model failed to load on a single node, there would be no point attempting multi-node at all.

Assumptions and Reasoning Visible in the Message

The message reveals several important assumptions and reasoning patterns:

Assumption 1: The model will OOM on single-node. The assistant states this explicitly: "This single-node test will likely OOM since the FP8 model is ~119GB and the node only has 120GB." This is a reasonable assumption — with only 1GB of headroom, the KV cache overhead alone would likely exceed available memory. However, the assistant treats this as acceptable because the test's purpose is validation, not production serving.

Assumption 2: The image's entrypoint script handles the model correctly. The assistant notes that the image uses --max-model-len 8192 by default and maps the model to /models/model. This was discovered through earlier trial and error — the first attempt to run the container failed because the assistant passed vllm serve arguments directly, which conflicted with the image's own entrypoint script. The correction was to use environment variables (VLLM_MAX_MODEL_LEN, VLLM_GPU_MEMORY_UTIL, etc.) instead of command-line arguments.

Assumption 3: The log output confirms correct model architecture detection. The assistant interprets the engine initialization log line as proof that "vLLM 0.17 resolved the architecture to Qwen3_5MoeForConditionalGeneration." This is an inference from the fact that the engine started loading without throwing an architecture-not-found error — earlier attempts with older vLLM versions (like the 0.14 image that came pre-installed on the Spark) failed because they lacked the Qwen3.5 model class.

Assumption 4: Multi-node will work if single-node works. The assistant's plan is to "move to the dual-node setup" after this single-node validation. This assumes that the multi-node Ray-based infrastructure (which the assistant plans to build using vLLM's Ray integration) will work correctly once the basic model loading is confirmed. This is a reasonable assumption given that vLLM's multi-node support is more mature than SGLang's on this hardware.

Input Knowledge Required to Understand This Message

To fully grasp the significance of this message, a reader needs knowledge of:

  1. The DGX Spark hardware platform: Understanding that it uses NVIDIA GB10 (SM121 Blackwell) with 120GB unified memory, ARM Cortex-X925 CPU, and InfiniBand RoCE networking is essential to understanding the memory constraints and the NCCL networking issues.
  2. The Qwen3.5-122B-A10B-FP8 model: This is a 122-billion parameter Mixture-of-Experts model with 10B active parameters per token, quantized to FP8. The FP8 quantization reduces the model size to approximately 119GB, making it barely fit in the DGX Spark's 120GB unified memory.
  3. The vLLM inference engine: Understanding that vLLM uses a V1 engine architecture, supports FP8 quantization with custom kernel fusions (norm_quant, act_quant), and uses Ray for multi-node orchestration.
  4. The NCCL and distributed computing concepts: The earlier failures were due to NCCL deadlocks during multi-node initialization, which requires understanding of NCCL communicator groups, channel setup, and the CustomAllreduce optimization.
  5. Docker container management: The assistant uses Docker extensively, including image pulling, container lifecycle management, SSH-based image transfer between nodes, and environment variable configuration.

Output Knowledge Created by This Message

This message creates several important pieces of knowledge:

  1. Confirmation that the hellohal2064/vllm-qwen3.5-gb10 image works: The community-maintained Docker image correctly identifies and loads the Qwen3.5-122B-A10B-FP8 model on DGX Spark hardware.
  2. Validation of the FP8 quantization pipeline: The log line Enabled custom fusions: norm_quant, act_quant confirms that vLLM's FP8-aware kernel optimizations are active and compatible with the SM121 Blackwell architecture.
  3. Documentation of the correct launch configuration: The message implicitly documents that the image expects the model at /models/model and is configured via environment variables (VLLM_MAX_MODEL_LEN, VLLM_GPU_MEMORY_UTIL, etc.) rather than command-line arguments.
  4. A baseline for multi-node expectations: The single-node test establishes that the model loads correctly, setting the stage for the multi-node deployment that will follow in subsequent messages.

The Thinking Process: What the Message Reveals

The assistant's thinking process is visible in the structure of the message itself. The opening exclamation — "It's working!" — conveys relief and excitement after a long debugging session. But the assistant immediately follows this emotional release with precise technical analysis: the architecture resolution, the quantization detection, the backend selection (TRITON for MoE, FLASHINFER for attention).

The next paragraph shows the assistant's ability to hold two competing thoughts simultaneously: the test is working (positive), but it will likely OOM (negative). The assistant doesn't treat the OOM as a failure — instead, it reframes it as a successful validation: "it proves the model/image combo works." This reframing demonstrates a sophisticated understanding of what constitutes a meaningful test result.

The final bash command and log output serve as evidence. The assistant doesn't just claim success — it shows the log lines that prove it. This is characteristic of the assistant's thorough approach throughout the session: every claim is backed by observable evidence.

Conclusion

Message <msg id=6665> is a pivotal moment in a complex multi-node deployment session. It represents the successful validation of a critical hypothesis — that the Qwen3.5-122B-A10B-FP8 model can be loaded and served on DGX Spark hardware using the community-maintained vLLM Docker image. While the single-node test was expected to OOM due to the tight memory constraints (119GB model on 120GB hardware), it proved that the architecture detection, quantization pipeline, and engine initialization all function correctly. This validation was the necessary precondition for the multi-node deployment that would follow, and the message captures both the technical achievement and the strategic reasoning behind the test.