The Pivot: How a Failed SGLang Deployment Led to the Discovery of hellohal2064/vllm-qwen3.5-gb10

Introduction

In the sprawling narrative of deploying Qwen3.5-122B-A10B-FP8 across two NVIDIA DGX Spark systems, there comes a moment that epitomizes the difference between rigid adherence to a plan and adaptive problem-solving in real-world ML infrastructure. Message [msg 6649] captures this moment perfectly: a brief but decisive pivot from a failing SGLang multi-node deployment to a vLLM-based alternative, triggered by a web search that uncovers a purpose-built Docker image. This message, though only a few lines of reasoning followed by search results, represents the culmination of hours of debugging, the recognition of a dead end, and the birth of a new strategy that would ultimately succeed.

The Message in Full

The assistant writes:

vLLM 0.14 already has transformers 5.0 but no Qwen3.5 model support in the registry. The model class Qwen3_5MoeForConditionalGeneration was added later (after 0.14).

>

Let me check the scitrera vLLM image — they might have a newer version:

>

[exa_web_search_exa] {"numResults":5,"query":"scitrera dgx-spark-vllm Docker image latest Qwen3.5 support 2026"}

>

Title: hellohal2064/vllm-qwen3.5-gb10 - Docker Image URL: https://hub.docker.com/r/hellohal2064/vllm-qwen3.5-gb10 Published: N/A Author: N/A Highlights: hellohal2064/vllm-qwen3.5-gb10 - Docker Image ... vLLM optimized for NVIDIA DGX Spark GB10 (ARM64, SM120/121, CUDA 13)

This message is a turning point. It is the moment the assistant abandons SGLang for multi-node inference and embraces vLLM, a decision that would ultimately lead to a successful deployment.

Context: The SGLang Dead End

To understand why this message was written, one must appreciate the ordeal that preceded it. The assistant had been attempting to deploy Qwen3.5-122B-A10B-FP8—a massive 119GB Mixture-of-Experts model—across two DGX Spark nodes, each equipped with a single NVIDIA GB10 GPU (SM121 Blackwell architecture, ARM Cortex-X925 CPU, 120GB unified memory). The two nodes were connected via InfiniBand RoCE (RDMA over Converged Ethernet), providing a high-speed interconnect for tensor parallelism.

The initial approach used SGLang, a popular inference framework known for its performance optimizations. The assistant had built a custom launch script, configured NCCL for InfiniBand transport, and successfully formed a distributed process group—both nodes reported CustomAllreduce is disabled because this process group spans across nodes, confirming they had found each other. But then, nothing. The logs showed NCCL channel setup completing—channels 00 through 07 establishing send/receive pairs over NET/IBext_v11—but the process never progressed to weight loading. GPU memory usage remained at a paltry 416MB per GPU, far below what loading a 119GB model would require.

The assistant tried multiple remedies: adding NCCL debug logging, adjusting GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME, setting --disable-custom-all-reduce, and waiting increasingly long periods (60 seconds, 120 seconds, 180 seconds, 300 seconds). Each time, the result was the same: a deadlock after NCCL communicator initialization, with the last log line frozen at CustomAllreduce is disabled.

The root cause was never definitively identified, but the assistant's analysis pointed to a second NCCL communicator group creation hanging—possibly a Gloo backend group or a pipelining group that SGLang creates internally. This is a known issue with SGLang on multi-node DGX Spark configurations, as hinted at by forum discussions about enable_flashinfer_allreduce_fusion causing crashes.

The Reasoning Behind the Pivot

Message [msg 6649] begins with the assistant processing the results of a quick test on the existing vLLM Docker image (vllm-node). The assistant had just run:

docker run --rm vllm-node python3 -c "
import vllm; print(\"vLLM:\", vllm.__version__)
import torch; print(\"PyTorch:\", torch.__version__)
import transformers; print(\"Transformers:\", transformers.__version__)
# Check for Qwen3_5
try:
    from vllm.model_executor.models.registry import _VLLM_MODELS
    q35 = [k for k in _VLLM_MODELS if \"3_5\" in k or \"3.5\" in k]
    print(\"Qwen3.5 in registry:\", q35)
except Exception as e:
    print(f\"Error: {e}\")
"

This revealed vLLM 0.14.0rc2 with transformers 5.0.0rc3, but critically, no Qwen3.5 model in the registry. The model class Qwen3_5MoeForConditionalGeneration was added to vLLM after version 0.14, meaning this existing image could not serve the model.

The assistant's first sentence in the message—"vLLM 0.14 already has transformers 5.0 but no Qwen3.5 model support in the registry"—is a concise summary of this finding. It demonstrates the assistant's understanding of vLLM's model registration system: each model architecture must be explicitly registered in _VLLM_MODELS for the engine to recognize and load it. Transformers 5.0 might have the model class, but without the vLLM integration code, it's useless for inference.

The second sentence—"Let me check the scitrera vLLM image—they might have a newer version"—reveals the assistant's strategic thinking. Rather than attempting to patch the existing vLLM image (which would require rebuilding from source, a time-consuming process on ARM), the assistant reasons that someone else in the DGX Spark community has likely already solved this problem. The mention of "scitrera" is a reference to a known community member who maintains Docker images for the DGX Spark platform.

The Web Search and Its Discovery

The assistant then executes an exa_web_search_exa tool call with the query: "scitrera dgx-spark-vllm Docker image latest Qwen3.5 support 2026". This is a remarkably well-crafted search query. It specifies:

Assumptions and Their Validity

This message rests on several key assumptions:

Assumption 1: The SGLang approach is fundamentally blocked. The assistant assumes that the NCCL deadlock is not a transient issue or a configuration error that can be fixed with more tuning. After multiple attempts with different configurations, NCCL debug levels, and wait times, this assumption is well-supported by evidence. The deadlock is reproducible and consistent.

Assumption 2: A vLLM-based approach will not have the same NCCL issue. This is a significant assumption. The NCCL deadlock could be a system-level issue (e.g., InfiniBand configuration, firmware bug, or ARM-specific NCCL behavior) that would affect any framework. However, the assistant's experience suggests the issue is specific to SGLang's multi-node initialization sequence, and vLLM's different initialization path might avoid it.

Assumption 3: The hellohal2064 image is trustworthy and functional. The assistant has no prior experience with this image. It could contain outdated dependencies, incorrect CUDA configurations, or even security issues. The assistant implicitly trusts that a community-maintained image with 2.4K pulls is reliable enough to try.

Assumption 4: The image supports the specific model (Qwen3.5-122B-A10B-FP8). The image name includes "qwen3.5" and "gb10", but the FP8 variant of the 122B model might require specific quantization kernels or memory configurations that the image may not support.

These assumptions are reasonable given the context. The assistant is operating under time pressure (the user is waiting for a working deployment) and has exhausted the SGLang approach. The risk of trying a community image is low compared to the cost of building from source or continuing to debug SGLang.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of vLLM's model registry system: Understanding that _VLLM_MODELS is a registry dict that maps model architecture names to their implementation classes, and that Qwen3.5 support requires both the HuggingFace transformers model class AND the vLLM integration code.
  2. Knowledge of the DGX Spark ecosystem: Understanding that the DGX Spark (formerly Project DIGITS) is an ARM64-based desktop supercomputer with a GB10 Blackwell GPU, and that its unique architecture requires specially compiled software.
  3. Knowledge of the SGLang multi-node deadlock: The preceding messages establish that SGLang's NCCL initialization hangs after channel setup, a problem that the assistant has spent considerable effort debugging.
  4. Knowledge of the scitrera community: Understanding that "scitrera" is a known entity in the DGX Spark community who maintains custom Docker images for vLLM and SGLang.
  5. Knowledge of the model: Qwen3.5-122B-A10B-FP8 is a Mixture-of-Experts model with 122B total parameters, 10B active parameters per token, using FP8 quantization. It requires significant engineering to deploy efficiently.

Output Knowledge Created

This message produces several valuable outputs:

  1. A confirmed negative result: The existing vLLM 0.14 image on the DGX Spark does NOT support Qwen3.5. This prevents wasted effort trying to use it.
  2. A discovered resource: The hellohal2064/vllm-qwen3.5-gb10 Docker image, which is specifically built for this use case. This becomes the foundation for the successful deployment.
  3. A strategic decision: The pivot from SGLang to vLLM for multi-node inference. This decision shapes the remainder of the deployment effort.
  4. A reasoning chain: The assistant's thought process—from recognizing the SGLang dead end, to testing the existing vLLM image, to searching for a community alternative—provides a template for troubleshooting similar infrastructure problems.

The Thinking Process

The message reveals a sophisticated reasoning process compressed into a few lines. The assistant is operating at multiple levels simultaneously:

At the diagnostic level, the assistant has correctly interpreted the vLLM registry check. The absence of Qwen3.5 in _VLLM_MODELS is not a bug or configuration issue—it's a fundamental capability gap. The assistant recognizes that adding support would require code changes, not just configuration.

At the strategic level, the assistant evaluates the options: (a) build vLLM from source with Qwen3.5 patches, (b) attempt to patch the existing image, or (c) find a pre-built image. Option (a) could take hours on ARM, option (b) is fragile and might miss dependencies, and option (c) is the fastest if a suitable image exists.

At the research level, the assistant formulates a precise search query targeting the scitrera repository. The query includes the platform, the model, the technology, and a temporal constraint—all in six words. This is the mark of someone who knows exactly what they're looking for.

At the evaluation level, the assistant processes the search result. The image hellohal2064/vllm-qwen3.5-gb10 is not from scitrera but from hellohal2064—a different maintainer. The assistant doesn't dismiss it; instead, the description ("vLLM optimized for NVIDIA DGX Spark GB10 (ARM64, SM120/121, CUDA 13)") clearly matches the requirements. The assistant implicitly accepts this as a viable alternative.

Mistakes and Incorrect Assumptions

While the message is well-reasoned, there are potential pitfalls:

The assumption that vLLM will work where SGLang failed is the most significant risk. Both frameworks use NCCL for inter-node communication, and if the NCCL deadlock is caused by a system-level issue (e.g., InfiniBand firmware, ARM-specific NCCL behavior, or the SEV-SNP IOMMU configuration that caused problems in earlier segments), vLLM would face the same problem.

The trust in a community image is another risk. The hellohal2064/vllm-qwen3.5-gb10 image has 2.4K pulls, which is modest. It could contain incorrect CUDA version mismatches, outdated kernel modules, or other incompatibilities that manifest only at runtime.

The assumption that "Qwen3.5 support" in the image name means FP8 support is also uncertain. The image might support the BF16 variant of Qwen3.5-122B but not the FP8 quantized version, which requires specific CUDA kernels for FP8 matrix multiplication on Blackwell.

However, these risks are managed by the assistant's methodology: the next step would be to test the image, and if it fails, the assistant would have gained information about what doesn't work, narrowing the search space.

Conclusion

Message [msg 6649] is a masterclass in pragmatic infrastructure engineering. It captures the moment when a seasoned practitioner recognizes a dead end, rapidly evaluates alternatives, and pivots to a new approach—all in the span of two sentences and a web search. The message is deceptively simple, but it represents the convergence of deep technical knowledge (vLLM's model registry, NCCL initialization, the DGX Spark ecosystem), strategic thinking (evaluating build-from-source vs. pre-built image), and research skills (crafting the right search query).

This message is the hinge point of the entire multi-node deployment effort. Before it, the assistant was locked in a losing battle with SGLang's NCCL deadlock. After it, the assistant would successfully deploy Qwen3.5-122B-A10B-FP8 using the hellohal2064 image, achieving ~27 tok/s single-request throughput with correct reasoning output. The pivot was not just a change of tools—it was a change of strategy, from "make SGLang work" to "find what already works," and it paid off.