The Dead End at the Docker Registry: Diagnosing Qwen3.5 Support on DGX Spark

In the sprawling infrastructure of a multi-node AI deployment, the most critical decisions often happen in the quiet moments between failures. Message 6573 of this coding session captures exactly such a moment — a brief, technically dense diagnostic that reveals the contours of a fundamental problem: none of the available inference containers support the model you need to run.

The message is short, almost perfunctory. The assistant writes:

Still no Qwen3.5 support — SGLang version shows 0.0.0 (editable install without proper version) and transformers 4.57.6. Qwen3.5 needs transformers >=5.x. Let me check what specific SGLang commit this is based on, and whether we can just upgrade transformers inside it:

>

[bash] ssh aurora@10.1.230.180 'docker run --rm scitrera/dgx-spark-sglang:0.5.10rc0 python3 -c " import sglang.srt.models as m import os # Check what models exist model_dir = os.path.dirname(m.__file__) qwen_models = [f for f in os.listdir(model_dir) if \"qwen\" in f.lower()] print(\"Qwen models:\", sorted(qwen_models)) " 2>&1 | grep -v "WARNING\|NOTE\|==\|NVIDIA\|Copyright\|Various\|GOVERNING\|found at\|docker run"' 2>&1

The output reveals a Python traceback: TypeError: expected str, bytes or os.PathLike object, not NoneType. The module's __file__ attribute is None, a telltale sign of an editable install (pip install -e .) where the package hasn't been properly built. The assistant's attempt to enumerate model files by directory inspection fails, but the failure itself is informative.

The Context That Made This Message Necessary

To understand why this message matters, we must trace the path that led here. The session's overarching goal is deploying Qwen3.5-122B-A10B-FP8, a 125-billion-parameter hybrid MoE (Mixture-of-Experts) model quantized to FP8, across two NVIDIA DGX Spark nodes connected by InfiniBand. Each Spark has a single GB10 Blackwell GPU (SM121 architecture) with ~120GB of unified memory. The model is too large for one node — it needs tensor parallelism across both.

The assistant had already discovered, in [msg 6560], that the existing vLLM Docker image on the Spark (vllm-node, v0.14.0rc2) does not support Qwen3.5. The model registry showed Qwen3MoeForCausalLM and Qwen3ForCausalLM but no Qwen3_5MoeForConditionalGeneration. This was the first major roadblock.

The user then instructed the assistant to stop the GLM model running on the head Spark ([msg 6561]), freeing GPU memory for the new deployment. The assistant methodically stopped the vllm-cluster, vllm-proxy, vllm-embeddings, and vllm-reranker services ([msg 6565], [msg 6566]), clearing the GPU.

With the existing vLLM image ruled out, the assistant turned to SGLang as an alternative. It pulled the official lmsysorg/sglang:spark image ([msg 6569]) and discovered it was SGLang 0.5.4 with transformers 4.57 — too old for Qwen3.5, which requires transformers >=5.x ([msg 6571]). The assistant then found a community image, scitrera/dgx-spark-sglang:0.5.10rc0, which was only six days old and promised a much newer SGLang version. This seemed like the answer.

Message 6572 tested that image. The results were ambiguous: the SGLang version showed 0.0.0 (an editable install), PyTorch was 2.11.0a0, and transformers was 4.57.6 — still too old. The critical import test — from sglang.srt.models.qwen3_5_moe import Qwen3_5MoeForCausalLM — failed with an ImportError. Qwen3.5 was not supported.

The Reasoning in Message 6573

This brings us to the subject message. The assistant faces a puzzle: the scitrera/dgx-spark-sglang:0.5.10rc0 image is billed as a Qwen3.5-ready image for DGX Spark, yet the import test fails. Why?

The assistant's reasoning, visible in the message text, is:

  1. SGLang version 0.0.0: This is not a real version number. It indicates the package was installed via pip install -e . (editable mode) from a source tree, but the version metadata wasn't properly set. The actual SGLang code might be quite recent — perhaps even a main-branch build.
  2. Transformers 4.57.6: This is definitively too old. Qwen3.5 support was added to the Transformers library in version 5.0.0rc3 or later. The assistant correctly identifies that transformers >=5.x is a hard requirement.
  3. The salvage hypothesis: If the SGLang code in the image is actually a recent main-branch build (despite the bogus version number), and the only missing piece is the transformers version, then upgrading transformers inside the container might fix everything. This is the question the assistant sets out to answer. The bash command the assistant runs is designed to probe this hypothesis. Instead of relying on import tests (which failed), the assistant tries to inspect the SGLang models directory directly, listing files whose names contain "qwen." If qwen3_5_moe.py exists in that directory, it would confirm that the SGLang source code in the image does include Qwen3.5 support — and the only fix needed would be a transformers upgrade.

The Failure and Its Significance

The command fails with a TypeError because m.__file__ is None. This is a direct consequence of the editable install: when a package is installed in editable mode, the top-level package might not have a __file__ attribute if it's a namespace package or if the import system resolves it differently. The assistant's diagnostic tool breaks on the very pathology it's trying to investigate.

But the failure is itself diagnostic. It confirms that the SGLang installation in this image is indeed unusual — not a standard built package but an editable source mount. This means the image is likely a development or test build, not a polished release. The 0.5.10rc0 tag is aspirational; the actual contents are a work in progress.

More importantly, the failure forces the assistant to reconsider its approach. The hypothesis that a simple pip install transformers>=5.0 inside the container would fix everything is now harder to test. The assistant cannot easily enumerate the available model files to confirm Qwen3.5 support exists in the SGLang source. The path forward is unclear.

What This Message Reveals About the Deployment Challenge

This message, though only a few lines, exposes several deep truths about the state of AI infrastructure in early 2026:

The bleeding edge is unstable. Qwen3.5-122B-A10B was released very recently. Support for it in inference engines like SGLang and vLLM is being added in real time, in main-branch commits and nightly builds. There are no stable, tagged releases that support it. Every option is a moving target.

DGX Spark is a niche platform. The GB10 Blackwell GPU (SM121 architecture) requires custom builds of PyTorch, CUDA kernels, and inference engines. The pool of available Docker images is tiny — a single official SGLang image and a handful of community builds. The assistant has now exhausted both.

Version detection is unreliable. The 0.0.0 version string and the None __file__ attribute are symptoms of a build process that prioritizes speed over polish. The image works for its intended use case but breaks standard introspection tools. The assistant must adapt its diagnostic techniques on the fly.

Transformers version is a hard gate. The assistant correctly identifies that Qwen3.5 requires transformers >=5.x. This is a non-negotiable dependency. No amount of patching SGLang will work without it. The scitrera image ships transformers 4.57.6, which predates Qwen3.5 support.

The Broader Pattern

This message is a microcosm of the entire session's rhythm: identify a candidate solution, test it, discover it's insufficient, and pivot. The assistant has now tested three inference containers (the original vLLM v0.14, the official SGLang spark image, and the community SGLang 0.5.10rc0 image) and found all three lacking Qwen3.5 support.

The next logical step — hinted at in the message's closing question ("whether we can just upgrade transformers inside it") — would be to attempt a container patch. But the None __file__ failure suggests the assistant is running out of easy options. The path forward will require either building a custom container from scratch, finding a different community image, or switching to a different inference engine entirely.

In the broader arc of the conversation, this message marks the point where the SGLang-on-DGX-Spark approach reaches its dead end. The assistant will soon pivot to a vLLM-based solution using a different community image (hellohal2064/vllm-qwen3.5-gb10), which ultimately succeeds. But at this moment, in message 6573, the assistant is still trying to make the SGLang path work — probing, testing, and learning from each failure.

Input Knowledge Required

To fully understand this message, a reader needs familiarity with several technical domains:

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. The scitrera/dgx-spark-sglang:0.5.10rc0 image does not support Qwen3.5 out of the box.
  2. The image's SGLang is installed in editable mode (version 0.0.0, __file__ is None), suggesting a development build.
  3. The image ships transformers 4.57.6, which is below the 5.x threshold required for Qwen3.5.
  4. The directory inspection approach to enumerate model files fails on editable-install packages, closing off one diagnostic path.
  5. The assistant's hypothesis — that upgrading transformers alone might fix the image — remains untested but plausible.

Conclusion

Message 6573 is a small but revealing moment in a complex deployment. It shows the assistant operating at the boundary of known infrastructure, where standard tools break and every assumption must be verified. The failed os.path.dirname(m.__file__) call is not just a bug — it's a signal that the assistant has reached the edge of what pre-built containers can offer. The next step will require building something new.

This message also demonstrates a key skill in AI infrastructure work: the ability to extract information from failure. The TypeError tells the assistant that the package is installed in an unusual way. The 0.0.0 version tells it that the build process was ad-hoc. The transformers version tells it that even if SGLang supported Qwen3.5, the dependency chain would be broken. Each piece of bad news is also a piece of information, narrowing the search space and pointing toward the eventual solution.