The Transformers Upgrade: A Pivotal Docker Build That Brought Qwen3.5 to Life on DGX Spark

In the sprawling, multi-threaded narrative of deploying a 122-billion-parameter Qwen3.5 MoE model across two NVIDIA DGX Spark nodes, there is a single message that captures the essence of pragmatic systems engineering. At message index 6578, the assistant executes what appears to be a trivial operation—writing a two-line Dockerfile and running a build—but this moment represents the culmination of a careful diagnostic chain and the decisive pivot that made the entire deployment possible. The message reads:

Confirmed: the image has the SGLang model files for Qwen3.5, but transformers 4.57.6 doesn't know qwen3_5_moe architecture. I need to upgrade transformers inside the container. This is actually a simple fix — just pip install --upgrade transformers inside the container. Let me create a derivative container:

`` ssh aurora@10.1.230.180 'cat > /tmp/Dockerfile.qwen35 << '\''EOF'\'' FROM scitrera/dgx-spark-sglang:0.5.10rc0 RUN pip install --no-cache-dir "transformers>=5.0" accelerate EOF docker build -t sglang-qwen35 -f /tmp/Dockerfile.qwen35 /tmp/ 2>&1 | tail -20' 2>&1 #5 6.604 #5 6.605 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. #5 6.605 sglang 0.0.0 requires torchaudio==2.9.1, which is not installed. #5 6.605 compressed-tensors 0.14.0.1 requires transformers<5.0.0, but you have transformers 5.5.0 which is incompatible. #5 6.605 sglang 0.0.0 requires cuda-python==12.9, but you have cuda-python 13.1.1 which is incompatible. ``

The Road to This Moment

To understand why this message matters, one must appreciate the journey that preceded it. The assistant was tasked with deploying Qwen3.5-122B-A10B-FP8—a massive Mixture-of-Experts reasoning model from Alibaba's Qwen team—across two DGX Spark systems. Each Spark is an ARM-based (Cortex-X925) NVIDIA GB10 system with 120GB of unified memory, connected via InfiniBand RoCE. The model itself weighs approximately 119GB on disk in FP8 quantization, meaning it would barely fit across the two nodes with tensor parallelism.

The deployment had already encountered multiple roadblocks. The assistant had stopped the existing GLM-4.7-Flash service on the head Spark to free GPU memory, and had discovered that the pre-installed vLLM 0.14 Docker image lacked Qwen3.5 support entirely. A search for alternatives led to two SGLang Docker images: the official lmsysorg/sglang:spark (which turned out to be SGLang 0.5.4 with transformers 4.57—too old), and the community-maintained scitrera/dgx-spark-sglang:0.5.10rc0, which was much more recent at just six days old.

The critical discovery came in the messages immediately preceding [msg 6578]. When the assistant ran the scitrera/dgx-spark-sglang:0.5.10rc0 image and inspected its model directory, it found qwen3_5.py and qwen3_5_mtp.py—the SGLang model implementations for Qwen3.5. This was enormously promising: the SGLang code itself was ready. But when the assistant tried to load the model's configuration using HuggingFace's AutoConfig.from_pretrained, transformers 4.57.6 returned an error because it did not recognize the Qwen3_5MoeForConditionalGeneration architecture string. The model files existed, but the library responsible for parsing the model's config was too old.

The Diagnostic Insight

The assistant's reasoning in [msg 6578] reveals a clear understanding of the software stack's layered dependencies. The problem was not that SGLang lacked Qwen3.5 support—it was that HuggingFace's transformers library, which serves as the configuration parser and model registry, had not yet been updated to include the new architecture. The SGLang model code (qwen3_5.py) would work perfectly fine once transformers could correctly parse the model's config.json and identify the architecture.

This is a subtle but important distinction. Many engineers would have assumed the entire container was incompatible and searched for a completely different image. Instead, the assistant recognized that the fix was narrow and surgical: upgrade transformers to a version (5.x or later) that includes the Qwen3.5 architecture definitions. The accelerate library was added as a dependency because newer transformers versions often require it for device mapping.

The decision to create a derivative Docker image rather than modifying the container at runtime was also strategic. A derivative image is persistent, reproducible, and can be pushed to a registry or transferred to the second Spark node. A runtime pip install inside a running container would be ephemeral and harder to manage in a multi-node setup.

Assumptions and Their Risks

The assistant made several assumptions in this message, each carrying some degree of risk:

First, it assumed that upgrading transformers alone would be sufficient. The SGLang model code (qwen3_5.py) was written against a particular version of the SGLang API, and it was possible that newer transformers introduced breaking changes in how model configurations are loaded. The build output shows that compressed-tensors 0.14.0.1 explicitly requires transformers<5.0.0, meaning the upgrade to transformers 5.5.0 creates a known incompatibility. The assistant implicitly judged that compressed-tensors would not be needed for this deployment, or that any issues would surface at load time rather than preventing the build.

Second, the assistant assumed the model file naming convention. Earlier in the session ([msg 6570]), it had tried to import Qwen3_5MoeForCausalLM from qwen3_5_moe and failed. The actual file is qwen3_5.py, not qwen3_5_moe.py. The assistant correctly inferred that the class name within the file might differ from the filename, and that the SGLang model registry would handle the mapping regardless.

Third, the assistant assumed that the second Spark node (which lacks internet access) would be able to use this derivative image. The plan was to either push the image to a registry accessible from both nodes, or to save and transfer the Docker image tarball over the InfiniBand link. This assumption proved correct in subsequent messages, but it added operational complexity.

The Build Output: Reading the Warning Signs

The build output shown in the message is revealing. The pip dependency resolver prints warnings about three conflicts:

sglang 0.0.0 requires torchaudio==2.9.1, which is not installed. The SGLang version string shows as 0.0.0 because it is an editable install from source without a proper version tag. The torchaudio requirement is likely a packaging metadata issue rather than a real runtime dependency.
compressed-tensors 0.14.0.1 requires transformers<5.0.0, but you have transformers 5.5.0. This is the most significant warning. compressed-tensors is a library for loading quantized models (FP8, INT4, etc.). If the Qwen3.5 FP8 model uses compressed-tensors for its quantization format, this incompatibility could cause a load failure. The assistant would need to verify this later.
sglang 0.0.0 requires cuda-python==12.9, but you have cuda-python 13.1.1. The container ships with CUDA 13.1, but SGLang's metadata pins cuda-python to 12.9. This is a version mismatch in the packaging metadata that is unlikely to cause runtime issues since the actual CUDA runtime libraries are versioned independently. The assistant's decision to proceed despite these warnings reflects a pragmatic risk assessment: packaging metadata conflicts are often spurious, and the core functionality (transformers parsing the config, SGLang loading the model) should work. The build itself succeeded (exit code 0), meaning the Docker image was created.

The Broader Significance

This message is a textbook example of the "layer cake" debugging pattern in machine learning infrastructure. When a model deployment fails, the issue can exist at any layer of the stack: hardware compatibility, CUDA runtime, PyTorch, model serving framework (SGLang/vLLM), HuggingFace transformers, or the model's own configuration. The assistant systematically eliminated layers until it found the exact point of failure: the transformers version.

The fix—a two-line Dockerfile—is almost comically simple compared to the complexity of the overall deployment. But that simplicity is the mark of precise diagnosis. The assistant did not rebuild SGLang from source, did not patch the model code, did not downgrade CUDA, and did not switch to a different serving framework. It identified the single mismatched dependency and fixed it with surgical precision.

In the messages that follow [msg 6578], the assistant verifies that the new image can parse the Qwen3.5 config correctly ([msg 6579]), then proceeds to download the 119GB model and begin the multi-node deployment. The derivative image sglang-qwen35 becomes the foundation for the entire Qwen3.5 serving stack on the DGX Sparks.

Conclusion

Message 6578 is a masterclass in targeted infrastructure intervention. It demonstrates that the most effective fixes are often the smallest ones, provided they are applied at the right layer of abstraction. The assistant's reasoning—"the model files exist, but the config parser is too old"—is a model of diagnostic clarity. By upgrading a single Python package rather than replacing the entire container or framework, the assistant turned an apparent dead end into a working deployment path. The warning messages in the build output serve as a reminder that even clean fixes can have loose ends, but in the high-stakes world of large model deployment, getting the model to serve is the priority, and edge cases can be addressed as they arise.