The Incremental Verification: Testing Single-Node Before Multi-Node Deployment of Qwen3.5-122B on DGX Spark

In the complex world of distributed AI inference deployment, the most successful engineers are those who resist the temptation to leap directly to the final, integrated system. Instead, they methodically verify each component in isolation before assembling the whole. Message [msg 6588] of this opencode session captures a perfect example of this philosophy in action: a deliberate, single-node smoke test performed while a multi-node model transfer runs in the background, designed to catch fundamental issues before they compound into hours of debugging.

The Message in Full

The assistant writes:

While the rsync runs, let me first test a single-node launch on the head to verify the image/model combination works before attempting multi-node:

>

``bash ssh aurora@10.1.230.180 'docker run --rm \ --gpus all --ipc=host --network host \ --ulimit memlock=-1 --ulimit stack=67108864 \ -v /home/aurora/models/Qwen3.5-122B-A10B-FP8:/models/Qwen3.5-122B-A10B-FP8 \ sglang-qwen35 \ python3 -c " from transformers import AutoConfig c = AutoConfig.from_pretrained(\"/models/Qwen3.5-122B-A10B-FP8\", trust_remote_code=True) print(\"Architecture:\", c.architectures) print(\"Num experts:\", c.text_config.num_experts if hasattr(c, \"text_config\") else \"N/A\") print(\"MoE intermediate:\", c.text_config.moe_intermediate_size if hasattr(c, \"text_config\") else \"N/A\") import torch print(\"GPU:\", torch.cuda.get_device_name(0)) print(\"GPU Memory:\", torch.cuda.get_device_properties(0).total_mem / 1e9, \"GB\") print(\"Compute:\", torch.cuda.get_device_capability(0)) " 2>&1 | tail -10' 2>&1 ``

>

``` Using CUDA 13.1 driver version 590.48.01 with kernel driver version 580.95.05. See https://docs.nvidia.com/deploy/cuda-compatibility/ for details.

>

Traceback (most recent call last): File "<string>", line 9, in <module> AttributeError: 'torch._C._CudaDeviceProperties' object has no attribute 'total_mem'. Did you mean: 'total_memory'? Architecture: ['Qwen3_5MoeForConditionalGeneration'] Num experts: 256 MoE intermediate: 1024 GPU: NVIDIA GB10 ```

The Strategic Context: Why This Message Exists

To understand why this message was written, we must trace the arc of the preceding conversation. The assistant has been engaged in a multi-hour effort to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter, 10-billion-active-parameter Mixture-of-Experts reasoning model — across two NVIDIA DGX Spark systems. These are compact ARM-based workstations (GB10 GPU, SM121 Blackwell architecture, 120GB unified memory each) connected via InfiniBand RoCE.

The path to this moment had already involved numerous critical decisions. The assistant had:

1. Stopped the existing GLM-4.7-Flash containers to free GPU memory ([msg 6566])
2. Discovered that the official lmsysorg/sglang:spark image (SGLang 0.5.4, transformers 4.57) was too old to support Qwen3.5 ([msg 6571])
3. Found the community scitrera/dgx-spark-sglang:0.5.10rc0 image which had Qwen3.5 model files but still lacked transformers support (<msg id=6572-6577>)
4. Built a custom Docker image (sglang-qwen35) by upgrading transformers to >=5.0 inside the container (<msg id=6578-6579>)
5. Downloaded the full 119GB model from HuggingFace (<msg id=6581-6586>)
6. Started an rsync to copy the model to the second Spark node ([msg 6587]) The rsync, running in the background with nohup, would take significant time to transfer 119GB even over the 640MB/s InfiniBand link. Rather than idly waiting, the assistant seized the opportunity to validate the deployment stack on a single node. This is the essence of the message: parallelized risk reduction.

The Reasoning: Incremental Verification as a Strategy

The assistant's opening line — "While the rsync runs, let me first test a single-node launch on the head to verify the image/model combination works before attempting multi-node" — reveals a deliberate, layered reasoning process. The key insight is that multi-node inference introduces a host of additional failure modes: NCCL initialization across nodes, Ray cluster formation, network routing, shared filesystem access, and tensor parallelism coordination. If the fundamental image/model combination is broken, discovering that during multi-node debugging would waste hours.

The assistant is applying a classic engineering principle: test the simplest possible configuration first. A single-node launch eliminates all distributed-system variables. If the model config can't be parsed, if the GPU isn't accessible, if the Docker container has library conflicts, or if the CUDA version is incompatible, those issues will surface immediately in this minimal test. Each of these failure modes would present differently in a multi-node context, potentially masked by or conflated with networking errors.

The timing is also strategic. By running this test concurrently with the rsync, the assistant effectively doubles throughput. If the test fails, the rsync can be aborted before completing, saving the transfer time. If the test passes, the assistant gains confidence that the multi-node attempt will at least have a working foundation.

The Technical Construction: A Carefully Designed Smoke Test

The Docker command itself is a study in production-grade container configuration. Every flag serves a specific purpose:

• --gpus all: Grants GPU access to the container, necessary for any CUDA operation
• --ipc=host: Shares the host's IPC namespace, critical for PyTorch's NCCL initialization and shared memory operations
• --network host: Uses the host's network stack directly, avoiding Docker NAT complications — essential for the subsequent multi-node setup where network interfaces matter
• --ulimit memlock=-1 --ulimit stack=67108864: Removes memory locking limits, required by NVIDIA's container toolkit for CUDA memory allocation
• -v /home/aurora/models/...:/models/...: Mounts the downloaded model into the container at a predictable path The Python test script probes four distinct layers of the deployment stack: 1. Model architecture compatibility (AutoConfig.from_pretrained): This validates that the transformers library can parse the model's config.json and recognize the Qwen3_5MoeForConditionalGeneration architecture. If this fails, no amount of distributed setup will help. 2. Model structural parameters (num_experts, moe_intermediate_size): These confirm the MoE configuration matches expectations. 256 experts with an intermediate size of 1024 are characteristic of the Qwen3.5-122B architecture. 3. GPU accessibility (torch.cuda.get_device_name): Verifies the container can see the GPU and that it's the expected NVIDIA GB10 (Blackwell) hardware. 4. GPU compute capability (torch.cuda.get_device_capability): Checks the CUDA compute capability, which determines which kernels and optimizations are available.

The Outcome: Partial Success Revealing a Minor Bug

The test produces a mixed result. The critical information comes through clearly:

• Architecture: Qwen3_5MoeForConditionalGeneration — confirmed
• Number of experts: 256 — matches the Qwen3.5-122B specification
• MoE intermediate size: 1024 — correct
• GPU: NVIDIA GB10 — the expected Blackwell GPU However, the line torch.cuda.get_device_properties(0).total_mem raises an AttributeError because the correct attribute name is total_memory, not total_mem. This is a minor Python API error — the developer likely confused the attribute name with a common abbreviation. The error is harmless in this context; it doesn't affect the model loading or inference pipeline, only the reporting of GPU memory size. The CUDA forward compatibility notice is also noteworthy: the container has CUDA 13.1, but the host kernel driver is version 580.95.05 (which supports CUDA 13.0). The system reports "CUDA Forward Compatibility mode ENABLED," meaning the newer CUDA toolkit in the container is being allowed to run on the slightly older driver. This is a potential risk point — forward compatibility is not always perfect, especially for cutting-edge Blackwell GPUs with custom kernels.

Assumptions and Their Validity

The assistant makes several implicit assumptions in this test:

1. The custom Docker image is correctly built: The sglang-qwen35 image was created by upgrading transformers inside the scitrera/dgx-spark-sglang:0.5.10rc0 base. The test confirms this assumption holds — transformers can parse the model config.
2. The model download is complete and uncorrupted: The 119GB model was downloaded from HuggingFace in the preceding messages. The test validates this by successfully loading the config, but doesn't verify individual weight files.
3. GPU access works within the container: The Docker flags are standard for NVIDIA containers, and the test confirms GPU visibility.
4. The model path mapping is correct: The volume mount maps /home/aurora/models/Qwen3.5-122B-A10B-FP8 on the host to /models/Qwen3.5-122B-A10B-FP8 in the container, and the Python script uses the container-side path.
5. The single-node test is representative of multi-node viability: This is the most important assumption. A working single-node configuration doesn't guarantee multi-node success — NCCL initialization, network topology, and tensor parallelism introduce entirely new failure modes. However, a failing single-node test would guarantee multi-node failure, making this a useful gating check.

Input Knowledge Required

To fully understand this message, one must be familiar with:

• DGX Spark / GB10 architecture: NVIDIA's compact AI workstation with a Blackwell GPU and ARM CPU, 120GB unified memory
• SGLang and vLLM: Inference serving frameworks for large language models
• Qwen3.5 model family: Alibaba's Mixture-of-Experts reasoning models, specifically the 122B-A10B variant with 256 experts
• Docker for GPU workloads: The specific flags needed for CUDA containers (--gpus, --ipc=host, --ulimit memlock)
• HuggingFace transformers: The AutoConfig.from_pretrained API and model configuration parsing
• Multi-node inference concepts: Tensor parallelism, NCCL, InfiniBand, Ray cluster formation
• CUDA forward compatibility: The mechanism allowing newer CUDA toolkits to run on older kernel drivers

Output Knowledge Created

This message produces several valuable pieces of knowledge:

1. The custom Docker image is viable: The sglang-qwen35 image can load and parse the Qwen3.5-122B model config, confirming the transformers upgrade was successful.
2. The model architecture is as expected: 256 experts, MoE intermediate size 1024, architecture Qwen3_5MoeForConditionalGeneration.
3. GPU is accessible: The NVIDIA GB10 GPU is visible from within the container with compute capability 12.1 (SM120 Blackwell).
4. CUDA forward compatibility is active: CUDA 13.1 in the container is running on a driver that nominally supports CUDA 13.0, which may cause issues with certain operations.
5. A minor API bug exists: The total_mem attribute name is incorrect; the correct name is total_memory.
6. The deployment is ready for multi-node testing: With the single-node smoke test passing, the assistant can proceed to the more complex distributed setup with reasonable confidence.

The Broader Engineering Lesson

This message exemplifies a pattern that recurs throughout successful infrastructure engineering: parallelize validation with execution. Rather than treating verification as a sequential gate that blocks progress, the assistant integrates it into the workflow, running it concurrently with a long-running data transfer. This approach maximizes resource utilization and minimizes idle time.

The choice of what to test is equally instructive. The assistant doesn't attempt a full inference run — no model loading, no token generation. Instead, it tests the minimal set of preconditions that would cause the most catastrophic failures: model config parsing and GPU access. This is a tiered verification strategy: if the shallow checks pass, deeper checks (model loading, forward pass, multi-node coordination) can be attempted in subsequent rounds.

The message also demonstrates the importance of reading error output carefully. The AttributeError about total_mem is clearly a typo in the test script, not a real infrastructure problem. The assistant correctly interprets this as a non-issue and proceeds to the next round ([msg 6589]) with a refined test that checks compute capability and CUDA tensor operations instead. This ability to distinguish between "test script bug" and "infrastructure failure" is a hallmark of experienced engineers.

In the subsequent message ([msg 6589]), the assistant runs a follow-up test that confirms compute capability (12, 1) and successfully executes a CUDA tensor operation, further validating the deployment stack. The multi-node deployment then proceeds, eventually achieving ~27 tok/s single-request throughput with correct reasoning output — a successful outcome built on the foundation of this incremental verification step.