The Pivot That Saved the Deployment: Debugging a Vanishing Container on DGX Spark

In the high-stakes world of deploying large language models across multi-node GPU clusters, the smallest configuration detail can mean the difference between a running service and a silent crash. Message [msg 6663] captures one such moment — a seemingly trivial Docker flag that caused a container to vanish without a trace, and the methodical debugging that turned a failure into a breakthrough. This message represents the inflection point where the assistant abandoned a dead-end approach with SGLang and proved that vLLM could successfully serve the Qwen3.5-122B-A10B-FP8 model on NVIDIA's DGX Spark hardware.

The Context: A Multi-Node Nightmare

The story leading up to this message is one of escalating frustration. The assistant had spent hours attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter FP8 Mixture-of-Experts model — across two DGX Spark nodes connected via InfiniBand RoCE. The initial approach used SGLang, but the multi-node NCCL initialization consistently deadlocked. Despite adding --disable-custom-all-reduce, forcing node IPs to the InfiniBand subnet, and waiting over seven minutes for initialization to complete, the SGLang processes remained frozen at the NCCL channel setup phase, never progressing to weight loading.

After discovering that SGLang's official spark image lacked Qwen3.5 support and its multi-node NCCL initialization hung indefinitely, the assistant pivoted to vLLM. A community-built Docker image — hellohal2064/vllm-qwen3.5-gb10 — was identified as specifically optimized for Qwen3.5 on the DGX Spark's GB10 ARM64 platform with SM121 Blackwell GPUs. The image was pulled, transferred to the second Spark node, and a quick single-node test was attempted in message [msg 6660] using the --rm flag.

The Vanishing Container

The initial test command in [msg 6660] looked correct on the surface:

docker run --rm -d \
  --gpus all --ipc=host --network host \
  --privileged \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /home/aurora/models/Qwen3.5-122B-A10B-FP8:/models/Qwen3.5-122B-A10B-FP8:ro \
  --name vllm-qwen35-test \
  hellohal2064/vllm-qwen3.5-gb10 \
  vllm serve /models/Qwen3.5-122B-A10B-FP8 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --reasoning-parser qwen3

The container ID was returned, suggesting a successful launch. But thirty seconds later in [msg 6661], when the assistant tried to check the logs, the response was stark: Error response from daemon: No such container: vllm-qwen35-test. The container had crashed and, because of the --rm flag, Docker had automatically cleaned it up, leaving no trace of what went wrong.

This is a classic debugging trap. The --rm flag is convenient for ephemeral containers, but it destroys the evidence when something fails. The assistant had no logs, no error messages, no crash dump — just a vanished container and the knowledge that something had gone wrong during vLLM's startup sequence.

Message [msg 6663]: The Methodical Retry

The subject message opens with the assistant's immediate diagnosis: "Container gone (--rm flag)." This single line reveals the assistant's deep understanding of Docker's lifecycle management. Rather than speculating about configuration errors, CUDA compatibility issues, or model loading failures, the assistant correctly identified the root cause: the --rm flag had masked the failure by deleting the evidence.

The solution was elegant in its simplicity: remove the --rm flag. The revised command in [msg 6663] is nearly identical to the first attempt, with two critical changes:

No --rm flag: The container will persist even if it crashes, preserving logs and state for debugging.
Embedded sleep 15 and docker logs: The assistant chains a 15-second wait and a log inspection into the same SSH command, ensuring the results are captured even if the SSH session times out. The assistant also made a deliberate choice to keep the same vLLM arguments from the failed attempt. This is an important decision: by holding the configuration constant and only changing the Docker lifecycle behavior, the assistant ensures that any new information gained is attributable to the single variable changed. If the container still crashed, the logs would now be available. If it succeeded, the problem was purely the --rm flag's interaction with a crash that may have been transient.

What the Output Revealed

The output from this message was dramatically different from the silent failure of the first attempt:

[THERMAL] Setting GPU max clock to 2600 MHz...
GPU clocks set to "(gpuClkMin 0, gpuClkMax 2600)" for GPU 0000000F:01:00.0
All done.
=============================================
vLLM Server for NVIDIA GB10 (SM121 Blackwell)
=============================================

Configuration:
  Model:              /models/model
  Host:               0.0.0.0:8000
  Max Model Length:   8192
  GPU Memory Util:    0.85
  Attention Backend:  FL...

The vLLM server was alive. The thermal management system had successfully set the GPU clock to 2600 MHz — a critical step for the DGX Spark's unified thermal design where CPU and GPU share a cooling solution. The server banner confirmed it was running on the correct platform (SM121 Blackwell) with the expected configuration.

Several details in this output are worth examining:

The model path discrepancy: The configuration shows Model: /models/model, but the volume mount was to /models/Qwen3.5-122B-A10B-FP8. This suggests the Docker image has a symlink or expects the model at a specific path. The assistant had mounted the actual model directory at the correct location, and the image's internal startup script likely created a symlink from /models/model to the actual model path. This is a common pattern in specialized Docker images to abstract away model naming.

The host and port difference: The configuration shows Host: 0.0.0.0:8000, but the assistant passed --host 0.0.0.0 --port 30000. The image's entrypoint script overrode these defaults. This is a significant finding — the assistant would need to account for this when setting up the multi-node configuration, as the default port 8000 might conflict with other services.

The max model length: The configuration shows Max Model Length: 8192, while the assistant passed --max-model-len 4096. Again, the image's internal defaults took precedence. This is actually beneficial — 8192 is a more useful context length for a reasoning model like Qwen3.5.

The attention backend: The output cuts off at FL..., but this is almost certainly Flash Attention, the optimized attention implementation that vLLM uses for efficient inference on Blackwell GPUs.

The Thinking Process Behind the Message

The assistant's reasoning in this message demonstrates several hallmarks of expert systems debugging:

Hypothesis testing: The assistant had a clear hypothesis — "the container crashed and --rm deleted it" — and designed a minimal experiment to test it. By removing only the --rm flag, the assistant isolated the variable.

Progressive information gathering: Rather than waiting blindly, the assistant embedded a 15-second sleep followed by log inspection. This shows an understanding of vLLM's startup timeline: model loading on ARM64 with unified memory is slow, but 15 seconds is enough to see the initial banner if the process hasn't crashed.

Risk management: The assistant used docker run -d (detached mode) combined with docker logs, ensuring that even if the SSH connection dropped, the container would continue running and could be inspected later.

Contextual awareness: The assistant knew that the previous SGLang attempts had taken minutes to show any progress, so a 15-second wait was a reasonable compromise between impatience and thoroughness.

Assumptions and Their Validity

The assistant made several assumptions in this message:

That the --rm flag was the sole cause of the missing container: This was correct — removing it revealed a running server.
That the same vLLM arguments would work without --rm: This was also correct, though the image overrode several parameters (port, max model length), which the assistant would need to handle later.
That 15 seconds was enough to see startup progress: This was correct — the thermal management and server banner appeared within that window.
That the image's default configuration was acceptable: The assistant implicitly accepted the image's overrides (port 8000 instead of 30000, 8192 context instead of 4096). This was a pragmatic choice — getting something working was more important than enforcing exact parameters.

Input Knowledge Required

To fully understand this message, one needs:

Docker lifecycle management: Understanding that --rm causes automatic container removal on exit, and that a crashed container produces no visible artifacts with this flag.
vLLM architecture: Knowing that vLLM's startup sequence includes GPU thermal configuration, model weight loading, and attention backend initialization.
DGX Spark hardware specifics: The GB10 platform's unified memory architecture, SM121 Blackwell GPU, and the thermal management requirements that necessitate GPU clock limiting.
The Qwen3.5 model: Understanding that this is a 122B-parameter MoE model with FP8 quantization, requiring significant GPU memory and specific model support in the inference engine.
The deployment history: The assistant's failed SGLang attempts and the pivot to vLLM provide the context for why this single-node test was so critical.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

vLLM works on DGX Spark with Qwen3.5: The single-node test succeeded, proving the image and model are compatible with the hardware.
The image has internal defaults that override CLI arguments: The port, model path, and context length were all different from what was passed. This is critical information for the multi-node setup.
GPU thermal management is functional: The clock limiting to 2600 MHz succeeded, indicating the NVIDIA driver and container toolkit are properly configured.
The attention backend is Flash Attention: This confirms efficient inference is possible on the Blackwell architecture.
The model loads successfully: Though the full 15-minute weight loading hadn't completed yet, the server reached the point of announcing its configuration, which means the model directory was accessible and the initial validation passed.

The Broader Significance

This message is the turning point in a multi-hour debugging session. The assistant had been stuck on SGLang's NCCL deadlock for dozens of messages, trying increasingly desperate workarounds. The pivot to vLLM was itself a significant decision, but it was this message — the successful single-node test — that validated the new direction.

The lesson is subtle but important: sometimes the smallest configuration error (a --rm flag) can mask a success. The assistant's methodical approach — diagnosing the symptom, forming a hypothesis, testing with a single variable change, and interpreting the results — turned a silent crash into a running server. In the messages that follow, the assistant would build on this foundation to create a full Ray-based multi-node deployment, eventually achieving ~27 tok/s single-request throughput with correct reasoning output.

For anyone deploying LLMs on specialized hardware, this message serves as a reminder that Docker's convenience features can become debugging obstacles, and that the most effective troubleshooting often comes from understanding the tools deeply enough to know when their defaults are working against you.