The Vanishing Container: Debugging an Immediate Crash in Multi-Node LLM Deployment

Introduction

In the middle of a complex multi-node deployment of the Qwen3.5-122B-A10B-FP8 language model across two NVIDIA DGX Spark systems, the assistant encounters a frustratingly opaque failure: a Docker container that crashes so fast it evaporates before any logs can be retrieved. Message [msg 6662] captures a brief but pivotal debugging moment — the assistant's attempt to diagnose why a freshly launched vLLM inference server exited immediately, only to find that the container and all evidence of its failure have already disappeared.

This message, though only a single bash command and its terse error output, reveals a great deal about the challenges of deploying large language models on cutting-edge hardware, the realities of debugging distributed systems, and the assistant's methodical approach to problem-solving under uncertainty.

Context: The Broader Deployment Effort

The assistant has been working through Segment 42 of a long-running session to deploy Qwen3.5-122B-A10B-FP8 — a 122-billion parameter Mixture-of-Experts model quantized to FP8 — across two DGX Spark nodes. These are compact ARM-based systems (NVIDIA GB10, SM121 Blackwell GPU, 120GB unified memory) connected via InfiniBand RoCE. The deployment has already encountered significant obstacles.

Earlier attempts using SGLang for multi-node inference failed due to a deadlock in NCCL initialization. The NCCL communicator channels would establish correctly over InfiniBand (NET/IBext_v11), but the process would hang indefinitely after printing "CustomAllreduce is disabled because this process group spans across nodes." The assistant spent several messages diagnosing this deadlock, trying different SGLang configurations (including --disable-custom-all-reduce), but the same hang persisted.

Pivoting to vLLM, the assistant discovered a specialized Docker image — hellohal2064/vllm-qwen3.5-gb10 — built specifically for Qwen3.5 on DGX Spark GB10 hardware. This image contains vLLM 0.17.1rc1 with explicit Qwen3.5 model support files (qwen3_5.py, qwen3_5_mtp.py, qwen3_5_moe.py). In message [msg 6660], the assistant launched a single-node test container to verify the image worked before attempting the more complex multi-node setup:

docker run --rm -d \
  --gpus all --ipc=host --network host \
  --privileged \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v /home/aurora/models/Qwen3.5-122B-A10B-FP8:/models/Qwen3.5-122B-A10B-FP8:ro \
  --name vllm-qwen35-test \
  hellohal2064/vllm-qwen3.5-gb10 \
  vllm serve /models/Qwen3.5-122B-A10B-FP8 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000 \
    --reasoning-parser qwen3

The container started and returned a container ID. But when the assistant checked the logs 30 seconds later in message [msg 6661], Docker reported: "No such container: vllm-qwen35-test." The container had already exited and been removed.

The Subject Message: A Debugging Attempt

Message [msg 6662] is the assistant's immediate response to this vanishing act:

Crashed immediately. Let me check what happened:

>

``bash ssh aurora@10.1.230.180 'docker logs vllm-qwen35-test 2>&1 | tail -30 ; docker ps -a | grep qwen35' 2>&1 ``

>

`` Error response from daemon: No such container: vllm-qwen35-test ``

The assistant's opening line — "Crashed immediately" — is a reasoning conclusion drawn from the previous message's result. The container was gone after only 30 seconds, which for a model that takes 15 minutes to load weights means the crash happened during initialization, likely within seconds of startup. The assistant then takes the logical next step: try to retrieve the container's logs to understand the failure mode.

The bash command is well-constructed for this diagnostic purpose. It first attempts docker logs to retrieve the container's stdout/stderr output. If the container had simply exited (rather than being removed), this command would work — Docker preserves logs for stopped containers. The 2>&1 redirect ensures any error messages are captured. The tail -30 limits output to the most recent lines, which is where the crash error would appear.

The second command, docker ps -a | grep qwen35, is a fallback diagnostic. If the container had been renamed or if the name was slightly different, this would find any container (running or stopped) whose name or image matches "qwen35." It's a sensible hedge against the possibility that the container name was mistyped or that Docker created it under a different identifier.

The result, however, is a dead end. The first command returns the same "No such container" error, confirming the container has been fully removed from Docker's records. The second command produces no output at all — no matching containers exist in any state. The container has been completely erased.## Why the Container Vanished: The --rm Flag

The key to understanding this failure lies in the Docker command from message [msg 6660]. The container was launched with the --rm flag, which tells Docker to automatically remove the container's filesystem and metadata when it exits. This is a common practice for test containers to avoid cluttering the system with stopped containers. However, it has a critical downside for debugging: when a container crashes immediately, --rm destroys all evidence of the failure.

The assistant had used --rm in combination with -d (detached mode). The -d flag runs the container in the background, returning the container ID immediately. But because the container crashed during initialization — likely within milliseconds of starting the vLLM server process — Docker removed it before the assistant could inspect its logs.

This design choice reveals an assumption: the assistant expected the container to either run successfully or at least survive long enough for logs to be retrieved. In many deployment scenarios, this is a reasonable assumption. A vLLM server loading a 122B parameter model typically takes minutes to initialize, during which it prints progress messages to stdout. A crash during this phase would still leave the container in an "exited" state with its logs intact. But the --rm flag overrides this behavior, deleting the container immediately upon exit.

Assumptions and Their Consequences

Several assumptions are visible in this debugging episode:

Assumption 1: The container would survive long enough for log retrieval. This proved incorrect. The crash was so immediate that by the time the assistant checked 30 seconds later, the container had been fully removed. The assistant might have assumed that even a failed initialization would produce some output before the process terminated, but the crash appears to have been instantaneous — possibly a segfault, a missing shared library, or an immediate Python import error that caused the process to exit before writing anything to stdout.

Assumption 2: docker logs would work on a stopped container. This is normally true — Docker preserves logs for exited containers. But --rm bypasses this behavior. The assistant may have forgotten that --rm was used, or may not have anticipated such an immediate crash.

Assumption 3: The vLLM image would work with this specific model on this hardware. The image was specifically built for Qwen3.5 on GB10, so this was a reasonable assumption. But the crash suggests something is incompatible — perhaps a CUDA version mismatch, a missing kernel module for SM121 Blackwell, or a model configuration that the vLLM version cannot parse.

Input Knowledge Required

To fully understand this message, the reader needs knowledge of:

  1. Docker container lifecycle: The --rm flag, detached mode (-d), and how docker logs works for running vs. stopped vs. removed containers.
  2. The broader deployment context: That the assistant was testing a vLLM image specifically built for Qwen3.5 on DGX Spark, and that this was a single-node test before attempting multi-node.
  3. The previous failure mode: That SGLang had failed with NCCL deadlocks, making vLLM the fallback option.
  4. The hardware constraints: DGX Spark uses ARM64 architecture with NVIDIA GB10 (SM121 Blackwell) GPU and 120GB unified memory, which limits software compatibility.
  5. The model characteristics: Qwen3.5-122B-A10B-FP8 is a 122B parameter MoE model, requiring significant GPU memory and specific model support in the inference engine.

Output Knowledge Created

This message, despite its apparent failure, creates valuable knowledge:

  1. The vLLM image crashes immediately on this hardware/model combination. This is a negative result that rules out a simple single-node test. The assistant now knows that the image cannot even initialize the model on a single GPU, which suggests deeper issues.
  2. The crash is too fast for log capture with --rm. This informs the next debugging step: the assistant will need to run the container without --rm (or use alternative logging mechanisms) to capture the crash output.
  3. The problem is not NCCL-related. Unlike the SGLang deadlock which occurred during NCCL initialization, this crash happens during vLLM's own startup sequence, before any distributed communication is established. This suggests a different class of problem — possibly model loading, CUDA compatibility, or Python dependency issues.

The Thinking Process

The assistant's reasoning is visible in the structure of the debugging attempt. The opening line "Crashed immediately" is not just a statement of fact but a diagnostic inference. The assistant is reasoning backward from the observation (container gone after 30 seconds) to the conclusion (crash during initialization). This is sound reasoning: a 122B model takes 15+ minutes to load, so if the container is gone in 30 seconds, it never reached the weight loading phase.

The dual-command approach in the bash script also reveals the assistant's thinking. The first command (docker logs) assumes the container exists but is stopped. The second command (docker ps -a | grep qwen35) handles the possibility that the container was renamed or that the assistant misremembered the container name. This shows a methodical, hypothesis-driven approach: "If the container exists, get its logs. If it doesn't exist under this name, search for any container matching a pattern."

The fact that both commands return nothing forces the assistant to reconsider. The container didn't just exit — it was removed. This realization would lead to the next logical step: run the container without --rm so that logs survive even after a crash.

Broader Implications

This message illustrates a common challenge in deploying AI models on novel hardware: the "silent crash" problem. When a containerized application fails during initialization, it often produces no useful error output before dying. The combination of Docker's --rm flag, detached mode, and an immediate segfault or import error creates a perfect information vacuum. The developer sees only that the container started and then disappeared, with no indication of why.

This is especially problematic for ARM64 Blackwell systems like DGX Spark, where software compatibility is still maturing. Many Docker images are built for x86_64, and even those built for ARM64 may have subtle issues with CUDA version compatibility, PyTorch version mismatches, or missing model-specific patches. The assistant's experience here is typical of the bleeding edge of AI infrastructure — where every deployment requires iterative debugging, and negative results are as valuable as positive ones.

The assistant's response to this failure — a calm, methodical attempt to gather information rather than frustration or random retrying — demonstrates the mindset required for this work. The vanishing container is not a dead end; it's a signal that guides the next debugging step. And indeed, in subsequent messages, the assistant would adapt the approach, running without --rm and capturing the actual error, eventually achieving a successful multi-node deployment with vLLM and Ray.