The Silent Crash: How a Missing Container Revealed the Path Forward

The Message

[assistant] [bash] sleep 30 && ssh aurora@10.1.230.180 'docker logs vllm-qwen35-test 2>&1 | tail -20' 2>&1
Error response from daemon: No such container: vllm-qwen35-test

At first glance, this message appears trivial — a simple bash command that failed because a Docker container no longer existed. But in the arc of a complex multi-node deployment, this single line of output represents a critical inflection point. The assistant had just launched a test container to validate a newly discovered Docker image for running Qwen3.5-122B-A10B-FP8 on a DGX Spark system. Thirty seconds later, the container was gone. No logs. No crash dump. No error message beyond the stark "No such container" response. This silence forced the assistant to reconsider its assumptions, change its debugging strategy, and ultimately discover a fundamental incompatibility between the image's design and the launch command.

The Road to This Moment

To understand why this message matters, one must appreciate the journey that led to it. The assistant had spent the better part of an hour wrestling with SGLang, attempting to deploy the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter FP8-quantized mixture-of-experts model — across two DGX Spark nodes connected via InfiniBand. SGLang's multi-node NCCL initialization kept deadlocking. The logs showed NCCL channels being established over NET/IBext_v11 (InfiniBand), but the process never progressed past CustomAllreduce is disabled. It hung there for seven minutes, then fourteen, completely frozen.

After extensive debugging — checking NCCL communicator groups, adjusting --disable-custom-all-reduce, waiting through multiple timeouts — the assistant pivoted. A web search revealed hellohal2064/vllm-qwen3.5-gb10, a Docker image specifically built for running Qwen3.5 on DGX Spark (GB10) with vLLM 0.17.1rc1, ARM64 support, and SM120/121 Blackwell compatibility. This was a lifeline. The assistant pulled the image, transferred it to the second Spark node, and prepared to test it.

The test was meant to be straightforward: launch a single-node container, verify the image could load the model, then scale to the dual-node Ray-based deployment. The assistant issued the docker run command with the --rm flag (auto-remove on exit), pointed the model volume to the correct path, set tensor-parallel-size to 1, and waited. The container ID printed successfully. The assistant then waited 30 seconds before checking the logs.

What the Error Actually Tells Us

The --rm flag is a Docker convenience that automatically removes a container once it exits. It is invaluable for ephemeral tasks — run a command, get the output, clean up automatically. But in a debugging context, it is a double-edged sword. When a container crashes immediately, --rm erases the evidence. The container, its logs, its exit code — all gone.

The error No such container: vllm-qwen35-test is therefore not just a failure message; it is a diagnostic signal in itself. It tells us:

  1. The container exited within 30 seconds. The assistant's sleep 30 was intended to give the model time to begin loading (a 119GB model on ARM with unified memory is slow). But the container didn't survive that long.
  2. The exit was not graceful. A healthy container starting to load a model would still be running after 30 seconds. The fact that it exited — and was auto-removed — indicates a crash during initialization, likely before any significant log output was produced.
  3. The --rm flag concealed the failure mode. Without the container, the assistant could not inspect logs, check the exit code, or see stderr output. The failure was a black box.

Assumptions Under the Microscope

This message exposes several assumptions that the assistant made, each of which turned out to be incorrect:

Assumption 1: The container would still be running. The assistant expected the model to begin loading slowly — ARM CPUs are not fast, and a 119GB model takes time to initialize. Waiting 30 seconds before checking logs seemed reasonable. But the assumption that the container would survive its own startup was wrong.

Assumption 2: The --rm flag was safe for a test. In many debugging workflows, --rm is standard practice — it prevents clutter from failed containers. But when a container fails silently, --rm becomes destructive. It removes the only source of diagnostic information. The assistant learned this the hard way and immediately switched to running without --rm in the next attempt.

Assumption 3: The vllm serve command would work as expected. The assistant passed vllm serve /models/Qwen3.5-122B-A10B-FP8 as the container command, assuming the image's default entrypoint would forward arguments correctly. In reality, the image had its own entrypoint script that expected the model at /models/model and set default arguments internally. The command was being duplicated, causing a crash. This was only discoverable after removing --rm and inspecting the actual logs.

Assumption 4: The image would produce useful startup logs before crashing. Even without --rm, the assistant assumed that some log output would be captured. But the crash may have happened so early — perhaps during argument parsing or GPU detection — that no meaningful log was written before the process died.

The Thinking Process Revealed

The assistant's reasoning in this moment is visible through the subsequent actions. The very next message (msg 6662) shows the assistant processing the failure:

"Crashed immediately. Let me check what happened:"

This is the key insight: the assistant correctly interpreted "No such container" as "crashed immediately." The error message, though minimal, was sufficient to diagnose the timing of the failure. The container did not hang — it exited. The assistant then ran the same command without --rm, which allowed the container to persist even after crashing, revealing the actual error logs. Those logs showed the image's custom entrypoint conflicting with the explicit vllm serve command.

The assistant's debugging methodology here is worth noting. It did not:

The Broader Significance

In the context of the entire deployment effort, this message is the turning point. The SGLang approach had failed with a deadlock that provided plenty of logs but no clear path forward. The vLLM image approach failed with a crash that provided no logs at all — yet this second failure was easier to diagnose once the visibility issue was resolved. Within three more messages, the assistant had the container running successfully, confirming that vLLM 0.17 could load Qwen3.5-122B-A10B-FP8 with correct architecture detection and FP8 quantization support.

The message also illustrates a recurring theme in infrastructure engineering: the most informative failures are often the ones that look like nothing at all. A container that silently disappears forces the operator to question every assumption — about the image, the command, the flags, the environment. The assistant rose to this challenge, turning a moment of apparent failure into the foundation for eventual success.

Conclusion

The message Error response from daemon: No such container: vllm-qwen35-test is, on its surface, a mundane error. But within the narrative of this deployment, it represents a critical diagnostic moment. The assistant's ability to interpret this sparse signal — to recognize that a missing container implies a crash, that the --rm flag had destroyed evidence, and that the fix was to preserve that evidence — demonstrates the kind of systematic reasoning that separates effective infrastructure engineering from trial-and-error. The message is a reminder that in complex systems, the most valuable information often comes not from what the error says, but from what it implies about the state of the system and the assumptions of the operator.