The Diagnostic Pivot: How a Simple docker ps Revealed Infrastructure State in a Multi-Node AI Deployment

Introduction

In the complex world of distributed AI inference, the most revealing diagnostic commands are often the simplest. Message <msg id=6733> in this opencode session is a case study in how a single docker ps command — executed across two DGX Spark nodes — served as the critical turning point in a troubleshooting workflow. The message, consisting of a single bash invocation that checks running Docker containers on both nodes of a multi-node vLLM deployment, appears unremarkable at first glance. But within the context of the conversation, it represents a moment of diagnostic clarity: the assistant discovers that its carefully orchestrated Qwen3.5-122B-A10B-FP8 deployment has been silently replaced by old GLM containers, forcing a complete reassessment of the situation.

Context: The State of Play

To understand why this message was written, we must reconstruct the circumstances. The user had just issued a simple command in <msg id=6731>: "relaunch the models." This request came immediately after the assistant had successfully benchmarked the Qwen3.5-122B-A10B-FP8 model running across two DGX Spark nodes (NVIDIA's ARM-based Blackwell mini-servers), achieving ~27 tok/s single-request throughput. The deployment had been a significant engineering achievement, requiring the assistant to solve Ray networking issues, OOM killer problems during CUDA graph capture, and NCCL configuration challenges across an InfiniBand RoCE interconnect.

The assistant's first response to the relaunch request was <msg id=6732>, where it attempted to check the health of both systems by querying their API endpoints directly. The results were alarming: the Spark endpoint returned nothing (the curl command produced no output, and the fallback "SPARK DOWN" message was not printed either — suggesting the command may have hung or returned empty), and the RTX PRO 6000 LXC container at 10.1.230.174 was completely unreachable with "No route to host." The assistant's reasoning block in that message reveals its mental model: "On the Sparks: Qwen3.5-122B-A10B-FP8 via vLLM. On the RTX PRO 6000 (LXC container): Qwen3.5-122B-A10B BF16 via SGLang with MTP." It expected both systems to be running the models it had previously deployed.

The Subject Message: A Diagnostic Pivot

Message <msg id=6733> represents the assistant's pivot from direct API health checks to infrastructure-level diagnostics. Rather than trying to query the API endpoints again or assuming the services were down, the assistant chose to SSH into the head Spark node (10.1.230.180) and run docker ps to see what containers were actually running. It then chained an additional SSH command to query the worker Spark (192.168.200.13) for the same information, separated by a --- delimiter.

The command itself is carefully constructed:

ssh aurora@10.1.230.180 'docker ps --format "{{.Names}} {{.Status}}" 2>/dev/null && echo "---" && ssh aurora@192.168.200.13 "docker ps --format \"{{.Names}} {{.Status}}\"" 2>/dev/null' 2>&1

Several design decisions are visible here. First, the assistant uses --format "{{.Names}} {{.Status}}" to produce clean, parseable output showing only container names and their running status — no extraneous columns. Second, it redirects stderr to /dev/null for the docker ps commands to suppress any warnings or connection errors that might clutter the output. Third, it uses && to chain the commands, meaning if the first docker ps fails, the entire chain stops — a deliberate choice that treats the head node's Docker daemon availability as a prerequisite for the rest of the check. Fourth, the assistant SSHs from the head Spark to the worker Spark through the head Spark, rather than running two separate SSH commands from the client machine — this is an efficiency choice that reduces network round-trips.

The Output: A Revealing Discovery

The output delivered a critical finding:

vllm_node Up 24 minutes
reranker Up 24 minutes
vllm_embeddings Up 24 minutes
---
vllm_node Up 24 minutes

The Qwen3.5 containers were gone. Instead, the old GLM-based services (vllm_node, reranker, vllm_embeddings) were running, having been started approximately 24 minutes prior. This was the same GLM-4.7-Flash setup that the assistant had stopped earlier in the session to free GPU memory for the Qwen3.5 deployment. Something — likely an auto-restart mechanism, a systemd service, or a manual intervention — had resurrected the old containers and replaced the Qwen3.5 deployment.

This discovery fundamentally changed the assistant's understanding of the system state. The API endpoint checks in <msg id=6732> that returned empty results were now explained: the old GLM containers were running on a different port or with different API paths than the Qwen3.5 deployment, so the curl to localhost:30000 (the port used for Qwen3.5) would naturally fail. The assistant wasn't dealing with a crashed service — it was dealing with a replaced service.

Assumptions and Their Consequences

Several assumptions underpinned this message and were implicitly tested by its results.

Assumption 1: The Qwen3.5 containers would still be running. The assistant had deployed the model, benchmarked it, and confirmed it was working. There was no reason to expect the containers had been removed. The docker ps output disproved this assumption, revealing that the infrastructure had state the assistant was not tracking.

Assumption 2: Docker was available and responsive on both nodes. The command structure with && chaining and stderr suppression suggests the assistant was reasonably confident in Docker's availability. This assumption held — both nodes responded cleanly.

Assumption 3: The head Spark could reach the worker Spark via SSH. The assistant used the IB subnet IP (192.168.200.13) for the inner SSH, which was correct — the InfiniBand/RoCE interconnect had been configured earlier and was functioning.

Assumption 4: The container names would be informative. The assistant chose to display container names, which proved to be the right decision. The names vllm_node, reranker, and vllm_embeddings immediately identified the services as the GLM stack rather than the Qwen3.5 deployment (which would have been named differently, likely vllm-qwen35 based on earlier messages).

Input Knowledge Required

To interpret this message fully, a reader needs to understand:

1. The DGX Spark topology: Two ARM-based Blackwell mini-servers connected via InfiniBand RoCE, with IPs 10.1.230.180 (external/management) and 192.168.200.x (interconnect subnet).
2. The deployment history: The assistant had previously deployed GLM-4.7-Flash on these Sparks, then replaced it with Qwen3.5-122B-A10B-FP8 using a custom Docker image (hellohal2064/vllm-qwen3.5-gb10). The old GLM containers had been stopped and removed.
3. The user's request: "relaunch the models" — an ambiguous command that could mean restart the current models, or restart all models including the RTX PRO 6000 setup.
4. Docker's behavior on these systems: The containers were running with specific names that identified their purpose.

Output Knowledge Created

This message produced several pieces of actionable knowledge:

1. The Qwen3.5 deployment was no longer active on either Spark node. The containers the assistant had launched were gone.
2. The old GLM services had been restored approximately 24 minutes ago, suggesting an automated restart mechanism (systemd, Docker restart policies, or a cron job).
3. The worker Spark had only vllm_node running, while the head Spark had three containers (vllm_node, reranker, vllm_embeddings). This asymmetry hinted at the original GLM deployment architecture where the head node ran additional services.
4. Both nodes were reachable and Docker was functional, ruling out connectivity or Docker daemon issues as the cause of the API check failures.
5. The 24-minute uptime provided a temporal anchor: whatever restarted the GLM containers happened roughly 24 minutes before the check, which was during or shortly after the Qwen3.5 benchmarking phase.

The Thinking Process

The assistant's reasoning, visible across messages <msg id=6732> and <msg id=6733>, follows a clear diagnostic pattern:

1. Direct check fails (API endpoint unreachable)
2. Escalate to infrastructure check (Docker container listing)
3. Interpret results (old containers running, new containers gone)
4. Form new hypothesis (auto-restart mechanism, systemd service) The assistant could have taken other paths. It could have tried to restart the Qwen3.5 containers directly, assuming they had crashed. It could have checked systemd services or Docker restart policies first. But the docker ps approach was the most efficient: it provided a complete picture of what was actually running with a single command, eliminating ambiguity about whether the services were crashed, stopped, or replaced.

Conclusion

Message <msg id=6733> exemplifies a fundamental principle of infrastructure debugging: when higher-level health checks fail, drop down to the lowest reliable diagnostic layer. The assistant's use of docker ps — the container runtime's equivalent of "is it plugged in?" — cut through the confusion of unreachable API endpoints and silent failures to reveal the true state of the system. The discovery that old GLM containers had replaced the Qwen3.5 deployment forced a strategic pivot: rather than restarting services, the assistant would need to understand why the old containers returned and prevent it from happening again. This message, in its simplicity, is a masterclass in diagnostic discipline — knowing when to stop guessing and start observing.