The Cleanup Before Creation: A Single Bash Command's Role in Multi-Node AI Deployment

Introduction

In the sprawling, multi-session saga of deploying large language models across heterogeneous GPU infrastructure, most of the dramatic action happens in the visible moments: the model loading, the benchmark numbers, the triumphant "server is UP" announcement. But between those peaks lies a vast terrain of unglamorous, essential work—the cleanup operations that clear the way for what comes next. Message [msg 6737] in this opencode conversation is a masterclass in that hidden labor: a single bash command, executed over SSH, that systematically dismantles an entire inference stack across two DGX Spark nodes to prepare for a fresh deployment. This message, on its surface a mundane teardown script, reveals deep truths about infrastructure management, the assumptions embedded in automation, and the invisible scaffolding that supports cutting-edge AI serving.

The Message: What Was Actually Said

The assistant executes the following command via a bash tool:

ssh aurora@10.1.230.180 'echo "=== Stop old services ===" && \
  sudo systemctl stop vllm-cluster.service vllm-proxy.service \
    vllm-embeddings.service vllm-reranker.service 2>/dev/null; \
  docker rm -f vllm_node reranker vllm_embeddings 2>/dev/null; \
  sleep 3 && nvidia-smi | grep -E "MiB|%" && \
  echo "=== Worker ===" && \
  ssh aurora@192.168.200.13 \
    "docker ps -q | xargs -r docker rm -f 2>/dev/null; sleep 2; \
     nvidia-smi | grep -E MiB\\|%"' 2>&1

The output confirms success: on the head node, GPU memory shows only 18MiB used by Xorg and 6MiB by gnome-shell—essentially zero GPU utilization by inference workloads. The worker node similarly shows only 18MiB for Xorg. The old services are dead, the containers are gone, and the GPUs are pristine.

The Context: Why This Message Exists

To understand why this message was written, we must trace the chain of events that led to it. The conversation's broader arc (Segment 42) had just completed an extraordinary engineering feat: deploying the Qwen3.5-122B-A10B-FP8 model—a 119-billion-parameter Mixture-of-Experts reasoning model—across two NVIDIA GB10 DGX Spark systems using multi-node vLLM with Ray. This deployment had required solving a cascade of thorny problems: SGLang's official image was too old for Qwen3.5 support; SGLang's multi-node NCCL initialization hung indefinitely; Ray's auto-detection used the wrong IP address, making the worker node unreachable; and Ray's OOM killer terminated the process during CUDA graph capture because memory usage hit 95% of the 120GB unified memory pool.

After the deployment succeeded and benchmarks showed ~27 tok/s single-request throughput, the user asked simply: "relaunch the models" ([msg 6731]). The assistant dutifully checked both systems—the DGX Sparks and the RTX PRO 6000 LXC container—and discovered that the Sparks were running the old GLM-4.7-Flash containers again, not the freshly deployed Qwen3.5. The RTX host was unreachable. The user then clarified: "Just on the sparks, those were rebooted" ([msg 6736]).

This clarification is the key that unlocks the reasoning behind message [msg 6737]. The DGX Sparks had been rebooted. A reboot means the running state is lost—Docker containers are gone, systemd services that weren't enabled for auto-start are stopped, and any in-memory model weights are cleared. But the Sparks were not starting from a clean slate: they had old GLM services running again, likely because those services had been configured with systemctl enable and auto-started after the reboot. The freshly deployed Qwen3.5 containers, launched manually without systemd integration, were lost in the reboot.

So the assistant faced a specific situation: the Sparks had rebooted, the old GLM stack had resurrected itself via systemd auto-start, and the new Qwen3.5 stack was gone. Before the assistant could redeploy Qwen3.5, it needed to tear down the GLM infrastructure that had come back to life. Message [msg 6737] is that teardown.

The Reasoning: A Systematic Cleanup Strategy

The command reveals a carefully structured, multi-phase cleanup strategy. The assistant does not simply docker rm -f everything; it proceeds in deliberate stages.

Phase 1: Stop systemd services. The command begins by stopping four systemd units: vllm-cluster.service, vllm-proxy.service, vllm-embeddings.service, and vllm-reranker.service. This is the correct order of operations—stopping the service manager units before removing containers ensures that systemd doesn't try to restart containers that are about to be deleted, and that any service-level cleanup (socket closures, health check termination) happens cleanly. The 2>/dev/null suppression of stderr is a pragmatic choice: if a service doesn't exist or is already stopped, the error messages are noise, not signal.

Phase 2: Force-remove Docker containers. Three containers are targeted: vllm_node, reranker, and vllm_embeddings. The -f flag ensures removal even if containers are running. The names reveal the architecture of the old GLM stack: a primary vLLM inference container, a reranker container (likely for retrieval-augmented generation or response scoring), and an embeddings container (for vector embeddings). This is a non-trivial inference pipeline, not a single model server.

Phase 3: Wait and verify. The sleep 3 is a deliberate pause—it gives the system time to release GPU memory after container destruction. Then nvidia-smi | grep -E "MiB|%" checks that GPU memory has actually been freed. This is the verification step: the assistant is not assuming success; it is confirming it.

Phase 4: Clean the worker node. The command then SSHes into the second Spark node (192.168.200.13) and performs a more aggressive cleanup: docker ps -q | xargs -r docker rm -f removes all running containers, not just specific named ones. This is a reasonable strategy for the worker, which may have orphaned containers from the reboot or from the old GLM setup. The -r flag on xargs prevents execution if no containers are running, avoiding an unnecessary error. Another sleep 2 and nvidia-smi check confirms the worker's GPU is also clean.

Assumptions Embedded in the Command

Every automation carries assumptions, and this message is rich with them.

Assumption 1: The head node is 10.1.230.180 and the worker is 192.168.200.13. This reflects the networking architecture established earlier in the session, where the head node's external IP is used for SSH access but the internal InfiniBand subnet (192.168.200.x) is used for inter-node communication. The assistant assumes this network topology is stable across the reboot.

Assumption 2: The old services are named exactly as specified. The service names (vllm-cluster.service, etc.) and container names (vllm_node, reranker, vllm_embeddings) are hardcoded. If the reboot had changed service names or if the GLM stack used different naming conventions, these commands would silently fail (due to 2>/dev/null). The assistant is relying on prior knowledge from earlier exploration of the Spark nodes.

Assumption 3: The worker node is reachable and has the same SSH key setup. The assistant SSHes from the head node to the worker using aurora@192.168.200.13. This assumes passwordless SSH is configured between the two Sparks, which is reasonable for a multi-node Ray cluster but not guaranteed.

Assumption 4: GPU memory should be near-zero after cleanup. The verification step checks for lines matching "MiB" or "%" in nvidia-smi output. The assistant expects to see only Xorg and gnome-shell using minimal memory. This assumes no other GPU-using processes exist (no CUDA daemons, no monitoring tools, no other users). In a shared environment, this could be a dangerous assumption.

Assumption 5: The user wants Qwen3.5 redeployed specifically. The user said "relaunch the models" and then clarified "Just on the sparks, those were rebooted." The assistant infers that the goal is to restore the Qwen3.5 deployment that was running before the reboot, not to reinstall the old GLM stack. This is a reasonable but unstated inference.

Input Knowledge Required

To understand this message fully, a reader needs substantial context:

  1. The hardware topology: Two DGX Spark nodes (NVIDIA GB10, ARM Cortex-X925, 120GB unified memory, InfiniBand RoCE interconnect) connected via a 192.168.200.x subnet for inter-node communication, with the head node also accessible at 10.1.230.180.
  2. The previous deployment: The assistant had just finished deploying Qwen3.5-122B-A10B-FP8 using a custom vLLM image (hellohal2064/vllm-qwen3.5-gb10) with Ray multi-node, after solving numerous networking and memory issues. The old GLM-4.7-Flash stack was still present in systemd configuration.
  3. The reboot event: The user stated the Sparks were rebooted, which means the Qwen3.5 containers (launched manually without systemd integration) were lost, but the old GLM services (configured with systemctl enable) auto-restarted.
  4. The user's intent: "relaunch the models" implies restoring inference service, and "Just on the sparks" narrows the scope to the DGX Spark nodes only, excluding the unreachable RTX PRO 6000 setup.
  5. Linux system administration: Understanding systemd, Docker, SSH, and GPU memory management is necessary to parse the command's structure and intent.

Output Knowledge Created

This message produces several important outputs:

  1. A clean state on both Spark nodes: All GLM-related services are stopped, all containers are removed, and GPU memory is verified as free. This is the prerequisite state for any fresh deployment.
  2. Confirmation of the reboot's effects: The nvidia-smi output confirms that the GPUs are idle and that no inference workloads survived the reboot. The Xorg and gnome-shell memory usage (18MiB and 6MiB respectively) is negligible—these are display server overheads, not model weights.
  3. Documentation of the old stack's architecture: The service and container names reveal that the old GLM deployment used a three-component architecture: a primary vLLM node, a reranker, and an embeddings service. This is valuable architectural knowledge for anyone maintaining this system.
  4. A verified foundation for the next deployment: The assistant can now proceed to redeploy Qwen3.5 without worrying about port conflicts, GPU memory fragmentation, or service interference from the old stack.

Mistakes and Potential Pitfalls

While the command achieves its goal, several aspects warrant scrutiny.

The silent error suppression is a double-edged sword. By redirecting 2>/dev/null for both systemctl stop and docker rm -f, the assistant loses visibility into failures. If a service unit file was corrupted or a container was already removed, the command would proceed without warning. In a production environment, this could mask problems that would surface later. A better approach might be to log stderr to a file for post-hoc inspection.

The worker cleanup is overly aggressive. docker ps -q | xargs -r docker rm -f removes all containers on the worker, not just the GLM-related ones. If the worker had other containers running (monitoring agents, data pipelines, etc.), they would be destroyed. The assistant assumes the worker only runs inference containers, which is reasonable given the dedicated nature of these DGX Spark nodes but is not explicitly verified.

No check for systemd enablement. The assistant stops the services but does not disable them (systemctl disable). This means that if the Sparks reboot again before the Qwen3.5 deployment is fully integrated with systemd, the old GLM services would restart once more. The assistant is treating the symptom (running services) rather than the root cause (systemd auto-start configuration).

The verification is superficial. Checking nvidia-smi for "MiB" lines confirms that GPU memory is low, but it does not verify that the old model weights are actually gone from memory—it only checks the current allocation. CUDA contexts or lingering processes could hold memory that nvidia-smi reports as "used" but that isn't captured by the grep pattern. A more thorough check would use nvidia-smi --query-gpu=memory.used --format=csv or check for remaining Docker networks and volumes.

The Deeper Significance

Message [msg 6737] is, on its surface, a routine cleanup operation. But it illuminates several profound truths about AI infrastructure management.

First, the lifecycle of AI services is often more complex than their deployment. The assistant spent dozens of messages deploying Qwen3.5—solving networking issues, memory problems, and model compatibility. But the teardown of the old GLM stack, which was a prerequisite for that deployment, happens in a single command. The asymmetry is striking: building up requires iterative debugging and creativity; tearing down is mechanical but essential.

Second, system state persistence across reboots is a design decision, not a given. The old GLM stack survived the reboot because someone had configured it with systemctl enable. The new Qwen3.5 stack did not survive because it was launched ad-hoc. This message implicitly critiques that asymmetry—the assistant is forced to manually undo what systemd automatically restored. The lesson is that production deployments should be integrated with the service manager from day one.

Third, the assistant demonstrates a sophisticated understanding of operational sequencing. The order of operations—stop services, remove containers, wait, verify, clean worker, verify again—reflects a mental model of dependency chains and state verification that is characteristic of experienced infrastructure engineers. This is not a novice blindly running commands; it is a systematic operator who understands that cleanup must be methodical to be reliable.

Finally, the message reveals the hidden labor of multi-node orchestration. Managing two machines requires SSH, coordination, and parallel verification. The assistant must think about both nodes simultaneously, ensuring that the head and worker are both clean before proceeding. This dual-node awareness is the essence of distributed systems management, and it is on full display in this single, dense command.

Conclusion

Message [msg 6737] is a small but revealing window into the operational reality of deploying large language models across distributed GPU infrastructure. A single bash command, executed over SSH, encapsulates hours of prior debugging, deep knowledge of the system's architecture, and a methodical approach to state management. It demonstrates that in AI infrastructure, the invisible work of teardown and cleanup is just as important as the visible work of deployment and benchmarking. The assistant's systematic approach—stop services, remove containers, verify memory, repeat on worker—is a template for how to reliably manage state across multi-node systems. And the assumptions embedded in the command remind us that every automation, no matter how routine, carries the weight of its context: the network topology, the service naming conventions, the reboot history, and the user's unspoken expectations. In the end, this message is not just about stopping services; it is about creating the conditions for the next act of creation.