The Verification That Unlocks Deployment: A Post-Cleanup Health Check Across Two DGX Spark Nodes

In the intricate dance of multi-node AI inference deployment, no step is more critical than the moment of verification after cleanup. Message [msg 6618] captures exactly such a moment: a single bash command executed by the assistant to confirm that GPU memory has been successfully freed on a remote DGX Spark node, paving the way for a fresh attempt at deploying the Qwen3.5-122B-A10B-FP8 model across two nodes. The message is deceptively simple — a one-liner SSH command with its output — but it represents the culmination of a multi-step debugging process and the gateway to a successful deployment.

The Message

ssh -J aurora@10.1.230.180 aurora@192.168.200.13 'nvidia-smi && free -h | head -2'

Output:

Thu Apr  9 00:48:14 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|       ...

Why This Message Was Written: The Context of Failure

To understand the motivation behind this seemingly trivial health check, we must trace the events that led to it. The assistant had been engaged in a complex multi-node deployment of the Qwen3.5-122B-A10B-FP8 model — a 119-billion-parameter Mixture-of-Experts model quantized to FP8 — across two NVIDIA DGX Spark systems. These compact ARM-based workstations each feature a single NVIDIA GB10 GPU with 120GB of unified memory, connected via InfiniBand-over-Ethernet (RoCE) for high-speed inter-node communication.

The deployment strategy was straightforward: use SGLang's native multi-node support with --nnodes 2 and --tp 2 (tensor parallelism across two nodes), where each node holds one shard of the model. The assistant had successfully rsynced the 119GB model to the second Spark at ~640MB/s over the InfiniBand link ([msg 6592]). Launch scripts were prepared and distributed. The worker was started on the second Spark ([msg 6602]), then the head on the first Spark ([msg 6603]).

But the launch failed. When the assistant checked the worker logs ([msg 6611]), it found a CUDA out-of-memory error during init_torch_distributed — a failure occurring before any model weights were even loaded. This is an unusual failure point. Typically, OOM errors happen during model loading or KV cache allocation, not during distributed initialization. The root cause was a pre-existing vLLM embeddings service and a reranker process running on the second Spark, consuming approximately 11.7GB of GPU memory ([msg 6612]). On a DGX Spark with unified memory architecture, where GPU memory and system memory share the same 120GB pool, every gigabyte counts. The CUDA context allocation for the new SGLang process was failing because the memory was already occupied.

The Cleanup Campaign

What followed was a methodical cleanup operation spanning messages [msg 6613] through [msg 6617]. The assistant first tried systemctl stop vllm-embeddings.service, then docker stop and docker rm on the embeddings container. When the GPU memory didn't fully clear, the assistant identified a persistent python3 rerank_server.py process (PID 2563) running as root, consuming 7.8GB. Killing this process required escalating to sudo kill -9, which itself required a password — a complication on the jump-host SSH setup. The assistant ultimately resorted to docker kill $(docker ps -q) to terminate all running containers, which succeeded ([msg 6617]).

The Verification: Message 6618

Message [msg 6618] is the verification step — the assistant's way of saying "did my cleanup actually work?" before proceeding to the next attempt. The command uses SSH jump host syntax (-J) to reach the second Spark (192.168.200.13) through the first Spark (10.1.230.180), reflecting the network topology where only the head node is directly accessible from the assistant's environment. The command runs two diagnostic tools: nvidia-smi to check GPU memory usage, and free -h to check system memory.

The output tells a nuanced story. The nvidia-smi output shows GPU 0 with 0 MiB used out of 95,896 MiB — all GPU memory is free. The cleanup succeeded. However, the free -h output (partially visible in the truncated display) tells a different story: system memory shows only ~3.2Gi free with ~1.2Gi available. The "available" figure is critically low — on a unified memory system, the GPU and CPU compete for the same physical memory. The 99Gi shown as buffer/cache includes the 119GB model files that were just rsynced and are sitting in the Linux page cache. This means that while the GPU is technically free, the system is under memory pressure. The assistant would likely need to drop the page cache (echo 3 > /proc/sys/vm/drop_caches) or accept that some of the 119GB model files will need to be re-read from disk when the model loads.

Assumptions and Their Validity

The assistant makes several assumptions in this message. First, it assumes that nvidia-smi accurately reflects GPU memory availability on a unified memory architecture. This is generally true — nvidia-smi reports the GPU-allocated portion of unified memory — but it doesn't capture the full picture. On GB10, memory allocated by CUDA drivers, NCCL buffers, and the PyTorch runtime may not appear in nvidia-smi until the process actually starts allocating tensors. The OOM during init_torch_distributed in the previous attempt suggests that CUDA context creation itself can fail even when nvidia-smi shows free memory, because the CUDA runtime needs to reserve internal buffers.

Second, the assistant assumes that freeing GPU memory is sufficient to resolve the OOM. This is correct as a first-order fix — the 11.7GB consumed by the embeddings service was the primary obstacle. However, the low system memory availability (1.2Gi "available" on a 120GB system) hints at deeper issues. The model shard requires approximately 63GB (half of 119GB for TP=2), plus CUDA workspace, NCCL buffers, and KV cache. With only 1.2Gi of genuinely available memory (after accounting for buffer/cache), the system may still struggle.

Third, the assistant assumes that the jump-host SSH setup (-J) is the correct way to reach the second Spark. This is a reasonable networking assumption given the topology — the second Spark is on a private InfiniBand subnet (192.168.200.x) that is not directly routable from the assistant's environment.

Input Knowledge Required

To fully understand this message, the reader needs several pieces of contextual knowledge:

  1. DGX Spark architecture: These are ARM-based (Cortex-X925) systems with a single NVIDIA GB10 GPU and 120GB of unified memory, where GPU and system memory share the same physical RAM pool. This is fundamentally different from discrete GPU setups where GPU VRAM and system RAM are separate.
  2. SSH jump host syntax: The -J flag in SSH creates a chain through an intermediate host. Here, aurora@10.1.230.180 is the head DGX Spark (publicly accessible), and aurora@192.168.200.13 is the second Spark on the private InfiniBand subnet.
  3. nvidia-smi interpretation: The output shows GPU memory usage, process list, and compute mode. The key metric is the "Memory-Usage" column showing MiB used vs total.
  4. Linux memory management: The free -h output shows total, used, free, shared, buffer/cache, and available memory. The "available" field is the most relevant — it estimates memory available for new allocations without swapping.
  5. Previous failure context: The worker OOM during init_torch_distributed ([msg 6611]) is the reason this verification is needed. The assistant had attempted to launch SGLang multi-node and hit a CUDA OOM error before model loading even began.

Output Knowledge Created

This message produces several critical pieces of information:

  1. GPU memory is completely free: 0 MiB out of 95,896 MiB used. The cleanup was successful in removing the embeddings and reranker processes.
  2. System memory is tight: With only ~1.2Gi available and 99Gi in buffer/cache (the rsynced model files), the system is under memory pressure. The assistant may need to drop page caches or accept that model loading will trigger disk I/O to fault in the model files.
  3. No other GPU processes remain: The nvidia-smi process list (not fully shown in the truncated output) would confirm that no other CUDA processes are running.
  4. The path forward is clear: With GPU memory freed, the assistant can proceed to re-launch the SGLang worker on the second Spark, now with a reasonable expectation of success.

The Thinking Process

The reasoning behind this message reflects a methodical debugging approach. The assistant follows a classic "identify cause → apply fix → verify fix → retry" cycle. The OOM during distributed initialization was unexpected and pointed to a resource contention issue rather than a model size issue. By tracing the GPU memory usage on the second Spark, the assistant identified the embeddings and reranker processes as the culprits. After attempting multiple cleanup methods (systemd, docker stop/rm, kill -9), the assistant finally succeeded with docker kill. Now, instead of blindly retrying the launch, the assistant pauses to verify that the fix actually worked. This verification step prevents wasted time on a launch that would fail for the same reason.

The choice of nvidia-smi && free -h is deliberate: nvidia-smi checks the GPU-specific memory, while free -h checks the system-level memory. On a unified memory architecture, both are relevant. The && ensures that free only runs if nvidia-smi succeeds, which is a sensible ordering — if the GPU check fails (e.g., the SSH connection drops), there's no point checking system memory.

Significance in the Larger Narrative

Message [msg 6618] is a hinge point in the deployment story. It marks the transition from debugging (why did the worker OOM?) to execution (now let's launch again). The verification confirms that the obstacle has been removed, allowing the assistant to proceed with confidence. In the subsequent messages, the assistant would re-launch the SGLang worker and head, successfully deploy the model, and achieve ~27 tok/s single-request throughput with correct reasoning output — a significant milestone in the multi-node deployment effort.

This message exemplifies a principle that experienced systems engineers know well: never assume a fix worked — verify it. The 15 seconds spent running nvidia-smi and free saved potentially 15 minutes of debugging a launch that would have failed for the same reason. It's a small investment with outsized returns, and it's the mark of a disciplined approach to infrastructure management.