The Unified Memory Epiphany: Diagnosing an OOM on the DGX Spark

In the sprawling narrative of deploying a 122-billion-parameter Qwen3.5 model across two NVIDIA DGX Spark systems, message [msg 6613] marks a pivotal diagnostic pivot. The assistant has just attempted a multi-node SGLang launch spanning two GB10-based DGX Spark nodes, each equipped with 120GB of unified memory and connected via InfiniBand RoCE. The worker node on the second Spark has crashed with a "CUDA error: out of memory" during the init_torch_distributed call — an error that occurs before any model tensors are loaded. This message captures the assistant's reasoning as it untangles what initially appears to be a straightforward GPU memory shortage but turns out to be a more subtle consequence of the GB10's unified memory architecture.

The Context: A Multi-Node Deployment in Progress

To understand the significance of this message, we must step back and survey the broader deployment effort. The assistant has been orchestrating a complex multi-node deployment of the Qwen3.5-122B-A10B-FP8 model — a 119GB FP8 quantized MoE model with 256 experts — across two DGX Spark nodes. The head Spark (10.1.230.180) has just finished downloading the model from HuggingFace and rsyncing the full 127GB of model files to the second Spark (192.168.200.13) at ~640MB/s over the InfiniBand link. A custom Docker image (sglang-qwen35) has been built and transferred to both nodes. A launch script has been written, revised (to remove a spurious --language-model-only flag), and copied to both machines.

The deployment architecture uses SGLang's native multi-node support: the head node runs with --nnodes 2 --node-rank 0 --tp 2, the worker with --nnodes 2 --node-rank 1 --tp 2, both connecting via --dist-init-addr pointing to the head's InfiniBand IP. The assistant launched the worker first ([msg 6608]), then the head ([msg 6609]), waited 30 seconds, and checked the logs.

The head log ([msg 6610]) showed the model loading process proceeding normally — deprecation warnings about BaseImageProcessorFast, unrecognized rope_parameters keys, a DeepGemm accuracy warning about FP8 scale format on Blackwell — and then reaching "Init torch distributed begin." This is the point where the head waits for the worker to connect via NCCL distributed initialization. The worker, however, had already crashed.

The Initial Diagnosis: A Seemingly Straightforward OOM

In message [msg 6611], the assistant reads the worker's log and finds a traceback ending in init_torch_distributed() with an out-of-memory error. The immediate suspicion is the embeddings service. Earlier in the conversation, the user had instructed the assistant to keep the GLM embeddings service running on the secondary Spark node. The assistant checks with nvidia-smi and free in message [msg 6612], discovering:

The Core Reasoning in Message 6613

This is where message [msg 6613] begins. The assistant opens with a careful analysis of the memory situation:

Unified memory is mostly used — only 3.3GB free and 101GB "available" (buffer/cache reclaimable). The GPU shows ~11.7GB used by embeddings. But the "CUDA error: out of memory" might actually be about the CUDA context failing to allocate because system memory is tight. With GB10 unified memory, GPU memory IS system memory.

This statement reveals a critical architectural insight. The NVIDIA GB10 in the DGX Spark uses a unified memory architecture where the GPU does not have dedicated VRAM. Instead, GPU memory and system memory share the same 120GB LPDDR5X pool. This is fundamentally different from a traditional NVIDIA GPU with dedicated HBM or GDDR memory. When CUDA allocates memory, it competes with the operating system, Docker, and all running processes for the same physical memory.

The assistant then makes a difficult decision:

The embeddings on the second spark need to go — this model needs all available memory. The user said to keep embeddings running on secondary, but with 120GB unified memory and needing ~63GB for the model shard plus KV cache, we can't spare 12GB.

This is a moment of technical judgment overriding a prior instruction. The user explicitly asked to keep the embeddings service running on the secondary node. But the assistant correctly recognizes that the unified memory constraint makes this impossible. The embeddings are consuming 12GB of the shared 120GB pool, and the model needs every available byte. This is not a case of "try harder" — it is a hard physical constraint.

The assistant briefly considers a workaround:

Let me check if we can at least try with the embeddings running, by using a lower --mem-fraction-static:

This is a plausible mitigation. SGLang's --mem-fraction-static parameter controls what fraction of available GPU memory is reserved for model weights versus KV cache. Lowering it could reduce the memory pressure. But the assistant quickly abandons this line of reasoning.

The Deeper Insight: CUDA Context Allocation Failure

The assistant then re-examines the evidence more carefully:

Actually, looking more carefully at free — 18GB used, 3.3GB free, 99GB buffer/cache (which includes the just-rsynced 119GB model files in page cache). The real available is 101GB. The CUDA set_device call failing with OOM at setup time is very unusual — that's before any model loading. It might be that the Docker container itself can't allocate the CUDA context.

This is the key insight. The error occurs during init_torch_distributed(), which involves a torch.cuda.set_device() call — a CUDA context initialization step that happens before any model weights are loaded. The OOM at this stage is unusual because CUDA context creation typically requires only a few hundred megabytes. The fact that it's failing suggests something deeper: perhaps the Docker container's memory limit, or the CUDA driver's inability to reserve the necessary address space within the container's cgroup memory constraints.

The assistant correctly identifies that the 99GB of buffer/cache is mostly the freshly rsynced model files sitting in the Linux page cache. These are reclaimable by the kernel when memory pressure increases, but they still consume physical pages until actually reclaimed. The kernel's "available" metric (101GB) accounts for reclaimable cache, but the CUDA driver may not be aware of or able to wait for page cache reclamation during context initialization.

The Action Taken

The message concludes with the assistant executing cleanup:

Let me check if the embeddings process is blocking the CUDA device: [bash] ssh aurora@10.1.230.180 'docker rm -f sglang-qwen35-head 2>/dev/null; ssh aurora@192.168.200.13 "docker rm -f sglang-qwen35-worker 2>/dev/null" echo "Cleaned up failed containers"'

This is a practical first step: removing the failed containers to free any resources they might still hold (Docker networks, volumes, zombie processes). The assistant then proceeds to the next message ([msg 6614]) where it actually stops the embeddings service on the second Spark, confirming the diagnosis by observing GPU memory drop from ~11.7GB to near zero.

Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are sound:

  1. Unified memory means GPU memory IS system memory. This is correct for the GB10. Unlike traditional GPUs with dedicated HBM, the GB10's GPU shares the LPDDR5X pool. This has profound implications for memory management: nvidia-smi's memory usage numbers reflect allocations from the shared pool, and CUDA allocations can fail due to system memory pressure even when the GPU "should" have enough space.
  2. The embeddings must be stopped. This is a correct inference given the memory constraint. The model shard needs ~63GB, and with 120GB total, every gigabyte counts. The 12GB consumed by embeddings represents 10% of total memory — a significant fraction that could make the difference between a successful load and an OOM.
  3. The OOM at set_device is unusual. This is a valid observation. CUDA context creation typically requires modest memory. The failure at this stage points to either a systemic memory pressure issue (the page cache consuming physical pages that the CUDA driver cannot immediately reclaim) or a Docker cgroup memory limit being hit.
  4. Lowering --mem-fraction-static could help. This is technically correct — reducing the reserved fraction would leave more headroom. However, the assistant correctly abandons this approach because the fundamental issue is not KV cache sizing but CUDA context initialization, which happens before any model loading parameters take effect. One potential blind spot: the assistant does not explicitly check whether Docker has a memory limit on the container. The docker run command in the launch script does not include --memory or --memory-reservation flags, so the container should inherit the host's limits. But the GB10's unified memory means the host's "available" memory is what matters, and the 99GB of page cache could be causing the CUDA driver to fail when trying to pin memory for the GPU.

Input Knowledge Required

To fully understand this message, the reader needs:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. The embeddings service must be stopped on the second Spark to free ~12GB of unified memory. This is a hard requirement, not an optimization.
  2. The OOM at CUDA context initialization is a diagnostic signal that points to systemic memory pressure rather than a simple "not enough GPU memory" scenario. Future deployments on GB10 should monitor system memory (via free) as closely as GPU memory (via nvidia-smi).
  3. Page cache from large file transfers can interfere with CUDA initialization. The 99GB of buffer/cache from the rsync may be causing the CUDA driver to fail during context creation. A sync; echo 3 > /proc/sys/vm/drop_caches before launching could mitigate this.
  4. The workaround of lowering --mem-fraction-static is insufficient because the failure occurs before model loading parameters take effect.

The Thinking Process Visible in the Reasoning

The assistant's reasoning in this message follows a clear diagnostic arc:

  1. Observe the symptom: Worker OOM during init_torch_distributed
  2. Gather data: nvidia-smi shows 11.7GB used by embeddings; free shows 3.3GB free, 99GB buffer/cache, 101GB available
  3. Form initial hypothesis: Embeddings consuming too much memory
  4. Consider mitigation: Lower --mem-fraction-static
  5. Re-examine evidence: The OOM at set_device is unusual — it happens before model loading
  6. Refine hypothesis: The CUDA context itself can't allocate, possibly due to Docker or page cache pressure
  7. Take action: Clean up failed containers as first step This is a textbook diagnostic process: gather data, form hypothesis, test against domain knowledge, refine, act. The pivot from "embeddings are the problem" to "the CUDA context allocation itself is failing" shows the assistant's ability to distinguish between different failure modes based on when in the initialization sequence the error occurs.

Conclusion

Message [msg 6613] is a compact but rich example of systems-level diagnostic reasoning in the context of deploying large language models on non-traditional hardware. The assistant demonstrates deep understanding of the GB10's unified memory architecture, correctly identifies that a prior user instruction must be overridden due to physical constraints, and shows the ability to distinguish between different OOM failure modes based on timing. The message sets up the subsequent successful deployment (after stopping the embeddings service) and serves as a case study in the subtle ways that unified memory architectures differ from traditional discrete GPU setups. For anyone deploying large models on DGX Spark or similar unified-memory systems, the lesson is clear: system memory pressure can manifest as GPU OOM errors, and the Linux page cache from model file transfers can interfere with CUDA context initialization in unexpected ways.