The OOM Diagnosis: A Pivotal Moment in Multi-Node LLM Deployment

In the sprawling narrative of deploying the Qwen3.5-122B-A10B-FP8 model across two NVIDIA DGX Spark nodes, one message stands out as a diagnostic turning point. At index 6717 in the conversation, the assistant delivers a concise but information-dense analysis that transforms a cryptic failure into a tractable engineering problem. The message reads:

OOM on the head node! Ray killed the worker because memory usage hit 113.78GB / 119.70GB (95.05%). The model loaded successfully (it got all the way to CUDA graph profiling with 49.12 GiB KV cache available on the worker), but then CUDA graph capture on the head node exhausted memory and Ray killed it.

>

The key data: Available KV cache memory: 49.12 GiB — this means the model loaded and there was 49GB free for KV cache before CUDA graphs. But CUDA graph capture needs additional memory.

>

The fix: set RAY_memory_usage_threshold higher (or disable it), and reduce --gpu-memory-utilization slightly, or disable CUDA graphs.

This message represents the culmination of a multi-hour debugging session spanning networking, distributed systems, and memory management. To understand its significance, we must trace the journey that led to this moment.

The Long Road to This Diagnosis

The assistant had been attempting to deploy the Qwen3.5-122B-A10B-FP8 model—a 119-billion-parameter mixture-of-experts model quantized to FP8—across two DGX Spark machines connected via InfiniBand. The DGX Spark is NVIDIA's compact desktop AI supercomputer, featuring a GB10 Blackwell GPU with 120GB of unified memory and an ARM Cortex-X925 CPU. Each Spark had one GPU, so tensor parallelism across both nodes was essential.

The deployment had already survived multiple crises. The assistant had pivoted from SGLang to a custom vLLM 0.17.1rc1 image after discovering that SGLang's multi-node NCCL initialization hung indefinitely. It had wrestled with Ray's networking layer, discovering that the head node was registering itself with its external IP (10.1.230.180) while the worker node could only reach the head via the InfiniBand subnet (192.168.200.x). This caused PyTorch's c10d distributed backend to fail when the worker tried to connect to the unreachable external IP. The fix required forcing Ray to use --node-ip-address with the IB subnet address, setting GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME to the correct RoCE interface, and ensuring VLLM_HOST_IP matched the IB address.

After solving the networking puzzle, the assistant had successfully launched the vLLM serve process and watched it load model weights for approximately 12 minutes. The log showed Loading safetensors checkpoint shards: 21% (8/39) progressing at about 18 seconds per shard. Both GPUs were consuming memory. Then, at the 10-minute mark, the process crashed with the opaque error: "Engine core initialization failed."

The Diagnostic Breakthrough

The assistant's first attempt to diagnose the crash used a broad grep for error-related keywords, which returned only noise—Ray internal process listings and metrics exporter warnings. This is a common pitfall in distributed debugging: the root cause is often buried beneath layers of secondary errors and informational messages.

The assistant then pivoted to a more targeted approach, searching specifically for memory-related indicators. The breakthrough came from examining the log output more carefully. The assistant noticed two critical data points:

1. Memory usage at time of death: 113.78GB out of 119.70GB total system memory, hitting exactly 95.05%.
2. KV cache availability: The log showed Available KV cache memory: 49.12 GiB on the worker node, indicating the model had loaded successfully and there was substantial free GPU memory for the KV cache. The 95.05% figure was the smoking gun. Ray has a built-in memory monitor that kills worker processes when system memory usage exceeds a configurable threshold—defaulting to 95%. The model weights had loaded, the KV cache had been allocated, but then CUDA graph capture began. CUDA graphs are a performance optimization that pre-records sequences of GPU operations, allowing them to be launched with minimal CPU overhead. However, the graph capture process itself requires additional temporary memory allocations, and on the head node—which was already near capacity—this pushed memory usage past the 95% threshold, triggering Ray's OOM killer.

The Reasoning Process

The assistant's reasoning in this message demonstrates a sophisticated understanding of the vLLM lifecycle and Ray's resource management. The key insight is the distinction between the model loading phase and the CUDA graph capture phase. The model had loaded successfully—the 49.12 GiB KV cache allocation proved that. But CUDA graph capture is a separate, later stage that requires additional memory headroom.

The assistant correctly identifies that the problem is not insufficient GPU memory for inference (49GB of KV cache would have been more than adequate for most workloads), but rather a transient memory spike during initialization. This is a subtle but important distinction. Many engineers might have concluded that the model was simply too large for the available memory and given up. Instead, the assistant recognized that the memory pressure was temporary and could be mitigated by adjusting system parameters.

The proposed fixes reveal a tiered approach to the solution:

1. Increase RAY_memory_usage_threshold or disable the monitor: This addresses the immediate cause—Ray killing the process. By raising the threshold or setting RAY_memory_monitor_refresh_ms=0 (which disables the monitor entirely), the process would be allowed to complete CUDA graph capture even if memory temporarily exceeds 95%.
2. Reduce --gpu-memory-utilization: This reduces the amount of GPU memory reserved for the KV cache, leaving more headroom for CUDA graph capture. The trade-off is smaller KV cache capacity, which could reduce the maximum sequence length or batch size.
3. Disable CUDA graphs: This eliminates the memory spike entirely but sacrifices the performance optimization that CUDA graphs provide. The assistant chose to apply the first two fixes in combination, as shown in the subsequent messages ([msg 6718] and [msg 6719]), which edited the launch script to set RAY_memory_monitor_refresh_ms=0 and reduce GPU memory utilization.

Assumptions and Knowledge Required

This message relies on several pieces of implicit knowledge:

• Ray's memory monitor behavior: The assistant knows that Ray has a default 95% memory threshold and that this can be overridden. This is not obvious from the error message alone, which simply says "Engine core initialization failed."
• The vLLM initialization sequence: The assistant understands that model loading and CUDA graph capture are distinct phases, and that graph capture happens after weight loading. This knowledge is essential for interpreting the timing of the OOM event.
• CUDA graph memory characteristics: The assistant knows that CUDA graph capture requires additional temporary memory beyond what the final graph occupies. This is a relatively obscure detail about CUDA graph compilation.
• The DGX Spark memory architecture: The 120GB unified memory on the GB10 GPU is shared between CPU and GPU. The assistant correctly interprets the system memory usage (113.78GB / 119.70GB) as including both model weights and KV cache allocations.
• The relationship between system memory and GPU memory: On the DGX Spark's unified memory architecture, GPU allocations consume system memory. The assistant understands that nvidia-smi GPU memory figures and system free memory are measuring the same pool. One assumption the assistant makes is that the OOM is purely a memory pressure issue and not a bug in CUDA graph capture for the Blackwell architecture. This turns out to be correct—after applying the fixes, the model loads and serves successfully ([msg 6721] shows the API server routes being registered).

Output Knowledge Created

This message creates several pieces of actionable knowledge:

1. A precise diagnosis: The root cause is identified as Ray's OOM killer triggering during CUDA graph capture, not a fundamental incompatibility or model loading failure.
2. A validated debugging methodology: The assistant demonstrates the importance of looking past surface-level errors ("Engine core initialization failed") to find the actual cause. The approach of checking memory utilization percentages and KV cache availability is a reusable pattern.
3. A set of concrete fixes: Three options are presented, with the first two being applied. The subsequent messages confirm that these fixes work.
4. A performance characteristic: The 49.12 GiB KV cache figure provides a baseline for understanding the model's memory footprint on the DGX Spark hardware.

The Broader Significance

This message exemplifies the kind of diagnostic reasoning that separates effective infrastructure engineering from trial-and-error debugging. The assistant doesn't just report that something failed—it interprets the failure in the context of the system's architecture, identifies the specific phase of initialization where the failure occurred, and proposes targeted interventions.

The message also highlights the challenges of deploying large language models on memory-constrained hardware. The DGX Spark's 120GB unified memory is generous by consumer standards, but a 119B parameter model in FP8 still consumes approximately 60GB for weights alone (at 4 bits per parameter for the active parameters, plus the full-precision embeddings and output layers). When combined with KV cache allocations and CUDA graph overhead, the margin for error is razor-thin.

The assistant's decision to disable the Ray memory monitor rather than reduce GPU memory utilization or disable CUDA graphs reflects a pragmatic engineering trade-off. Disabling the monitor carries no performance penalty, while reducing GPU memory utilization would limit the model's serving capacity, and disabling CUDA graphs would increase per-request latency. By choosing the least invasive fix first, the assistant preserves the deployment's performance characteristics while eliminating the crash.

In the subsequent messages, we see the fix succeed: the model loads, the API server starts, and the deployment achieves approximately 27 tok/s single-request throughput with correct reasoning output. The OOM diagnosis at message 6717 was the critical turning point that made this success possible.