The Silence After Init: Diagnosing a Distributed Hang in SGLang's Model Parallel Initialization

In the course of deploying a large language model across multiple GPUs, few moments are as frustrating as the silent hang — a process that starts, prints a log message, and then simply stops responding, consuming resources without making progress. Message [msg 6179] captures precisely such a moment. The assistant, having spent the preceding messages fighting driver mismatches, NCCL configuration issues, and topology changes, arrives at a critical diagnostic narrowing: the distributed process group has initialized successfully, yet the server remains stuck. This message represents a pivot point in the debugging process, where the assistant reframes the problem from "NCCL/torch distributed init is hanging" to "the hang is in initialize_model_parallel or after."

The Context: A Model Too Large for One GPU

To understand this message, one must appreciate the broader context. The assistant is deploying Qwen3.5-122B-A10B, a 122-billion-parameter Mixture-of-Experts model in BF16 precision, consuming approximately 234 GB of memory. This far exceeds the 96 GB capacity of a single RTX PRO 6000 Blackwell GPU, necessitating tensor parallelism (TP=4) across four GPUs. The system had recently been reconfigured from an 8-GPU topology, splitting the Blackwell GPUs between an LXC container (running SGLang) and a SEV-SNP VM. This reconfiguration introduced a cascade of issues: a driver version mismatch (container userspace 565 vs. host kernel 590), P2P DMA corruption under IOMMU translation (requiring NCCL_P2P_DISABLE=1), and now a mysterious hang during distributed initialization.

The preceding messages ([msg 6153] through [msg 6178]) document a thorough diagnostic campaign. The assistant has:

• Verified NCCL init completes successfully with NCCL_DEBUG=INFO
• Used strace to confirm all four TP workers are blocked on futex and restart_syscall (socket reads)
• Inspected socket connections, confirming NCCL rendezvous sockets are established
• Located the relevant source code in model_runner.py:744 showing the init_torch_distributed method
• Confirmed the TCPStore port (35201) is listening and all ranks have established connections

The Message: A Diagnostic Reframing

The subject message reads in full:

All 4 ranks are connected to the TCPStore. So init_process_group actually completed — the hang is in initialize_model_parallel or after. Let me check the log one more time — maybe new output appeared:

>

[bash] ssh root@10.1.230.174 'wc -l /tmp/sglang_122b_v3.log; tail -3 /tmp/sglang_122b_v3.log; echo "---"; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader'

>

16 /tmp/sglang_122b_v3.log [2026-03-09 12:32:57 TP1] Mamba selective_state_update backend initialized: triton [2026-03-09 12:32:57 TP1] Init torch distributed begin. [2026-03-09 12:32:57 TP0] sglang is using nccl==2.29.3 --- 0, 1125 MiB 1, 1133 MiB 2, 1133 MiB 3, 1045 MiB

At first glance, this appears to be a routine check — run wc -l, tail, and nvidia-smi. But the reasoning that precedes the command is the heart of the message. The assistant has just examined the TCPStore connections ([msg 6178]) and found that all four ranks have established connections to the store on port 35201. This observation carries a crucial implication: init_process_group has already succeeded.

The Reasoning: What TCPStore Connections Tell Us

The assistant's chain of reasoning deserves careful examination. In PyTorch's distributed package, init_process_group performs several steps: it sets up a TCPStore-based rendezvous where ranks exchange metadata, establishes NCCL communicators, and synchronizes the group. If all four ranks are connected to the TCPStore with established sockets, the rendezvous phase has completed. NCCL communicator creation may still be in progress, but the earlier NCCL_DEBUG test ([msg 6157]) showed NCCL init completing successfully within 60 seconds. The combination of these two observations — NCCL init succeeded and TCPStore connections are established — logically implies that init_process_group is not the bottleneck.

This is a classic debugging technique: proving what is NOT the problem by gathering convergent evidence. The assistant has two independent data points pointing to the same conclusion, allowing the search space to be narrowed. The hang must lie downstream — in initialize_model_parallel (which sets up tensor-parallel communication groups within the model) or in the weight loading / kernel compilation that follows.

The Assumptions Embedded in the Reasoning

The assistant makes several assumptions, most of them well-founded but worth examining:

1. TCPStore connections imply init_process_group completion. This is generally true but not absolute. PyTorch's init_process_group can complete its store-based rendezvous while NCCL communicator initialization is still in-flight on other threads. However, the NCCL_DEBUG evidence from the earlier test corroborates that NCCL init completed, strengthening this assumption.
2. The hang is deterministic. The assistant assumes that the same hang will reproduce consistently, allowing iterative diagnosis. This is supported by the fact that multiple attempts (messages 6159, 6160, 6171) all produced the same "Init torch distributed begin" message with no further progress.
3. The log file is the sole source of truth for server progress. The assistant checks wc -l and tail to see if any new output appeared. This assumes that SGLang logs all significant events to stdout/stderr, which is reasonable given the framework's logging practices.
4. GPU memory usage not changing confirms no progress. The nvidia-smi output shows memory usage at ~1100 MiB per GPU — essentially the base driver reservation with no model weights loaded. This confirms that weight loading has not begun, consistent with the hang occurring before that phase.

What Input Knowledge Is Required

To fully understand this message, one needs familiarity with several layers of the ML serving stack:

• PyTorch Distributed: The init_process_group function and its relationship to TCPStore-based rendezvous. Understanding that TCPStore connections are established during the rendezvous phase, before NCCL communicators are fully initialized.
• NCCL: The NVIDIA Collective Communications Library, its initialization sequence, and how NCCL_DEBUG=INFO can reveal the state of communicator creation.
• SGLang Architecture: The server's initialization sequence — from argument parsing through model loading. The assistant has located the specific code path in model_runner.py showing that init_torch_distributed is followed by initialize_model_parallel and then weight loading.
• Tensor Parallelism: The concept of splitting a model across multiple GPUs, requiring distributed communication for every forward pass.
• GPU Memory Accounting: Understanding that ~1100 MiB represents base driver usage (CUDA context, NCCL buffers) with no model tensors loaded yet.

The Output Knowledge Created

This message produces several valuable pieces of diagnostic information:

1. A narrowed search space: The hang is now localized to the code path after init_process_group — specifically initialize_model_parallel or the subsequent weight initialization and loading phases. This eliminates hours of potential investigation into NCCL configuration, port conflicts, or store connectivity.
2. A negative result confirmed: The log has only 16 lines and has not grown since the last check. The "Init torch distributed begin" messages remain the last entries. This confirms the server is genuinely stuck, not merely slow.
3. GPU state unchanged: All four GPUs show identical memory usage to the baseline, confirming no rank has begun allocating model weights. This rules out a scenario where some ranks progress while others lag behind.
4. A hypothesis for the next diagnostic step: If the hang is in initialize_model_parallel, the issue may involve the custom all-reduce setup, the Torch symmetric memory initialization, or the MSCclpp backend. The assistant has already disabled custom all-reduce (--disable-custom-all-reduce), but other collective communication backends could be the culprit.

What Happens Next

The message ends with the assistant preparing to check for new log output. The subsequent messages (continuing into the chunk) will reveal that the hang is ultimately caused by P2P DMA corruption under SEV-SNP IOMMU, fixed by setting NCCL_P2P_DISABLE=1. But at this moment in the conversation, the assistant does not yet know this. The message captures the transition from "we don't know where it hangs" to "we know it hangs after init_process_group" — a small but critical step in the debugging journey.

The Broader Lesson in Distributed Debugging

This message exemplifies a pattern that recurs throughout distributed systems debugging: using partial observability to infer system state. The assistant cannot attach a debugger or inspect internal variables of the running processes. Instead, it must rely on indirect signals — socket connections, log messages, GPU memory usage — to triangulate the problem. The reasoning "All 4 ranks are connected to the TCPStore. So init_process_group actually completed" is an inference, not a direct observation. It is the kind of educated deduction that separates systematic debugging from trial-and-error.

The message also demonstrates the importance of knowing your framework's initialization sequence. The assistant had to read the SGLang source code (<msg id=6175-6176>) to understand what happens after "Init torch distributed begin." Without that code inspection, the TCPStore observation would have been just another data point without a clear interpretation. The combination of runtime observation and static code analysis is what enables the diagnostic narrowing.

Finally, the message shows the value of persistence in the face of negative results. The assistant has already tried multiple approaches — NCCL debug, strace, socket inspection, env var overrides — and each has ruled out one more hypothesis. The hang remains, but the search space is shrinking. This message represents the moment when the assistant realizes the hang is deeper in the initialization than previously thought, setting the stage for the eventual discovery of the IOMMU-related P2P corruption.