The Diagnostic Pivot: Isolating a Distributed Initialization Hang in SGLang

In any complex debugging session, there comes a moment when the fog of symptoms clears and a single, testable hypothesis emerges. Message [msg 6165] in this opencode conversation captures exactly such a moment. The assistant, after spending several rounds chasing a mysterious hang during the launch of a Qwen3.5-122B-A10B BF16 model with tensor parallelism across 4 NVIDIA Blackwell GPUs, executes a clean diagnostic pivot that transforms a vague "it hangs somewhere" into a precisely targeted experiment.

The Context: A Model That Won't Start

The session leading up to this message had been fraught with infrastructure challenges. The assistant was deploying a 234 GB BF16 model — Qwen3.5-122B-A10B — across 4 RTX PRO 6000 Blackwell GPUs using SGLang, a high-performance inference engine. The server process was consistently hanging at a log line that read "Init torch distributed begin," with no progress for minutes at a time. GPU memory usage remained flat at roughly 1 GB per card, indicating that no model weights had been loaded at all.

Earlier attempts to diagnose this had been inconclusive. The assistant had checked for port conflicts, verified NCCL debug output (which showed NCCL initializing successfully), and even fixed a driver version mismatch between the container's userspace libraries (565.57.01) and the host kernel module (590.48.01). Yet the hang persisted. The NCCL debug output from one attempt (msg [msg 6157]) showed NCCL completing its initialization — "RAS client listening socket," "Bootstrap timings total 0.004433" — which suggested the hang was occurring after NCCL init but before weight loading, squarely in the torch distributed process group initialization phase.

The TP=1 Experiment: A Critical Control

The breakthrough came from a simple control experiment. In msg [msg 6163], the assistant launched the server with --tp 1 (single GPU, no distributed init) and CUDA_VISIBLE_DEVICES=0. The result, visible in msg [msg 6164], was an OOM error — but importantly, the model started loading. The traceback showed it got all the way to allocating MoE expert weights before running out of memory:

File "/root/sglang-main/python/sglang/srt/models/qwen2_moe.py", line 170, in __init__
    self.experts = get_moe_impl_class(quant_config)(

This was a crucial piece of information. It told the assistant two things simultaneously. First, the model architecture was fully supported — the Qwen2 MoE model class, the fused MoE Triton layer, and the unquantized weight loading path all worked correctly. Second, the failure mode was specific to multi-GPU distributed initialization. The TP=1 path succeeded all the way to the OOM point; the TP=4 path never got past distributed setup.

Message 6165: The Diagnostic Pivot

With this evidence in hand, the assistant's message 6165 executes a clean diagnostic pivot. The reasoning is explicit and methodical:

OK — that's expected, TP=1 can't fit 250 GB model on a single 96 GB GPU. But importantly, this confirms the model starts loading just fine on TP=1 — it got all the way to allocating MoE expert weights before OOM. So the model itself is supported.

The TP=4 hang must be in torch distributed init. Let me check if there's a CUDA_VISIBLE_DEVICES or NCCL env issue, and try TP=4 without sitecustomize NCCL overrides:

The assistant has formed a clear hypothesis: the hang is in torch distributed init, and the most likely culprit is the NCCL environment variables set by sitecustomize.py. This is a reasonable suspicion because the sitecustomize file (visible in msg [msg 6155]) contained aggressive NCCL tuning parameters optimized for an 8-GPU configuration:

for _k, _v in [("NCCL_PROTO", "LL"), ("NCCL_ALGO", "Ring"), ("NCCL_P2P_LEVEL", "SYS"),
               ("NCCL_MAX_NCHANNELS", "4"), ("NCCL_BUFFSIZE", "4194304"),
               ("NCCL_NTHREADS", "256")]:
    _os.environ.setdefault(_k, _v)

These settings were tuned for a different topology (8 GPUs with specific PCIe layout) and were being inherited by the 4-GPU configuration. The NCCL_P2P_LEVEL=SYS setting, in particular, could cause NCCL to attempt P2P communication paths that don't exist or are broken in the current topology, leading to a deadlock.

The Experiment: Stripping NCCL Overrides

The assistant's response is to launch a controlled experiment: start the TP=4 server with all NCCL environment variables explicitly cleared:

NCCL_PROTO="" NCCL_ALGO="" NCCL_P2P_LEVEL="" NCCL_MAX_NCHANNELS="" \
NCCL_BUFFSIZE="" NCCL_NTHREADS="" NCCL_DEBUG=WARN \
nohup ~/ml-env/bin/python3 -m sglang.launch_server ...

This is textbook debugging methodology. The assistant has identified a specific variable (NCCL environment configuration) and is testing whether removing it changes the outcome. By clearing all NCCL overrides, the server will use NCCL's default behavior, which should be safe for any topology. If the hang disappears, the NCCL tuning was the root cause. If it persists, the problem lies elsewhere in the distributed initialization path.

The assistant also changes NCCL_DEBUG from INFO to WARN, reducing log noise while still capturing any actual errors. This is a thoughtful touch — the previous NCCL debug run produced voluminous output, and the assistant wants to focus on warnings and errors only.

Assumptions and Reasoning

The assistant makes several assumptions in this message, all of which are reasonable given the evidence:

The TP=4 hang is in torch distributed init, not NCCL itself. This is supported by the NCCL debug output from msg [msg 6157], which showed NCCL completing its bootstrap phase. The hang occurs after NCCL's ncclCommInitRank completes but before the model starts loading.
The NCCL environment variables from sitecustomize.py are the most likely cause. This is a good hypothesis because the variables were tuned for an 8-GPU configuration, and the topology has changed. NCCL_P2P_LEVEL=SYS could cause NCCL to attempt P2P communication across PCIe switches or NUMA domains that aren't properly connected.
The model itself is not the issue. The TP=1 test confirmed the model loads correctly (up to OOM). This eliminates model format, tokenizer, and quantization issues.
The SGLang build and configuration are correct. The assistant doesn't suspect a code bug in SGLang itself, which is reasonable since the TP=1 path worked.

Input Knowledge Required

To fully understand this message, one needs:

Understanding of tensor parallelism (TP): The --tp 4 flag splits the model across 4 GPUs, requiring NCCL and torch distributed to coordinate. TP=1 is a single-GPU fallback that skips distributed init entirely.
Knowledge of NCCL environment variables: NCCL_PROTO, NCCL_ALGO, NCCL_P2P_LEVEL, NCCL_MAX_NCHANNELS, NCCL_BUFFSIZE, and NCCL_NTHREADS all control NCCL's communication behavior. NCCL_P2P_LEVEL=SYS enables P2P communication across system-level boundaries (e.g., different PCIe root complexes), which can cause issues if the topology doesn't support it.
Familiarity with sitecustomize.py: This Python file is executed at interpreter startup and can set environment variables. Because it runs before the user's code, environment variables set there can override command-line or shell-level overrides (depending on whether setdefault or direct assignment is used).
Understanding of the SGLang launch process: SGLang initializes NCCL first, then torch distributed, then loads model weights. The hang at "Init torch distributed begin" indicates the process is stuck in torch.distributed.init_process_group.

Output Knowledge Created

This message creates several valuable pieces of knowledge:

A confirmed hypothesis: The TP=4 hang is in torch distributed init, not NCCL or model loading. This narrows the search space dramatically.
A testable experiment: The NCCL-clear launch will determine whether the sitecustomize NCCL overrides are the root cause. The result of this experiment (visible in subsequent messages) will either confirm the hypothesis or force the assistant to look elsewhere.
A debugging methodology: The assistant demonstrates a pattern of using controlled experiments (TP=1 vs TP=4, NCCL vars cleared vs default) to isolate variables. This is a model for systematic debugging of distributed systems issues.
Documentation of the sitecustomize NCCL configuration: The message implicitly documents that the NCCL tuning in sitecustomize.py may be topology-specific and can cause hangs when the GPU topology changes.

The Thinking Process

The assistant's thinking process in this message is a masterclass in diagnostic reasoning. It follows a clear chain:

Observe: TP=4 hangs at "Init torch distributed begin." TP=1 loads successfully (until OOM).
Infer: The hang is specific to distributed initialization, not model loading.
Hypothesize: The NCCL environment variables from sitecustomize.py, tuned for 8 GPUs, may be incompatible with the 4-GPU topology.
Test: Launch TP=4 with all NCCL overrides cleared.
Predict: If the hang was caused by NCCL tuning, the server should proceed past distributed init and start loading weights. The message also shows the assistant's awareness of the setdefault semantics — os.environ.setdefault only sets the variable if it's not already set, which means shell-level exports should override it. But the assistant wisely clears the variables at the shell level anyway, ensuring they're absent from the environment entirely.

Conclusion

Message [msg 6165] is a textbook example of diagnostic debugging in a complex distributed systems context. It demonstrates the power of controlled experimentation — using a TP=1 test to isolate the failure mode, forming a specific hypothesis about the root cause, and designing a clean experiment to test it. The message is concise yet rich with reasoning, showing how a single well-crafted diagnostic step can transform a confusing hang into a testable hypothesis. Whether or not the NCCL override hypothesis proves correct (the story continues in subsequent messages), the methodology itself is the lasting contribution: when faced with a distributed system hang, isolate variables, test one hypothesis at a time, and let the evidence guide the next step.