The Diagnostic Pivot: Isolating a Multi-GPU Hang with a Single-GPU Test

In the midst of a complex deployment debugging session, message [msg 6163] stands out as a textbook example of systematic diagnostic narrowing. The assistant, faced with a persistent hang during multi-GPU initialization of an SGLang inference server, made a deliberate decision to strip away all complexity and test the absolute minimum viable configuration: a single GPU running a single process. This seemingly simple bash command — launching the server with CUDA_VISIBLE_DEVICES=0 and --tp 1 — was the culmination of a frustrating diagnostic spiral and the key to isolating a subtle distributed computing bug.

The Context of Failure

To understand why this message was written, one must appreciate the sequence of failures that preceded it. The assistant had been deploying the Qwen3.5-122B-A10B model — a 234 GB BF16 Mixture-of-Experts model — on a machine with 8× RTX PRO 6000 Blackwell GPUs (each with 96 GB VRAM). The GPU topology had recently been reconfigured: 4 GPUs were assigned to an LXC container running SGLang, while the other 4 were passed through to a confidential computing VM via vfio-pci. The SGLang service was updated from TP=8 to TP=4 to match the new allocation.

Every attempt to start the server with TP=4 resulted in the same failure mode: the process would reach the log line "Init torch distributed begin" and then hang indefinitely. After 90 seconds, GPU memory usage remained flat at ~1 GB per card — no weights had been loaded. The assistant tried multiple interventions: fixing a driver version mismatch (container had 565 userspace against a 590 kernel module, resolved in [msg 6146]), checking for port conflicts, and even running NCCL_DEBUG=INFO to trace the hang ([msg 6157]).

The NCCL debug output was revealing but incomplete. It showed NCCL initializing its network plugin and bootstrap timings completing, but the output was truncated by a timeout 60 wrapper and tail -40 filter. The assistant interpreted this as "NCCL init actually completed fine" ([msg 6158]) and hypothesized the hang was in torch's init_process_group — a higher-level distributed coordination layer built on top of NCCL. Subsequent checks showed no rendezvous ports being opened, suggesting the worker processes were deadlocking before establishing communication.

The Diagnostic Decision

Message [msg 6163] represents a deliberate diagnostic pivot. After killing all hung processes and clearing GPU state ([msg 6162]), the assistant chose to test with --tp 1 — tensor parallelism of 1, meaning a single GPU running a single process. The command was:

CUDA_VISIBLE_DEVICES=0 nohup ~/ml-env/bin/python3 -m sglang.launch_server \
  --model-path /shared/models/Qwen3.5-122B-A10B \
  --served-model-name qwen3.5-122b \
  --tp 1 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --attention-backend triton \
  --kv-cache-dtype bf16 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  > /tmp/sglang_122b_tp1.log 2>&1 &
echo "PID=$!"

Several design choices are worth examining. First, CUDA_VISIBLE_DEVICES=0 explicitly restricts the process to GPU 0, eliminating any possibility of cross-GPU communication. Combined with --tp 1, this bypasses the entire torch distributed initialization code path — no init_process_group, no NCCL communicator setup across ranks, no rendezvous. Second, the use of nohup and output redirection to a log file allows the command to run asynchronously while the SSH session remains interactive for monitoring. Third, the assistant stripped out --disable-custom-all-reduce (present in the TP=4 attempts) and the speculative decoding flags, further simplifying the configuration.

Assumptions and Reasoning

The assistant's reasoning, visible in the preceding message ([msg 6162]), was explicit: "try with --tp 1 first to see if the model even loads." This reveals a key assumption — that the hang was specific to multi-GPU distributed initialization, not a fundamental model compatibility issue. The assistant believed that if the model loaded successfully on a single GPU (even if it ultimately OOM'd on the 96 GB card), it would confirm the model code was functional and the problem was in the distributed layer.

There was also an implicit assumption that the NCCL debug test had definitively shown NCCL init succeeding. In reality, the NCCL_DEBUG=INFO output was truncated — the timeout 60 killed the process before it could complete, and tail -40 showed only the beginning of NCCL initialization. The assistant may have been overly optimistic in concluding NCCL was fine.

The Result and Its Implications

The TP=1 test produced a critical result ([msg 6164]). After 30 seconds, the log showed the model progressing through weight loading all the way to MoE expert weight allocation before hitting an out-of-memory error:

File "/root/sglang-main/python/sglang/srt/models/qwen2_moe.py", line 170, in __init__
    self.experts = get_moe_impl_class(quant_config)(...
  File "/root/sglang-main/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 283, in __init__
    self.quant_method.create_weights(
  File "/root/sglang-main/python/sglang/srt/layers/quantization/unquant.py", line 191, in create_weights
    torch.empty(num...

This was a successful failure — the OOM was expected (234 GB model on a 96 GB GPU), but crucially, the model code executed correctly. The MoE layer initialization ran, the weight tensors began allocation, and the error was clean and predictable. This confirmed two things: the model was compatible with the SGLang build, and the hang was indeed specific to multi-GPU distributed initialization.

Input and Output Knowledge

The input knowledge required to understand this message includes: the model size (234 GB BF16), the GPU memory capacity (96 GB per card), the previous TP=4 hang behavior, the NCCL debug results, and the driver version fix that had just been applied. The assistant also needed to know that --tp 1 bypasses torch distributed initialization entirely — a non-obvious detail about SGLang's architecture.

The output knowledge created by this message was decisive. It ruled out model incompatibility, confirmed the software stack was functional, and isolated the problem to the multi-GPU distributed layer. This directly led to the next diagnostic step ([msg 6165]): attempting TP=4 again but with all NCCL environment variables cleared to test whether the sitecustomize.py overrides (optimized for the old 8-GPU topology) were causing the hang. That subsequent test revealed that NCCL_P2P_LEVEL=SYS — set in sitecustomize.py for the previous 8-GPU configuration — was incompatible with the new 4-GPU topology under the SEV-SNP IOMMU configuration.

The Broader Debugging Pattern

Message [msg 6163] exemplifies a fundamental debugging principle: when faced with a complex failure, reduce the system to its simplest possible configuration. The assistant could have continued chasing NCCL configuration, trying different NCCL_ALGO values, or debugging torch rendezvous mechanics. Instead, it took a step back and asked the most basic question: "Does the model even load?" By answering that question with a clean yes (OOM notwithstanding), the assistant eliminated an entire class of potential causes and narrowed the search space dramatically.

This single bash command, unremarkable in isolation, was the turning point in a multi-hour debugging session. It transformed a mysterious, silent hang into a well-understood distributed communication issue, ultimately traced to P2P DMA corruption under SEV-SNP IOMMU — a problem that would be solved by setting NCCL_P2P_DISABLE=1 in a subsequent round.