The Phantom Deadlock: How a TP Collective Desync Silently Wedged a Production LLM Server

Introduction

On June 18, 2026, at 20:38 UTC, a production deployment of DeepSeek-V4-Flash running on eight NVIDIA RTX PRO 6000 Blackwell GPUs stopped serving requests. The health check endpoints returned HTTP 200. The processes were alive. Memory was allocated. Logs continued to scroll. But no request ever completed. Every new request would prefill, then park forever at inflight-req:1, never reaching decode, never returning a response. The system was, to all appearances, healthy—yet completely dead.

This is the story of how the assistant diagnosed that silent failure, and the pivotal moment when py-spy stack traces revealed the true root cause. It is a story about the difference between a plausible hypothesis and a proven one, about the subtle ways distributed systems fail, and about a single message ([msg 13122]) that transformed the investigation.

The Scene: A Production Emergency

The deployment was sophisticated even by modern standards. The assistant had set up a prefill-decode (PD) disaggregated serving architecture using SGLang, where four GPUs handled the prefill phase (processing the prompt and generating the initial KV cache) and four separate GPUs handled the decode phase (generating tokens autoregressively). The two groups communicated over PCIe using the NIXL transfer backend with UCX transport, shuttling KV cache tensors from prefill to decode as each request progressed.

This architecture, while powerful, introduces a fundamental coordination challenge: the prefill and decode engines must stay synchronized across a distributed collective of tensor-parallel (TP) ranks. Each TP rank is a process that owns a shard of the model, and they communicate via NCCL (NVIDIA Collective Communications Library) for operations like all_reduce and broadcast. When everything works, this coordination is invisible. When it breaks, the failure mode is insidious—the processes don't crash, they just stop making progress.

The trigger was innocuous: the user cancelled a parallel agent that had approximately 13 in-flight KV transfers. Those cancellations sent AbortReq messages through the system, perturbing the scheduling state of each TP rank. What happened next would take the assistant multiple rounds of investigation to unravel.

The Initial Hypothesis: A Wedged NIXL Transfer Agent

The assistant's first diagnosis, presented in [msg 13118], was confident and specific: the NIXL/UCX transfer agent had entered a bad state. The reasoning was straightforward. The mass-abort of in-flight KV transfers could leave UCX worker handles dangling. Since UCX ran over PCIe without RDMA, the transfer agent state was fragile. The evidence seemed to fit: the control plane still worked (new requests could prefill, bootstrap registration succeeded), but the data plane was dead (no KV transfer ever completed). Every request would prefill, then park waiting for a KV transfer that would never arrive.

The assistant offered the user a choice: restart immediately, or first capture a py-spy snapshot of the frozen scheduler to pin the exact deadlock for a bug report. The user chose the latter—a decision that would prove decisive.

The Pivot: py-spy Reveals the Truth

In [msg 13120] and [msg 13121], the assistant ran py-spy dump on the scheduler processes. The results were stored to /tmp/opencode/pyspy/ and analyzed. What they showed was not a wedged NIXL agent at all. It was something far more fundamental—and far more interesting.

The stack traces revealed that the prefill TP ranks were split: TP0 and TP1 were blocked inside a HiCache all_reduce operation, while TP2 and TP3 had taken the on_idle branch of the event loop, which never calls that collective. The decode ranks showed the mirror image: TP0 had branched to on_idle, while TP1, TP2, and TP3 were blocked waiting for a broadcast in recv_requests.

This was not a transport-layer bug. This was a tensor-parallel collective desync—a classic distributed systems failure where ranks in the same NCCL communicator call different collective operations, causing a permanent hang with no timeout.

The Subject Message: A Definitive Diagnosis

Message [msg 13122] is where the assistant synthesizes these observations into a coherent root-cause analysis and acts on it. Let us examine it in full.

The message opens with the assistant's reasoning, which shows the moment of crystallization:

I'm seeing a clear picture of a TP collective desync happening on both servers, with the prefill ranks showing different states: TP0 is near or at the all_reduce checkpoint in the cache event check, TP1 is blocked waiting inside that same all_reduce operation, and TP2 appears to be in a different state.

The assistant then traces the exact execution paths:

The critical issue is that TP0 and TP1 entered the get_new_batch_prefill path which triggers the HiCache all_reduce collective, while TP2 and TP3 took the on_idle branch instead—a completely different code path that never calls that all_reduce. This is the deadlock: TP0 and TP1 are stuck waiting for TP2 and TP3 to join a collective they'll never reach because those ranks diverged at the prefill event loop decision point and went down the idle path instead. The ranks split on whether to build a new batch or idle, causing a permanent NCCL hang.

This is the key insight. NCCL collectives require all ranks in the communicator to participate in the same collective at the same time. If one rank calls all_reduce while another calls broadcast, or if one rank calls a collective while another does local work, NCCL has no mechanism to detect or recover from the mismatch. It simply waits forever. The processes stay alive, the health checks pass, and the system appears healthy—but no forward progress is possible.

The assistant then extends the analysis to the decode side:

In the decode stage, the same pattern emerges: TP0 took the on_idle branch while TP1, TP2, and TP3 are blocked waiting at a broadcast in recv_requests. Again, rank divergence—TP0 raced ahead into idle while the others are stuck waiting for it to participate in a collective it never reaches. Both servers exhibit the same fundamental bug: the disaggregation event loops allow tensor parallel ranks to diverge on control-flow decisions (whether to build a batch or go idle, whether to receive requests or idle), and those divergent paths contain mismatched collectives that deadlock.

The assistant correctly identifies that this is not solely a HiCache issue. While HiCache's check_hicache_events all_reduce is an extra desync point on the prefill side, the decode side has no HiCache and still deadlocks on the request broadcast. The root cause is deeper: the overlap event loop itself allows per-rank branch decisions that can diverge.

The Formal Diagnosis

After the reasoning section, the assistant presents a clean, structured diagnosis:

Now it's definitive. This is a TP collective desync bug in sglang's PD overlap event loop, and it's on both servers:

>

Prefill (event_loop_overlap_disagg_prefill): - TP0+TP1 went down the build-batch path → check_hicache_events → all_reduce (waiting) - TP2+TP3 took the on_idle branch (prefill.py:498) → never call that all_reduce - → TP0/1 wait forever for TP2/3 → NCCL hang

>

Decode (event_loop_overlap_disagg_decode): - TP0 took on_idle (decode.py:1814) - TP1+TP2+TP3 are blocked in the recv_requests broadcast (decode.py:1783), waiting for TP0 which never joins - → NCCL hang

This diagnosis is notable for its precision. The assistant identifies not just the general class of bug (TP collective desync), but the exact source lines in the sglang source code where the divergent branches occur: prefill.py:480 (build-batch path), prefill.py:498 (on_idle branch), decode.py:1783 (recv_requests broadcast), decode.py:1814 (on_idle branch). This level of specificity is what separates a hypothesis from a diagnosis.

The Trigger Mechanism

The assistant also explains why the abort causes this divergence:

So the real answer to your question: the abort lands right at a scheduling boundary and knocks the TP ranks out of lockstep — some ranks decide "there's work, build a batch" (which calls a collective) while others decide "idle" (which doesn't). The mismatched NCCL collective then hangs forever with no timeout.

The key insight is that the abort does not directly cause the deadlock. Rather, it perturbs the per-rank scheduling state in a way that causes the ranks to make different control-flow decisions at the next scheduling boundary. In a perfectly synchronized system, all ranks would agree on whether to build a batch or go idle. But when some ranks have processed the abort differently—perhaps because they received it at slightly different times, or because their internal queue states diverged—they can reach different conclusions. And once they diverge on control flow, the NCCL collective mismatch is fatal.

This is a fundamentally different class of bug from a crashed process or a network failure. It is a liveness bug in the distributed coordination protocol, where the system enters a state that is logically consistent from each rank's perspective but globally deadlocked.

The Action: Recovery

Having identified the root cause, the assistant acts decisively. It saves a summary to /tmp/opencode/pyspy/SUMMARY.txt documenting the exact deadlock state, then issues the restart:

`` systemctl restart sglang-dsv4-prefill sglang-dsv4-decode ``

The message concludes with the restart output, showing the summary file contents and confirming the restart was issued. The production system is on its way back to health.

Technical Deep Dive: Why NCCL Collective Desync Is So Dangerous

To understand why this bug is so pernicious, one must understand NCCL's design. NCCL is a collective communication library optimized for GPU-to-GPU communication within a tensor-parallel group. It provides operations like all_reduce (sum a tensor across all ranks and distribute the result to all ranks), broadcast (send a tensor from one rank to all others), and barrier (wait until all ranks reach this point).

The critical property of NCCL collectives is that they are synchronous across ranks: all ranks in the communicator must call the same collective operation in the same order. If rank 0 calls all_reduce while rank 1 calls broadcast, NCCL has no mechanism to detect the mismatch. It simply blocks each rank on its respective operation, waiting for the other ranks to join—which they never will.

This is fundamentally different from a network timeout or a process crash. In those cases, monitoring tools can detect the failure and trigger recovery. But NCCL collective desync produces no error signal. The processes stay alive, the GPUs remain allocated, the health endpoints return 200, and the system appears healthy to every external observer. The only symptom is that no work completes—and even that can be hard to detect if the system has low request volume.

The sglang watchdog, which is designed to detect stuck forward passes, does not fire in this case because the ranks are not stuck in a forward pass. They are stuck in NCCL collectives that look like normal communication operations. The watchdog sees idle processes and concludes everything is fine.

The Broader Significance

This bug is not specific to DeepSeek-V4 or to Blackwell GPUs. It is a fundamental challenge in designing distributed serving systems that use tensor parallelism with disaggregated prefill and decode. The overlap event loop, which allows the scheduler to overlap KV transfer with batch preparation, introduces per-rank control-flow decisions that can diverge under load perturbations.

The assistant's analysis in [msg 13122] correctly identifies the fix direction: disabling overlap scheduling (--disable-overlap-schedule) forces the ranks into a lockstep execution path where they cannot diverge on control flow. This trades some throughput for correctness, but in a production system, correctness is non-negotiable.

The message also surfaces a deeper architectural question: how should distributed serving systems handle the tension between performance optimization (overlapping communication with computation) and correctness (ensuring ranks stay synchronized)? The overlap event loop is an optimization that works beautifully in the common case but creates a fragile state space where rare perturbations can cause catastrophic failure. This is a recurring theme in systems engineering—the gap between "works in practice" and "works in all cases."

What This Message Teaches Us

The subject message is a masterclass in distributed systems debugging. Several lessons stand out:

1. Hypothesis falsification is as important as hypothesis formation. The assistant's initial NIXL/UCX hypothesis was plausible and well-reasoned. But rather than acting on it, the assistant gathered diagnostic evidence that either confirmed or refuted it. The py-spy data decisively refuted the NIXL hypothesis and revealed the true root cause.

2. Stack traces are the ultimate ground truth. When a distributed system fails silently, the first question should always be "what is each process actually doing?" py-spy provided the answer: the ranks were not stuck in NIXL transfer calls but in NCCL collectives. This changed everything.

3. Correlation is not causation. The mass-abort correlated with the hang, but it was not the direct cause. The abort was a trigger that perturbed scheduling state, which then caused rank divergence at the next scheduling boundary. The root cause was the rank divergence, not the abort itself.

4. Silent failures require active probing. The system returned HTTP 200 on its health endpoints. Traditional monitoring would have declared it healthy. Only active investigation—running py-spy, examining stack traces, comparing ranks—revealed the deadlock.

Conclusion

Message [msg 13122] represents the turning point in a production debugging saga. It is the moment when a plausible but incorrect hypothesis (wedged NIXL agent) was replaced by a precise, evidence-based diagnosis (TP collective desync in the overlap event loop). The assistant not only identified the bug but traced it to specific source lines, explained the trigger mechanism, and initiated recovery.

The deeper lesson is about the nature of distributed systems failures. The most dangerous bugs are not the ones that crash processes—those are easy to detect and recover from. The most dangerous bugs are the ones that leave the system looking healthy while silently preventing all forward progress. NCCL collective desync is a prime example, and the assistant's methodical investigation—from hypothesis to evidence to diagnosis to action—provides a template for how to approach such failures.

The fix, deployed in subsequent messages, was to disable overlap scheduling, which eliminated the per-rank control-flow divergence. The system recovered, the throughput returned, and the investigation moved on to other challenges. But the lesson of the phantom deadlock—that a system can appear healthy while being completely dead—remains a cautionary tale for anyone building distributed serving infrastructure.