Two Production Fires at Once: The PD Deadlock and the Tool-Call Leak

In the high-stakes world of production ML serving, problems rarely arrive one at a time. Message [msg 13140] captures a moment of compounding crisis: a user reporting that their production system has locked up again with the same PD deadlock that had consumed the previous several rounds of debugging, while simultaneously surfacing a mysterious second issue — a tool-call parser corruption that causes DeepSeek Markup Language (DSML) tokens to leak through as visible assistant output. The message is brief — just four lines of user text — but it carries immense diagnostic weight, forcing a fork in the investigation that would ultimately consume the next several segments of the conversation.

Context: The Conversation So Far

To understand the significance of this message, one must appreciate the debugging journey that preceded it. The assistant had been deep in the trenches of a PD (prefill-decode) disaggregation deadlock for multiple rounds. In [msg 13133], the assistant began researching the deadlock by checking the sglang fork version and searching upstream GitHub issues. By [msg 13135], the assistant had identified a family of related upstream bugs — most critically issue #26454, an open sglang issue describing a "Non-DP multi-node TP=8 hang in event_loop_overlap" that matched the deployment's symptom pattern almost exactly. The root cause was a tensor parallelism collective desynchronization: when a mass-abort of in-flight KV transfers perturbed per-rank scheduling decisions, some ranks entered collective operations (all_reduce, broadcast) while others branched to on_idle, creating a permanent NCCL/gloo hang that the /health endpoint could not detect.

In [msg 13139], the assistant had just completed a thorough analysis of the proposed fix — --disable-overlap-schedule — explaining its throughput impact (roughly single-digit to ~20% decode throughput loss at high concurrency, near-negligible at C=1) and recommending it as a stability tradeoff. The user had not yet approved applying the fix. Then production locked up again.

The Message Itself: Two Problems in One

The subject message ([msg 13140]) opens with a blunt declaration: "Locked up again" — confirming that the same PD deadlock has recurred before any fix could be deployed. This is a production outage in progress, and it demands immediate recovery action.

But the user immediately pivots to a second, separate issue: "Separately only seemingly under load we have this really odd issue that the model/tool call parser behaves incorrectly and outputs tool call tokens as assistant output which shouldn't even be possible."

The user then provides a concrete example of the corruption. The example shows a conversation round where the assistant's output contains the raw DSML markup tags — the <tool_calls>, <invoke>, and <parameter> markers that are supposed to be intercepted by sglang's tool-call parser and converted into structured API output. Instead, they appear as visible text in the assistant's response. The example also shows a structural anomaly: an extra closing </parameter> tag after the invoke closes, suggesting the parser is either double-closing tags or mishandling the nesting. The format itself looks like the model's native tool-call structure (with invoke name= and parameter name= syntax) wrapped in DeepSeek DSML delimiters, so the corruption is happening during parsing of this format.

The user's closing question — "some race in tool call parser? Or also related?" — reveals their diagnostic hypothesis: perhaps both the deadlock and the tool-call corruption share a root cause. The "also related" refers to the overlap scheduler that was the subject of the previous message. If the same underlying mechanism (the overlap scheduler's deferred result processing, which introduces a one-iteration skew between batch processing and result consumption) were corrupting both the collective synchronization and the parser's state, that would be a powerful unifying explanation.

Why This Message Matters

Message [msg 13140] is a turning point in the conversation for several reasons.

First, it forces triage. The PD deadlock is an active production outage. The tool-call corruption is a correctness issue that degrades output quality. Both demand attention, but the deadlock is immediately blocking all requests. The assistant's response in [msg 13141] reflects this priority: it immediately issues a restart command to recover the locked-up engines while simultaneously beginning diagnostic work on the tool-call leak.

Second, it introduces a new mystery that defies easy explanation. The user's phrase "shouldn't even be possible" is telling. The tool-call parser is a well-tested component of sglang, designed to handle DSML tokens as special markers. For them to appear as visible text suggests either a race condition in the streaming parser state (where per-request state gets corrupted under concurrent load), a tokenization issue (where the special tokens are not being recognized as special by the detokenizer), or a deeper corruption in the output token stream itself. The fact that it only manifests "under load" points toward a concurrency bug rather than a static configuration error.

Third, it raises the question of whether the two issues are connected. The user explicitly asks this. If the overlap scheduler's deferred result processing is corrupting shared state (the per-request parser buffers, the detokenizer state, or the token streaming pipeline), then disabling overlap might fix both problems simultaneously. This is an elegant hypothesis, but as later chunks in the segment reveal, the tool-call leak would prove to be independent of the deadlock — persisting even after overlap was disabled.

Input Knowledge Required

To fully grasp this message, the reader needs to understand several layers of the deployment architecture:

PD Disaggregation: The deployment separates prefill (processing input prompts) and decode (generating output tokens) onto different GPU sets, connected by a KV cache transfer mechanism. This introduces complex synchronization between the two halves.
Overlap Scheduling: sglang's overlap scheduler hides CPU-side batch preparation work behind GPU compute, introducing a one-iteration skew between batch execution and result processing. This is the mechanism that causes TP rank desynchronization under abort scenarios.
DSML (DeepSeek Markup Language): DeepSeek models use a special markup format to represent tool calls, with tokens like <tool_calls>, <invoke>, and <parameter> that the serving framework's parser recognizes and converts to structured API output. These are special tokens in the model's vocabulary, not regular text.
Tensor Parallelism (TP): The model is sharded across 4 GPUs per engine, requiring collective operations (all_reduce, broadcast) to stay synchronized. If ranks diverge in their execution paths, these collectives deadlock permanently.
The NIXL Transfer Race: The specific trigger for the deadlock is a race condition in the NIXL transfer worker, where an aborted KV transfer's cleanup removes a room from transfer_infos while a worker thread is still processing it, causing an AssertionError that desynchronizes the TP ranks.

Output Knowledge Created

This message generates immediate action and long-term investigation:

Immediate recovery: The assistant restarts both engines (prefill and decode), restoring service within 75 seconds ([msg 13142]). The py-spy evidence captured during the lockup confirms the same TP-desync signature, now with a smoking-gun assertion error in the NIXL transfer worker: assert room in self.transfer_infos — the abort cleanup race.
Parser configuration audit: The assistant discovers the deployment uses --tool-call-parser deepseekv4 with --reasoning-parser deepseek-v4 and a v3.2 chat template — a potential mismatch that is investigated but ultimately ruled out as the cause, since a static configuration error would cause consistent failures rather than load-dependent ones.
Deep parser code analysis: The assistant pulls the deepseekv4_detector.py, base_format_detector.py, and reasoning_parser.py files from the sglang source for local analysis, searching for the streaming partial-match buffering logic that could leak tokens under load.
Tokenization investigation: The assistant locates the DSML token definitions in tokenizer.json and begins tracing how special tokens flow through the detokenizer under concurrent streaming conditions.

Assumptions and Their Validity

The user makes several assumptions in this message, some more valid than others.

Assumption 1: The tool-call corruption is related to the deadlock. The user asks "Or also related?" — hypothesizing a shared root cause. This assumption is natural: two bugs appearing simultaneously under load in the same deployment are likely to share a mechanism. However, as later analysis in the segment shows, the tool-call leak proved to be a known DeepSeek-V4 model behavior where the model intermittently degenerates from DSML tool-call mode into text mode under cumulative context/tool-schema pressure. The deadlock was a serving-layer synchronization bug; the tool-call leak was a model behavior issue. They were correlated by load but not causally connected.

Assumption 2: The corruption "shouldn't even be possible." This reflects the user's understanding that the parser architecture should make this impossible — special tokens should be intercepted before they reach visible output. This assumption is technically correct for the intended design, but it underestimates the complexity of streaming token processing. Under concurrent load, the detokenizer may process tokens in larger batches, changing chunk boundaries in ways that expose bugs in the streaming parser's partial-token buffering logic. Additionally, if the model itself is producing malformed DSML (incomplete tags, mismatched nesting), the parser may have no choice but to emit the partial content as text.

Assumption 3: The deadlock is the same bug as before. This assumption proved correct. The py-spy evidence in [msg 13141] confirmed the same TP-desync signature, now with a smoking-gun assertion error in the NIXL transfer worker — the abort cleanup race that knocks the ranks out of lockstep.

The Thinking Process Visible in the Response

The assistant's reasoning in [msg 13141] reveals a disciplined diagnostic approach. It immediately recognizes two separate problems requiring different response strategies:

For the deadlock: Recovery first, analysis second. The assistant confirms the lockup via metrics (decode batches stalled, prefill stuck), captures py-spy evidence to confirm it is the same wedge pattern, then restarts both engines.
For the tool-call leak: Investigation in parallel with recovery. The assistant simultaneously pulls parser configuration flags, searches for DSML token definitions in the tokenizer, and locates the deepseekv4 detector code. The assistant considers several hypotheses for the tool-call leak: - A race condition in the streaming parser state under concurrent load - Special tokens not being recognized by the detokenizer - Chunk-boundary issues where partial special-token sequences are split across streaming increments - Interference between the reasoning parser and tool-call parser - Token-stream corruption from the overlap scheduler's deferred result processing The assistant correctly notes that a static configuration error (like the v3.2 chat template on a v4 model) would cause consistent failures, not load-dependent ones — ruling out the simplest explanation.

Mistakes and Incorrect Assumptions

The assistant's initial hypothesis that the tool-call leak might be caused by the same overlap-scheduler race as the deadlock was reasonable but ultimately incorrect. As the segment's later analysis reveals, the corruption persisted even after --disable-overlap-schedule was deployed, proving the two issues were independent. This is not a mistake per se — it is a legitimate hypothesis that was tested and falsified — but it is worth noting that the elegant unifying explanation was wrong.

A more subtle issue is the assistant's initial assumption that the tool-call leak was a serving-layer bug (a race in the parser or detokenizer). The user would later firmly reject this framing, noting that the same model works flawlessly from cloud providers at high parallelism, and directing the assistant to re-examine deployment-specific patches versus upstream sglang. This reframing — from "model deficiency" to "deployment-specific regression" — would prove crucial in the eventual root-cause analysis, which ultimately traced the corruption to the bf16 index-K patch and its interaction with HiCache under concurrent load.

Conclusion

Message [msg 13140] is a masterclass in production crisis communication. In four lines, the user reports two distinct failures, provides concrete evidence of one, and poses a diagnostic question that would drive the next several segments of investigation. The message captures the reality of debugging under pressure: problems compound, correlation is not causation, and the most elegant hypothesis is not always correct. The assistant's response — triage the outage, investigate the mystery, test the unifying hypothesis — demonstrates the disciplined approach that complex production debugging demands. The deadlock would be fixed with --disable-overlap-schedule, but the tool-call leak would prove far more elusive, ultimately traced to the bf16 index-K patch and its interaction with HiCache under concurrent load — a root cause that was anything but obvious from the initial report. The message stands as a testament to the value of clear, concise bug reporting in high-pressure production environments, where every word carries diagnostic weight.