The Decisive Test: Isolating a CUDA-Graph Corruption Bug Through Controlled Experimentation

In the high-stakes world of production ML inference debugging, few moments are as clarifying as the one captured in message 13409 of this opencode session. After days of methodical investigation into a persistent high-concurrency tool-call corruption affecting the DeepSeek-V4-Flash-NVFP4 model on Blackwell GPUs, the assistant executes a single, carefully designed experiment that cuts through a thicket of competing hypotheses. The message is deceptively brief — a bash command and its output — but it represents the culmination of a rigorous scientific debugging process and a critical branching point in the investigation.

The Message in Full

The assistant writes:

Agent Reasoning Now I'm running the reproduction test against the server with the same parameters as the PD test—60×4 batch size, context length of 300, and max tokens of 2500—to validate the bf16 performance with captured data in non-PD mode. Non-PD single server is serving. Now the decisive reproduction against it (bf16 + captured + no PD transfer):

>

`` [bash] ssh root@10.1.230.171 'echo "=== NON-PD bf16 + captured decode (no transfer) ==="; /root/venv_sglang211/bin/python /root/repro_agent.py --url http://127.0.0.1:30010/v1/chat/completions --sessions 60 --rounds 4 --ctx 300 --max-tokens 2500 --timeout 240 --tag nonpd-bf16-cap 2>&1 | grep -E "CORRUPTION|counts="' === NON-PD bf16 + captured decode (no transfer) === wall=241.5s counts={"done": 9, "no_tool": 3, "leak": 1, "maxrounds": 4, "error": 43} CORRUPTION sessions: 4/60 = 7% (leak=1 no_tool=3 error=43 ok-ish[done/maxrounds]=13) ``

The result: 7% corruption in a non-PD configuration. This single data point carries enormous diagnostic weight.

The Reasoning Behind the Experiment

To understand why this message matters, we must reconstruct the hypothesis space it was designed to collapse. The assistant had been investigating a corruption bug that manifested as tool-call failures in multi-turn agentic sessions. The corruption was specific to the bf16 index-K path — when the model used bf16 precision for its sparse attention index keys (as opposed to fp8), and when running under CUDA-graph capture (as opposed to eager mode), tool-call outputs became garbled at high concurrency.

Two subagents had recently completed deep code inspections ([msg 13405]), and both converged on an important challenge to the assistant's working theory. They pointed out that the assistant's "eager-versus-captured" distinction might be confounded by throughput: eager mode is slower, which reduces concurrency and might prevent a race condition from triggering, while captured mode is fast enough to create the conditions for corruption. More pointedly, the subagents cited the repository's own bisection data showing that non-PD bf16 corruption was ~2% while PD bf16 corruption was ~18% — a massive difference that suggested the disaggregated prefill-decode (PD) transfer path might be the real culprit.

This created a fork in the investigation. If the corruption was primarily a PD-transfer/slot-reuse race, then the fix would live in the transfer layer — perhaps in how the prefill node sends index-K buffers to the decode node, or how the decode node reuses memory slots. But if the corruption persisted even without PD, then the bug was deeper: a capture-local issue in the CUDA-graph replay mechanism itself, independent of disaggregation.

The assistant's reasoning in the message shows careful awareness of this fork. The experiment is designed with surgical precision: run the exact same bf16 + captured configuration, with the same reproducer parameters (60 sessions, 4 rounds, 300 context, 2500 max tokens), but on a single-server non-PD setup that eliminates the transfer layer entirely. The single-server script (examined in [msg 13407]) uses TP4 on GPUs 0-3, port 30010, with SGLANG_DSV4_BF16_INDEX_K=1 and the Triton indexer enabled — identical to the decode worker in the PD setup, but without any prefill-decode communication.

What the Result Tells Us

The 7% corruption rate in the non-PD test is the decisive signal. It is lower than the 18% seen in the PD configuration, but it is unambiguously non-zero. This eliminates the PD-transfer hypothesis as the sole cause. The corruption exists locally in the decode capture path, independent of disaggregation.

This result carries several profound implications:

  1. The bug is capture-local. The corruption originates in how CUDA-graph capture and replay interact with the bf16 index-K buffer on the decode node itself. It is not a transfer race, not a slot-reuse bug in the PD layer, and not a prefill-side issue.
  2. The PD configuration amplifies the corruption but does not cause it. The jump from 7% (non-PD) to 18% (PD) suggests that PD adds additional pressure or timing conditions that make the underlying bug more likely to trigger, but the root cause is present in both configurations.
  3. The fp8-captured path remains clean (as established in earlier tests), which means the bug is specifically about the 2× larger bf16 index-K buffer (256 bytes per token vs 128 bytes for fp8) interacting with the captured-graph memory pool.
  4. The eager mode remains clean (also established earlier), which means the bug is specifically about CUDA-graph replay — the kernel launches themselves are correct, but something about the replay memory environment corrupts the buffer.

Assumptions and Their Validation

The assistant made several assumptions in designing this experiment, all of which were validated by the result:

The Knowledge Created

This message creates critical output knowledge that reshapes the investigation:

The Thinking Process Visible

The assistant's reasoning in this message reveals a mature debugging methodology. The phrase "decisive reproduction" signals an awareness that this experiment is designed to be a falsification test — it must produce a clear result that eliminates at least one major hypothesis branch. The careful choice to match parameters exactly ("60×4 batch size, context length of 300, and max tokens of 2500") shows an understanding of experimental control: the only variable changed is the presence of PD transfer, so any difference in corruption rate can be attributed to that variable.

The assistant also demonstrates intellectual honesty in confronting the subagents' challenge. Rather than defending the earlier eager-vs-captured framing, the assistant redesigned the experiment to test the PD-transfer hypothesis directly. This willingness to pivot based on new evidence — even evidence that complicates one's own theory — is the hallmark of effective debugging.

The Broader Context

This message sits at a pivotal moment in a much longer investigation. In the chunk that follows ([chunk 72.1]), the assistant would go on to definitively root-cause the corruption to a multi-stream-overlap race condition, fixed by disabling SGLANG_OPT_USE_MULTI_STREAM_OVERLAP. But that breakthrough depended on first narrowing the hypothesis space to the capture-local mechanism — which is exactly what this experiment accomplished.

Without the non-PD test, the investigation could have spent days optimizing the PD transfer path, adding checksums and barriers to a layer that was never the source of the problem. The 7% corruption result redirected attention to the captured-graph memory pool, where the actual bug — a race between the C4 sparse indexer's alternate CUDA stream and the main stream's tensors in the shared captured-graph pool — was eventually found and fixed with a single environment variable.

In scientific terms, this message is the experiment that eliminates a confounding variable. In engineering terms, it is the moment the team stops looking in the wrong place. And in narrative terms, it is the turning point where a complex, multi-hypothesis investigation converges toward a single, actionable root cause.