The Decisive Test: Isolating a CUDA-Graph Corruption Bug Through Non-PD Single-Server Reproduction

Introduction

In the long and arduous hunt for a high-concurrency tool-call corruption bug in the DeepSeek-V4-Flash-NVFP4 inference stack, there comes a moment where hypothesis refinement must give way to decisive experimental evidence. Message [msg 13408] captures exactly such a moment. The assistant, having spent multiple rounds deploying subagents for deep code inspection, running canary instrumentation, and debating the relative merits of competing theories, executes a clean, high-signal test: run the bf16 index-K path with CUDA-graph capture enabled on a non-PD (non-disaggregated) single server, eliminating the prefill-decode transfer layer entirely. If the corruption persists, the bug is local to the CUDA-graph replay mechanism. If it disappears, the root cause lies in the disaggregated transfer path or slot-reuse logic.

This message is a masterclass in experimental design under uncertainty. It is brief—a single bash command and its output—but it represents the culmination of dozens of earlier messages, two parallel subagent investigations, and a growing recognition that the most parsimonious explanation might be wrong. Let us examine what makes this message so pivotal.

The Context: A Bug That Defies Easy Explanation

To understand message [msg 13408], one must first understand the bug it was designed to isolate. For many sessions, the assistant had been battling a persistent corruption affecting tool-calling quality in the deployed DeepSeek-V4-Flash model. The corruption was specific to a particular configuration: bf16 index keys (enabled via SGLANG_DSV4_BF16_INDEX_K=1) under CUDA-graph capture at high decode concurrency. The fp8 path, even under capture, remained perfectly clean. Eager mode (no CUDA graph) also remained clean. The corruption rate was staggering: approximately 15–18% of multi-turn sessions showed corrupted tool calls under the bf16+captured+PD configuration.

The assistant had methodically eliminated one hypothesis after another. It had verified that the prompt index-K transfer through the PD layer delivered correct data via checksums. It had confirmed that the store geometry, output location plans, readers, and downstream sparse-attention kernels were byte-identical between the clean fp8 path and the corrupted bf16 path. It had ruled out retraction logic, memory pool pressure, and the max_c4_seq_len metadata asymmetry. By elimination, the corruption had been narrowed to the persistent bf16 index-K buffer contents being corrupted under captured high-concurrency replay.

But a critical confound remained. As both subagents pointed out in their reports (summarized in [msg 13406]), the assistant's earlier A/B test—which toggled CUDA-graph capture on the decode node while keeping PD constant—might have conflated two different factors. The repo's own bisection data showed a stark difference: non-PD bf16 showed ~2% corruption, while PD bf16 jumped to ~18%. This suggested that the PD transfer layer, not the capture mechanism itself, might be the primary culprit. The eager-versus-captured difference could simply be a throughput effect: eager decode is slower, reducing concurrency and thus the probability of triggering a race condition in the PD transfer path. Under this alternative theory, the corruption was a PD-transfer or slot-reuse race that only manifested under the high throughput that CUDA-graph capture enabled.

The Reasoning: Designing the Decisive Experiment

The assistant's Agent Reasoning section in message [msg 13408] reveals a crisp, focused thought process. After absorbing the subagent findings and weighing the competing theories, the assistant identifies the single cleanest test to separate the two hypotheses:

"Running the decisive non-PD test: stop PD, launch the single TP4 server (bf16 + captured + triton indexer, no transfer), and reproduce. If it still corrupts, the bug is capture-local, not the PD transfer."

This is textbook experimental design. The assistant recognizes that the existing infrastructure already contains the perfect instrument: a pre-written single-server script (serve_dsv4_single.sh) that runs TP4 on GPUs 0–3 with the same bf16 index keys, the same Triton indexer, and the same CUDA-graph capture—but without any disaggregated prefill-decode transfer. The script was created precisely for this kind of isolation test, as its header comment states: "Non-PD single-server (TP4, GPUs 0-3) for isolating PD-transfer vs in-process bf16 corruption."

The reasoning also reveals a key prediction. The assistant notes that decode-generated index-K is computed and stored locally on the decode node regardless of whether PD is active—only the prompt index-K is transferred. Since the assistant had already verified that prompt transfer is correct via checksums, the corruption must be in the local decode-generated index storage. This leads to the expectation that the single server will reproduce the corruption, pointing to a capture-local bug. But the assistant is careful not to prejudge the outcome—the experiment is designed to let the data speak.

The Execution: A Carefully Orchestrated Bash Command

The bash command in message [msg 13408] is a marvel of operational pragmatism. Let us examine its structure:

  1. Stop the PD servers: systemctl stop sglang-dsv4-prefill sglang-dsv4-decode — this frees GPUs 0–3 (used by prefill) for the single server.
  2. Verify GPU memory freed: nvidia-smi --query-gpu=index,memory.used — all 8 GPUs show 4 MiB, confirming clean state.
  3. Launch the single server: nohup bash /root/serve_dsv4_single.sh > /tmp/single.log 2>&1 & — a background process with output redirected to a log file.
  4. Wait for health endpoint: A loop polling /health every 5 seconds for up to 60 iterations (300 seconds total).
  5. Verify serving capability: A second loop sending a minimal chat completion request every 3 seconds for up to 30 iterations (90 seconds total). The output confirms success: the health endpoint responded after 70 seconds, and the server was serving after just 3 more seconds. The server is ready for the reproduction test. Several design decisions are worth noting. First, the assistant chooses nohup over a systemd unit, explicitly acknowledging in the prior message ([msg 13407]) that "since this is a diagnostic test rather than production, nohup is acceptable even though my preference is systemctl." This is a pragmatic tradeoff: systemd would provide cleaner process management, but nohup is faster to deploy for a temporary diagnostic server. Second, the assistant uses a two-phase readiness check: first the lightweight /health endpoint, then an actual chat completion request. This ensures the model is fully loaded and serving before the reproduction script is launched. The 70-second startup time is consistent with loading a large model across 4 GPUs. Third, the assistant pipes all output through 2>&1 at the SSH level, ensuring the complete log is captured in the conversation for later analysis.

Assumptions and Potential Pitfalls

The experiment rests on several assumptions that deserve scrutiny:

Assumption 1: The single-server configuration is identical to the PD decode configuration except for the absence of transfer. The script exports SGLANG_DSV4_BF16_INDEX_K=1, SGLANG_SM120_MMA_FLASHMLA=1, and SGLANG_SM120_TRITON_INDEXER=1, matching the decode worker's environment. However, there may be subtle differences in how SGLang initializes the memory pool or schedules kernels in a non-disaggregated setup versus a decode-only worker. The assistant acknowledges this risk implicitly by framing the test as decisive but not infallible.

Assumption 2: The corruption, if capture-local, will reproduce at the same rate as in the PD setup. The single server runs on 4 GPUs (TP4) rather than the 4-GPU decode worker, and the concurrency profile may differ. If the corruption requires a specific interaction between concurrent request processing and the PD transfer layer, the single server might show a lower rate even if the capture mechanism is involved.

Assumption 3: Stopping and restarting the PD servers is safe. The assistant stops both prefill and decode simultaneously, which could leave in-flight requests hanging. The 3-second sleep before checking GPU state is minimal but sufficient for the systemd stop to complete.

Assumption 4: The reproducer script (repro_agent.py) can be pointed at the new port (30010) via --url. The assistant confirmed this in [msg 13407] by inspecting the script's argparse configuration.

Input Knowledge Required

To fully understand message [msg 13408], one needs:

Output Knowledge Created

Message [msg 13408] produces several valuable outputs:

  1. A running non-PD single server on port 30010, ready for the reproduction test. This is the experimental apparatus itself.
  2. Confirmation of clean GPU state: All 8 GPUs at 4 MiB after stopping PD, indicating no memory leaks or stuck processes.
  3. Startup timing data: 70 seconds to health, 73 seconds to serving. This baseline is useful for future diagnostics.
  4. A log file at /tmp/single.log containing the server's startup output, available for post-mortem analysis if the server crashes or exhibits errors during the reproduction test.
  5. The experimental design itself: The non-PD single-server test as a methodological template for isolating PD-specific effects from capture-local effects. This design is reusable for future bugs. The most important output, however, is not in the message itself but in what it enables: the reproduction test that will follow. The assistant will run repro_agent.py with --url http://127.0.0.1:30010/v1/chat/completions against this server, and the corruption rate will tell them definitively whether the bug lives in the PD transfer path or in the CUDA-graph capture mechanism.

The Thinking Process: A Window Into Diagnostic Reasoning

The Agent Reasoning section of message [msg 13408] is remarkably concise compared to the sprawling, multi-hypothesis reasoning of the preceding messages ([msg 13406] and [msg 13407]). This concision itself is significant: it signals that the assistant has reached a point of clarity after the subagent investigations. The earlier messages show the assistant spinning through multiple hypotheses—retraction issues, memory pool overlap, PD transfer races, capture write hazards—and weighing their relative plausibility. By message [msg 13408], the field has narrowed to a single binary question: PD or capture-local?

The reasoning also reveals a subtle but important shift in the assistant's epistemic stance. In earlier messages, the assistant was confident that the corruption was capture-local, having verified the PD transfer via checksums. But the subagents' challenge—that the eager-vs-captured distinction might be confounded with throughput—introduced genuine uncertainty. The assistant's willingness to design and execute a test that could falsify its own preferred hypothesis is a hallmark of good scientific practice. The non-PD test is, in effect, the assistant saying: "I think the bug is capture-local, but let me create the conditions where I could be proven wrong."

Conclusion

Message [msg 13408] is a brief but pivotal moment in a long debugging saga. It represents the transition from hypothesis generation to experimental verification, from armchair analysis to empirical test. The assistant takes a clean, well-defined action that will produce a clean, well-defined answer: run the bf16+captured configuration without the PD transfer layer, and observe whether the corruption reproduces.

The elegance of this experiment lies in its simplicity. The assistant does not attempt to instrument the code further, deploy more canaries, or write new diagnostic tools. Instead, it leverages existing infrastructure—the single-server script created for exactly this purpose—to create a controlled comparison. The only variable that changes is the presence or absence of PD transfer. Everything else—the model, the precision, the indexer, the capture mode—remains constant.

This is the essence of good debugging: not adding complexity, but subtracting it until the signal emerges from the noise. Whether the corruption reproduces or not, the assistant will have learned something definitive, and the path to a fix will be clearer. That is the value of a well-designed experiment, and message [msg 13408] is a textbook example of one.