The Wedge Test: A High-Stakes A/B Experiment on a Production LLM Inference System

Introduction

In the high-stakes world of production LLM inference, few things are more terrifying than a silent deadlock. A wedge — a condition where the system stops processing requests while appearing healthy to external monitoring — can go undetected for minutes or hours, silently accumulating latency and frustrating users. On a system serving agentic workloads with multi-turn tool-calling sessions, a wedge means stalled agents, timed-out tool calls, and cascading failures across the entire application stack.

Message <msg id=13505> captures a pivotal moment in a long-running optimization campaign for a DeepSeek-V4-Flash-NVFP4 deployment on Blackwell GPUs. In this message, the assistant performs a controlled A/B experiment: re-enabling a previously disabled scheduler optimization (the "overlap scheduler") on the decode worker of a prefill-decode (PD) disaggregated serving setup, then actively trying to trigger a known deadlock condition. The experiment passes — 24 serial requests complete cleanly with zero failures — but as we shall see, this is only the first act of a longer story about the tension between throughput optimization and correctness in distributed systems.

The Overlap Scheduler: What It Is and Why It Was Disabled

To understand the significance of this message, we must first understand the overlap scheduler. In SGLang's PD-disaggregated architecture, the decode worker runs a tight loop: receive a batch of request tokens from the prefill worker, run the transformer forward pass, perform an all-reduce across tensor-parallel (TP) ranks to synchronize the output, and return results. The "overlap scheduler" is an optimization that overlaps CPU-side batch preparation (building input tensors, managing KV cache metadata) with the GPU forward pass of the previous iteration. This hides CPU latency behind GPU computation, yielding meaningful throughput gains — estimated at 9-10% at batch size 73 and 16-18% at batch size 32.

However, the overlap scheduler has a dangerous flaw. In the PD-disaggregated decode loop, each TP rank independently decides whether to run a batch or go idle. Under the overlap scheduler, this run-versus-idle decision is made before the collective synchronization that normally keeps ranks in agreement. If one rank decides to run a batch while another decides to go idle, their subsequent collective operations (all-reduce, gloo-poll) become asymmetric — one rank participates while the other skips. This asymmetry causes a permanent desynchronization known as the "TP-collective desync wedge." Once wedged, the system stops processing requests entirely, with no error messages and no obvious signal in health checks. The only remedy is a full restart.

This bug, documented in the project's DSV4_PD_DEADLOCK_ISSUE.md and linked to upstream SGLang issue #26454, was the reason --disable-overlap-schedule had been added to the decode launch script. The flag forces the scheduler to run the forward pass synchronously — the non-overlapped path includes an implicit TP barrier that prevents the desync. The cost is the lost throughput gain.

The Message Itself

Let us first read the subject message in full — the raw text of the assistant's reasoning and the bash command it executed:

``` ## Agent Reasoning Overlap-schedule is now enabled on decode and the router's ready, so I'm running the wedge-trigger test with serial single requests spaced out to maximize rank drift and watch for hangs or desynchronization. I'm measuring each request's completion time and HTTP code, checking that decode batch lines keep flowing throughout, and looking for timeouts that would indicate a wedge condition.

>

I'll vary the request content to exercise different bootstrap rooms, space them 1.5 seconds apart to force decode idle periods, and capture code plus latency for each one. If any request exceeds 30 seconds or returns a 000 code, that's a wedge. After the serial test completes, I'll run a concurrent burst to stress the idle-to-busy transitions and see if that triggers the issue. Overlap-schedule is now enabled on decode. Now actively try to trigger the desync wedge — the subagent's most reliable trigger is serial single requests with idle gaps (maximizes per-rank free-spin drift). Wedges classically appear within 3-7: [bash] ssh root@10.1.230.171 'echo "=== WEDGE TEST: 24 serial single requests, idle gap between (force rank drift) ==="; fail=0; maxt=0; for i in $(seq 1 24); do t=$(curl -s -m 30 http://127.0.0.1:30001/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\":\"deepseek-v4-flash\",\"messages\":[{\"role\":\"user\",\"content\":\"count to $i please\"}],\"max_tokens\":40,\"temperature\":0.6}" -o /dev/null -w "%{http_code}:%{time_total}" 2>/dev/null); code=${t%%:}; sec=${t##:}; [ "$code" != "200" ] && { echo "REQ $i WEDGE/FAIL code=$code time=$sec"; fail=$((fail+1)); } || printf "req%d=%ss " $i $sec; sleep 1.5; done; echo; echo "=== serial result: failures=$fail/24 ==="; echo "=== decode still emitting batches? (last 3 batch lines) ==="; journalctl -u sglang-dsv4-decode --no-pager -n 100 2>/dev/null | grep "Decode batch" | tail -3 | grep -oE "Decode batch.*#running-req: [0-9]+|gen throughput \(token/s\): [0-9.]+" | head; echo "=== health + prefill inflight (wedge => inflight pinned) ==="; for p in 30000 30002 30001; do echo -n "$p=$(curl -s -m5 -o /dev/null -w "%{http_code}" http://127.0.0.1:$p/health) "; done; echo; journalctl -u sglang-dsv4-prefill --no-pager -n 30 2>/dev/null | grep -oE "#inflight-req: [0-9]+" | tail -2' 2>&1 === WEDGE TEST: 24 serial single requests, idle gap between (force rank drift) === req1=0.846807s req2=0.885400s req3=0.888073s req4=0.880729s req5=0.875970s req6=0.893109s req7=0.865277s req8=0.883991s req9=0.884345s req10=0.883675s req11=0.892931s req12=0.836005s req13=0.888628s req14=0.893188s req15=0.867431s req16=0.880243s req17=0.878411s req18=0.886972s req19=0.892509s req20=0.878444s req21=0.842440s req22=0.888962s req23=0.880267s req24=0.892404s === serial result: failures=0/24 === === ... ```

The message contains two distinct layers. The Agent Reasoning block (the assistant's internal monologue) reveals the strategic thinking behind the test: the overlap scheduler has been enabled, the router is confirmed ready, and now the assistant is executing the wedge-trigger protocol it designed in previous messages. The reasoning explicitly calls out the key design choices — spacing requests 1.5 seconds apart to "maximize rank drift," varying content to "exercise different bootstrap rooms," and defining the failure threshold (30-second timeout or HTTP 000 code). The second part is the bash command and its output, which is the empirical result of the experiment.

The Decision to Test

This message is the culmination of extensive research that precedes it. In <msg id=13503>, the assistant reads the deadlock issue document and realizes that the overlap-scheduler desync is distinct from a separate NIXL abort-cascade wedge that had already been fixed. The NIXL fix does not address the overlap desync — the wedge can trigger from as few as 3-7 serial requests, even without aborts.

In <msg id=13504>, the assistant launches a research subagent to deep-dive into the scheduler code. The subagent's investigation confirms that the desync hazard still exists in the current codebase, though it has been narrowed by a "MIN-poll" admission gate. The minimal correct fix is identified: an unconditional "agree-or-defer" all_reduce(MIN(have_batch)) barrier on the TP CPU group at the top of the decode overlap loop. But implementing this fix requires code changes, testing, and validation — engineering time that could be spent on other priorities.

The assistant faces a strategic choice. Option A: implement the fix upfront, guaranteeing correctness before enabling the optimization. Option B: empirically test whether the wedge actually triggers on the current build, and only implement the fix if it does. The assistant chooses Option B, and the reasoning is sound. The MIN-poll machinery may have narrowed the window sufficiently that the wedge no longer triggers in practice. If so, the throughput gain can be captured immediately without code changes. If the wedge does trigger, the test will reproduce it quickly (within 3-7 requests per the issue history), and the assistant can roll back and implement the fix with clear evidence that it's needed.

This is a pragmatic, evidence-driven approach. It prioritizes empirical data over theoretical risk, while maintaining a safety net (the rollback plan) and a clear threshold for action.

The Wedge Test Design

The test designed in <msg id=13505> is elegant in its simplicity. The assistant:

  1. Backs up the current decode script (serve_dsv4_decode.sh.ovlsched_$(date +%H%M%S).bak) for instant rollback.
  2. Removes the --disable-overlap-schedule flag from the decode launch script using sed.
  3. Restarts the decode systemd service and waits up to 60×5=300 seconds for it to become healthy.
  4. Verifies the router is ready by sending a test request.
  5. Runs the wedge-trigger test: 24 serial single-turn requests, each asking the model to "count to $i please" with max_tokens=40 and temperature=0.6, spaced 1.5 seconds apart. The 1.5-second idle gap between requests is the critical detail. The desync wedge is most reliably triggered when TP ranks drift apart during idle periods. When no requests are being processed, each rank independently spins in its event loop, making run-versus-idle decisions based on local queue state. If the queues empty at slightly different times across ranks (due to network latency, scheduling jitter, or NUMA effects), the ranks can diverge. The idle gap maximizes this drift. The test measures each request's HTTP status code and completion time. Any request exceeding 30 seconds or returning a non-200 code is flagged as a wedge. After the serial test, the assistant also checks that decode batch lines are still flowing in the journal and that all three endpoints (prefill:30000, decode:30002, router:30001) return healthy HTTP 200.

The Results

All 24 requests pass. Latencies are remarkably consistent: ranging from 0.836s to 0.893s, with no outliers, no timeouts, and no failures. The decode worker continues emitting batch lines. All health checks return 200. The system is clean.

This is a non-trivial result. Per the issue history, the wedge was expected to trigger within 3-7 serial requests. The fact that 24 requests pass cleanly suggests that either:

Assumptions and Risks

The assistant makes several implicit assumptions in this message:

  1. The wedge is reproducible within a small number of serial requests. This assumption comes from the issue history (#26454) which reports 3-7 requests. If the wedge requires specific timing conditions that are rarer in practice, the test may pass even though the bug is still present.
  2. Serial requests with idle gaps are the most reliable trigger. This is based on the subagent's analysis of the desync mechanism, but the wedge may also require concurrent load, specific batch sizes, or abort events to manifest.
  3. The decode worker is the critical path. The overlap scheduler is also used on the prefill worker, but the assistant only enables it on decode. If the prefill worker's overlap scheduler can also desync, this test would not catch it.
  4. The cluster being idle makes the test safe. Restarting the decode worker on an idle cluster minimizes production impact, but a restart always carries risk — the PD bootstrap state could degrade (as seen in a previous production incident documented in the session), or the new configuration could introduce unexpected behavior under subsequent load. The most significant risk is a false negative: the test passes, the assistant proceeds to benchmark and eventually ships the overlap scheduler enabled, and the wedge triggers hours or days later under production load. This is the classic tension between empirical testing and theoretical correctness — a test can only prove the presence of a bug, never its absence.

Knowledge Flow

Input knowledge required to understand this message:

The Broader Significance

Message <msg id=13505> is a microcosm of the engineering challenges in production ML inference. It sits at the intersection of distributed systems correctness, GPU optimization, and operational pragmatism. The assistant must balance the user's explicit priority on correctness ("correctness above all else") with the practical reality that implementing a fix for a bug that may not trigger is a poor use of engineering time.

The message also illustrates a mature approach to A/B testing in production: backup before modification, verify health after restart, design a targeted stress test based on deep understanding of the failure mechanism, measure precisely, and interpret results with appropriate caution. The assistant does not declare victory after 24 clean requests — it merely records the data point and proceeds to the next phase (benchmarking and stress testing under concurrent load).

This measured, evidence-driven approach is what separates reliable production systems from fragile prototypes. The wedge test is not the end of the story — as the session's subsequent chunks reveal, the overlap scheduler does eventually trigger the desync under more aggressive stress testing, confirming that the fix is needed. But the empirical approach was still correct: it established a baseline, validated the test methodology, and ensured that when the fix is implemented, it will be deployed against a well-understood problem with reproducible evidence.

In the end, the message is a testament to the value of disciplined experimentation in complex systems. When dealing with race conditions, silent deadlocks, and distributed state, the most dangerous assumption is that you understand the system well enough to skip the test.