The Pivot from Bottleneck Analysis to Throughput Execution: A Case Study in Evidence-Based ML Engineering

In the span of a single coding session, the trajectory of a production ML optimization campaign can shift dramatically based on a single piece of live evidence. This chunk of the DeepSeek-V4-Flash decode optimization story captures exactly such a pivot: from NCCL bottleneck analysis to a systematic campaign of re-enabling the overlap scheduler, driven by the user's directive to focus on decode throughput scaling from C60 to C90. What follows is a masterclass in evidence-based engineering — where hypotheses are tested against live data, costly dead ends are rejected with rigorous proof, and every configuration change is validated against correctness gates before deployment.

The Quick Question That Opened the Architecture

The session began innocuously. The user asked a seemingly simple question: "What's the current max parallel requests in decode?" ([msg 13471]). But this question, like so many in this conversation, revealed the hidden architecture of the deployment. The assistant discovered that the decode worker had max_running_requests=256 — set automatically by SGLang for the DeepSeekV4 model — but the CUDA graph batch size was capped at 32 (--cuda-graph-max-bs 32). This meant that any decode batch above 32 requests fell back to eager execution, losing the performance benefits of captured CUDA graphs (<msg id=13472-13473>).

The user, seeing that the token pool was healthy, immediately seized on this: "token pool is pretty healthy now, can you bump cuda graphs to 96?" ([msg 13474]). This was not a trivial request. The assistant had just root-caused and fixed a bf16 corruption bug that was specific to captured decode at batch sizes ≤32 — a multi-stream-overlap race condition during CUDA-graph capture. Bumping to 96 meant that batch sizes 33-96 would now run captured instead of falling back to eager, and the fix needed to hold across this expanded range.

Deploying and Verifying the CUDA Graph Expansion

The assistant approached this with characteristic rigor. First, a backup of the serve script was created. Then the configuration change was applied, the decode worker restarted, and the capture was verified. The results were clean: buckets now ranged up to 96 ([1,2,4,8,12,16,24,32,40,48,56,64,72,80,88,96]), using only 0.57 GB of GPU memory with 15.0 GB free — no out-of-memory errors (<msg id=13476-13477>).

But the critical test was the corruption re-verification. The assistant ran the repro agent at 96 sessions × 3 rounds and 60 sessions × 4 rounds. The result: 0% corruption across both configurations ([msg 13478]). The multi-stream-overlap fix held firm across the newly captured 33-96 range. A pure-decode benchmark at C=96 confirmed that the captured batch size reached 96 with zero eager fallback and zero errors. Throughput improved measurably: C=64 went from 657 to 711 tok/s, and C=96 achieved 760 tok/s (<msg id=13479-13480>).

The NCCL Bottleneck Investigation

With the CUDA graph expansion validated, the user pivoted: "Did we ever get to tuning nccl? What's the current bottleneck / any area for improvements? Gather evidence. Cluster currently under very heavy agentic workload" ([msg 13481]).

The assistant launched a comprehensive investigation. The NCCL environment was already tuned: NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=5, with 8-16 channels and 16 MB buffers. But the live metrics told a more nuanced story (<msg id=13482-13485>).

The GPU topology revealed a critical constraint: the 8 GPUs were split across two NUMA nodes, with no NVLink between them. Prefill ran on GPUs 0-3 (NUMA0) and decode on GPUs 4-7 (NUMA1). The decode GPUs were pegged at ~97% SM utilization but only drawing ~280W out of a 600W power budget — the classic signature of memory-bandwidth or communication-bound operation, not compute-bound. The KV transfer between prefill and decode was particularly slow: ~0.1-0.5 GB/s for 50-100 MB transfers, crossing NUMA0→NUMA1 over the UPI interconnect. Some transfers had tails exceeding 5 seconds under load.

The assistant identified the core bottleneck: decode was the constraint, and within decode, the combination of memory-bandwidth-limited MoE operations and PCIe-bound NCCL all-reduce was creating the sublinear scaling. Prefill GPUs sat mostly idle while decode struggled to keep up.

The Pivot to Decode Throughput Scaling

The user then refocused the investigation with two messages that would define the rest of the session. First: "cluster was a bit unloaded during the research/profiling, seeing load right now tho" ([msg 13488]). Then: "And most interesting now would be higher decode perf scaling, e.g. from C60 to C90 getting more linear" ([msg 13489]).

This was the decisive signal. The assistant pivoted from NCCL tuning to decode throughput scaling, analyzing live data from the production cluster under actual load. The scaling curve showed a clear pattern: marginal throughput gains declined from ~23 tok/s per added request at low concurrency to ~2.5 tok/s per added request at high concurrency. The step time followed step ≈ 18 + 1.05·bs ms — a fixed ~18 ms overhead plus ~1.05 ms per request. The fixed term included ~3 ms of NCCL all-reduce and ~7-9 ms of exposed CPU/scheduler overhead. The marginal term was driven by MoE weight-load and attention, both growing with batch size ([msg 13490]).

The Two-Batch Overlap Verdict

The user suggested a potential solution: "Two batch overlap would be extremely interesting for decode btw, esp at high C" ([msg 13491]). The assistant launched two independent subagents to investigate Two-Batch Overlap (TBO) feasibility — one for code analysis and one for profiling quantification.

Both subagents converged on the same verdict: TBO is a no-go on this stack (<msg id=13492-13493>). Four independent reasons emerged:

Hard config blocker: --enable-two-batch-overlap requires moe_a2a_backend to be configured, which demands expert parallelism (ep_size>1). But EP4 was already benchmarked on this hardware and performed worse (14 vs 23 tok/s) due to PCIe all-to-all overhead.
DSV4 isn't wired for it: TBO's init_new_tbo only accepts DeepseekV2, Qwen3Moe, and MiMoV2 layers. DeepseekV4DecoderLayer hits a NotImplementedError. V4's mHC hyper-connection residual structure is incompatible with TBO's op-split architecture.
Wrong target: TBO overlaps the EP all-to-all (dispatch/combine), not the TP all-reduce that was the actual bottleneck. With TP-only, those ops are no-ops.
No ceiling to chase: The "~19% all-reduce" figure was inflated by prefill in the profiled window. On the PD decode-only server, the all-reduce is ~3 ms/step — about 3-4% of wall time at high batch, and it's a batch-independent fixed cost. Hiding it would provide at most a one-time ~4% level shift, not the slope improvement needed for C60→C90 linearity. The assistant presented this verdict with characteristic clarity, along with ranked levers that would move the needle: finer CUDA graph buckets (trivial config), re-enabling the overlap scheduler (medium risk, ~8-10% gain), and MoE/attention occupancy work (the only thing that raises the asymptotic ceiling).

The Overlap-Scheduler Campaign

The user approved the plan: "Cluster unloaded; Actually skip #1 for now; Do #2 and validate connectness/benchmark, then go for #3; First commit any changes to current state, then write down project plans" ([msg 13494]).

The assistant began by committing the current state — the resolved bf16 corruption report and the deployed configuration — then wrote a comprehensive project plan document (DSV4_DECODE_PERF_PLAN.md) capturing the bottleneck analysis, evidence, and ranked optimization strategy (<msg id=13495-13500>).

The research into the overlap scheduler revealed a critical correctness concern. The --disable-overlap-schedule flag had been set to prevent a TP-collective desync wedge in the event_loop_overlap_disagg_* functions. When the overlap scheduler is enabled, the decode forward pass runs asynchronously — CPU batch preparation overlaps with GPU computation. But in PD-disaggregated mode with tensor parallelism, this creates a race condition: TP ranks can diverge on whether to run a batch or go idle, leading to asymmetric collective participation and a permanent deadlock. The NIXL abort fix (commit 90a52f44a) was a separate wedge and did not fix this desync hazard. Per upstream issue #26454, the desync could wedge from as few as 3-7 serial requests (<msg id=13501-13503>).

The subagent's code investigation confirmed the hazard still existed, though narrowed by a MIN-poll admission gate. The minimal correct fix would be an unconditional "agree-or-defer" all_reduce(MIN(have_batch)) barrier on the tp_cpu_group at the top of the decode overlap loop. But the assistant chose a correctness-first approach: first empirically measure the gain and actively try to trigger the wedge, before deciding whether to ship the scheduler fix ([msg 13503]).

The A/B Test and the Ambiguous Wedge

With the cluster idle, the assistant enabled overlap-schedule on decode only, backed up the serve script, and restarted. The wedge-trigger test ran 24 serial single requests with idle gaps to maximize rank drift. All 24 completed successfully with consistent ~0.88s latency — no wedge (<msg id=13504-13505>).

The throughput benchmark showed modest but real gains: C=1 at 55.4 (+4%), C=8 at 294 (+5%), C=32 at 575 (+2.6%), C=64 at 732 (+3-11%), and C=96 at 812.7 (+7%) — with zero errors or hangs during sustained load ([msg 13506]).

But the assistant recognized a critical gap: the sustained benchmark kept decode busy, which wouldn't trigger the idle-to-busy transitions that expose the desync. The real correctness test needed aggressive wedge stress with abort cascades and idle-busy churn. The stress test was launched — abort cascades with SIGINT mid-flight, followed by idle-to-busy bursts ([msg 13507]).

The result was ambiguous. The stress command returned only the header, suggesting the probe had hung. But when the assistant checked, decode was responding (probe returned 200 in 0.27s, batches flowing). However, a suspicious GPU utilization pattern emerged: GPUs 4 and 5 at 100% while GPUs 6 and 7 at 0% — exactly the desync signature where some TP ranks spin on a collective while others sit idle (<msg id=13508-13509>).

The assistant attempted to clean up and reassess, but the SSH command returned no output — a possible connection hiccup or a sign that the wedge had finally triggered. A minimal reconnect showed decode and prefill both active, GPUs idle, but one repro_agent process still lingering ([msg 13510]).

Lessons in Evidence-Based Engineering

This chunk of the session demonstrates several principles of production ML engineering at its best:

Evidence over intuition. Every hypothesis — from TBO to NCCL tuning to overlap-schedule re-enablement — was tested against live data before action. The TBO investigation consumed significant effort but produced a definitive verdict backed by code analysis and profiling, preventing weeks of wasted engineering.

Correctness as the first priority. The overlap-scheduler campaign was structured as a controlled experiment with rollback always ready. The assistant never shipped a configuration change without first validating it against the corruption repro and wedge triggers.

Documentation as engineering practice. Every finding was committed, every config change was backed up, and a comprehensive project plan was written and committed before execution began. This creates an evidence trail that survives beyond the session.

The discipline of the pivot. When the user redirected from NCCL tuning to decode scaling, the assistant pivoted cleanly — no sunk-cost fallacy, no attachment to the previous investigation. The evidence from the NCCL investigation was folded into the new plan, not discarded.

The session ends with the overlap-scheduler campaign in a state of productive tension: throughput gains are confirmed (~5-7% at high concurrency), the serial wedge test passed, but the abort-cascade stress test produced an ambiguous signal that demands further investigation. The assistant has committed to implementing the agree-or-defer fix to make the overlap scheduler provably correct before shipping — a decision that embodies the user's mandate: "correctness is of utmost importance."