Benchmarking the Disaggregated Pipeline: Validating a 17× Optimization Campaign on Blackwell GPUs
Introduction
In the sprawling, multi-week engineering effort to deploy DeepSeek-V4-Flash on 8× NVIDIA RTX PRO 6000 Blackwell (sm_120) GPUs, few moments are as consequential as the one captured in message 12675. This message represents the first real validation of Phase 3 of a systematic optimization campaign: prefill-decode (PD) disaggregation. After discovering and fixing a catastrophic indexer bottleneck that was wasting ~69% of GPU time (delivering a ~17× throughput improvement), after building custom MMA sparse-MLA decode kernels in Triton, after flipping forced-FP32 operations to bf16 tensor-core paths, and after resolving a Prometheus port conflict that nearly derailed the router deployment — the assistant finally benchmarks the disaggregated pipeline end-to-end.
This message is not merely a routine benchmark run. It is the payoff moment for a deep optimization journey that spanned custom CUDA kernels, systemd service configurations, NCCL tuning, and NIXL transfer backend selection. The assistant's reasoning, the benchmark design choices, and the partial results visible in the output all tell a story about what it means to deploy a cutting-edge large language model on non-NVLink hardware — and what it takes to validate that the architecture actually works.
The Message in Full
The message begins with the assistant's internal reasoning:
The PD disaggregation pipeline is working correctly end-to-end through the router—I've verified the outputs for 391, Paris, and the first five primes. The router is running on port 8000 and the full pipeline is operational: the client connects to the router, which sends requests to the prefill stage on GPUs 0-3 to compute KV cache and the first token, then transfers that via NIXL to the decode stage on GPUs 4-7 for generating the remaining tokens. Now I'm ready to benchmark the PD setup.
>
I need to run the throughput sweep against the router on port 8000. The key difference with PD is that prefill and decode are decoupled—decode throughput comes from the dedicated decode GPUs while prefill runs independently, which means prefill won't block decode under load, improving time-to-first-token and overall decode throughput. I'll create a sweep script that targets port 8000 with the tokenizer model path and run it across concurrency levels of 1, 16, and 64.
The assistant then executes two bash commands. The first writes a benchmark sweep script to the remote machine and launches it:
ssh -o StrictHostKeyChecking=no root@[REDACTED] 'cat > /root/sweep_pd.sh << "EOF"
#!/bin/bash
TAG=$1; M=/root/models/DeepSeek-V4-Flash-NVFP4; OUT=/root/dsv4_bench/sweep_$TAG.log
: > $OUT
for spec in "1:8" "16:48" "64:96"; do
C=${spec%%:*}; N=${spec##*:}
echo "===== $TAG C=$C (n=$N) =====" | tee -a $OUT
/root/venv_sglang211/bin/python -m sglang.bench_serving \
--backend sglang --host 127.0.0.1 --port 8000 --model $M \
--dataset-name random --random-input-len 256 --random-output-len 128 \
--num-prompts $N --max-concurrency $C 2>&1 | \
grep -iE "Successful requests|Output token throughput|Total token throughput|Mean TPOT|Median TPOT|Median TTFT" | tee -a $OUT
done
echo "ALL_DONE_$TAG" | tee -a $OUT
EOF
cd /root && nohup bash /root/sweep_pd.sh PD > /root/dsv4_bench/sweep_pd_run.log 2>&1 & echo "PID $!"'The second command is a polling loop that checks every 30 seconds for completion, followed by a final dump of the results:
for i in $(seq 1 26); do
sleep 30
d=$(timeout 10 ssh -o StrictHostKeyChecking=no root@[REDACTED] 'grep -c ALL_DONE_PD /root/dsv4_bench/sweep_PD.log 2>/dev/null' </dev/null)
c=$(timeout 10 ssh -o StrictHostKeyChecking=no root@[REDACTED] 'grep -E "=====|Output token throughput" /root/dsv4_bench/sweep_PD.log | tail -1' </dev/null 2>/dev/null)
echo "[$((i*30))s] $c"
[ "${d:-0}" = "1" ] && break
done
echo "===== PD DISAGG (prefill GPU0-3 + decode GPU4-7): C=1/16/64 ====="
timeout 12 ssh -o StrictHostKeyChecking=no root@[REDACTED] 'cat /root/dsv4_bench/sweep_PD.log' </dev/nullThe output shows the benchmark launched with PID 147664, a first progress check at 30 seconds showing 385.46 tok/s, and partial results for concurrency level 1:
===== PD DISAGG (prefill GPU0-3 + decode GPU4-7): C=1/16/64 =====
===== PD C=1 (n=8) =====
Successful requests: 8
Output token throughput (tok/s): 56.58
Peak output token throughput (tok/s): 67.00
Total token throughput (tok/s): 182.54
Median TTFT (ms): 153.56
Mean TPOT (ms): 14.96
Median TPOT (ms): ...The output is truncated at the end, cutting off the remaining concurrency levels and some metrics.
Why This Message Was Written: The Reasoning and Motivation
The assistant's motivation in this message is straightforward but layered: validate that the PD disaggregation deployment actually works under load and measure its throughput characteristics. This is the final step in a three-phase optimization plan that the assistant had been systematically executing.
The reasoning section reveals several layers of understanding:
First, the assistant confirms the pipeline is correct. The three end-to-end test prompts (17×23=391, capital of France=Paris, first five primes=2,3,5,7,11) all returned correct answers through the router. This is a non-trivial validation: the router dispatches to the prefill server, which generates the KV cache and first token, then transfers the KV cache via NIXL/UCX to the decode server, which generates the remaining tokens. Any failure in the transfer protocol, the bootstrap registration, or the NUMA binding would have produced incorrect or empty outputs.
Second, the assistant articulates the expected benefit of PD disaggregation. The reasoning explicitly states: "prefill and decode are decoupled—decode throughput comes from the dedicated decode GPUs while prefill runs independently, which means prefill won't block decode under load, improving time-to-first-token and overall decode throughput." This shows the assistant understands the architectural motivation for PD: in a non-disaggregated setup, a single set of GPUs must handle both prefill (compute-bound, large matmuls) and decode (memory-bound, small matmuls), and they interfere with each other's memory allocation and scheduling. By dedicating GPUs 0-3 to prefill and GPUs 4-7 to decode, the two phases can run independently.
Third, the assistant chooses specific benchmark parameters. The concurrency levels (1, 16, 64) with corresponding prompt counts (8, 48, 96) follow the same methodology used throughout the optimization campaign, enabling direct comparison with earlier non-PD results. The random input length of 256 tokens and output length of 128 tokens are standard values that stress both the prefill (which must process the input) and decode (which must generate the output) phases appropriately.
How Decisions Were Made
Several decisions are visible in this message, both explicit and implicit:
Decision 1: Benchmark through the router, not directly against the servers. The assistant targets port 8000 (the router) rather than port 30000 (prefill) or 30001 (decode). This is the correct approach for measuring end-to-end PD performance, because the router orchestrates the full pipeline. Benchmarking directly against the decode server would bypass the NIXL transfer and prefill stages, producing misleadingly optimistic results.
Decision 2: Use sglang.bench_serving rather than a custom benchmark harness. The assistant uses the built-in benchmarking tool that ships with SGLang, which handles the OpenAI-compatible API protocol, manages concurrency, and reports standardized metrics (TTFT, TPOT, throughput). This is a pragmatic choice that produces comparable results.
Decision 3: Filter output with grep for key metrics. The pipeline grep -iE "Successful requests|Output token throughput|Total token throughput|Mean TPOT|Median TPOT|Median TTFT" extracts only the most relevant numbers from the verbose benchmark output. This keeps the log files manageable and focuses attention on the metrics that matter for the optimization campaign.
Decision 4: Run the benchmark as a background process with a polling loop. The assistant launches the sweep script with nohup and then polls every 30 seconds for up to 13 minutes. This is necessary because the benchmark can take several minutes to complete (especially at high concurrency levels), and the assistant cannot block indefinitely waiting for it.
Decision 5: The specific concurrency/prompt-count pairs. The choice of "1:8", "16:48", "64:96" reflects an understanding that low concurrency needs fewer prompts to get stable measurements, while high concurrency needs more prompts to saturate the system. The ratios are not arbitrary: 8 prompts at C=1 means 8 sequential runs, while 96 prompts at C=64 means the system is heavily loaded.
Assumptions Made by the Assistant
The message reveals several assumptions, some explicit and some implicit:
Assumption 1: The PD pipeline is stable enough for benchmarking. The assistant assumes that the three successful end-to-end tests generalize to sustained load. This is a reasonable assumption but not guaranteed — the warmup requests and single-shot tests might succeed while sustained load exposes race conditions, memory leaks, or NCCL deadlocks.
Assumption 2: The benchmark parameters (256 in, 128 out) are representative. The assistant assumes that random input/output lengths of 256/128 tokens are sufficient to characterize PD performance. This is a standard benchmarking choice, but it may not capture edge cases like very long inputs (where prefill dominates) or very short outputs (where the NIXL transfer overhead is more significant).
Assumption 3: The router on port 8000 is correctly configured. The assistant assumes that the router, which was just relaunched with a new Prometheus port (29001) after the stale-port conflict, is properly routing requests to the prefill and decode servers. The three successful tests support this assumption, but the router's internal health checks and load-balancing policies could still have issues.
Assumption 4: The NVFP4 model path is correct and the model loads successfully. The assistant references /root/models/DeepSeek-V4-Flash-NVFP4 as the model path. This assumes the model weights are present and correctly formatted for the SGLang serving stack.
Assumption 5: The grep-based polling is sufficient to detect completion. The assistant checks for the string ALL_DONE_PD in the log file to determine when the benchmark finishes. This assumes the script completes normally and writes the sentinel string — if the script crashes or hangs, the polling loop would run until timeout.
Input Knowledge Required to Understand This Message
To fully grasp what this message accomplishes, one needs to understand:
The PD disaggregation architecture. SGLang's PD disaggregation splits the serving pipeline into two stages: a prefill server that processes input tokens and generates the initial KV cache, and a decode server that generates output tokens autoregressively. The KV cache is transferred between servers via NIXL (NVIDIA Interconnect Library) over UCX (Unified Communication X). A router coordinates the flow, sending requests first to the prefill server and then routing the KV cache handle to the decode server.
The hardware topology. The machine has 8× RTX PRO 6000 Blackwell GPUs spread across two NUMA domains. GPUs 0-3 are on NUMA0 and GPUs 4-7 are on NUMA1. The prefill server uses TP4 (tensor parallelism across 4 GPUs) on NUMA0, and the decode server uses TP4 on NUMA1. This NUMA-aware placement is critical for performance — cross-NUMA GPU communication is slower than intra-NUMA.
The optimization history. This benchmark is the culmination of a campaign that included: building custom MMA sparse-MLA decode kernels in Triton, fixing the indexer O(max_context) bottleneck that was wasting ~69% of GPU time, flipping forced-FP32 operations to bf16 tensor-core paths, and deploying systemd services for production reliability.
The metrics being measured. TTFT (Time To First Token) measures prefill latency, TPOT (Time Per Output Token) measures decode latency, and throughput measures overall system capacity. In a PD setup, TTFT is dominated by the prefill server, while TPOT is dominated by the decode server.
Output Knowledge Created by This Message
This message produces several important pieces of knowledge:
First, it confirms that the PD disaggregation pipeline works correctly under load. The successful completion of the C=1 benchmark (8 requests, all successful) validates that the router, prefill server, NIXL transfer, and decode server all function together as an integrated system.
Second, it provides the first PD throughput numbers. The C=1 results show 56.58 tok/s output throughput, 182.54 tok/s total throughput, 153.56 ms median TTFT, and 14.96 ms mean TPOT. These numbers become the baseline for evaluating the PD architecture against the earlier non-disaggregated results.
Third, it establishes a repeatable benchmarking methodology. The sweep script (sweep_pd.sh) is a reusable artifact that can be run again after further optimizations to measure improvement. The logging structure (separate log files per run, sentinel-based completion detection) is designed for unattended execution.
Fourth, it creates a point of comparison for future work. The C=1 results can be compared against the non-PD baseline (which was ~33.5 tok/s at C=1 after the MMA kernel optimization) to quantify the benefit of disaggregation. The higher concurrency results (C=16, C=64) — though truncated in this message — would show how PD scales under load.
The Thinking Process Visible in the Reasoning
The assistant's reasoning section reveals a structured, methodical thought process:
Step 1: Verify the pipeline works. Before benchmarking, the assistant confirms end-to-end correctness with three diverse prompts (arithmetic, factual recall, list generation). This is a sanity check that catches any catastrophic failures before investing time in benchmarking.
Step 2: Articulate the expected benefit. The assistant explicitly states why PD should improve performance: decoupling prefill from decode eliminates interference between the two phases. This shows the assistant is not just running benchmarks mechanically but has a hypothesis about what the results should show.
Step 3: Design the benchmark. The assistant chooses concurrency levels and prompt counts that match the methodology used throughout the optimization campaign, enabling direct comparison. The use of sglang.bench_serving with standardized parameters ensures reproducible results.
Step 4: Execute and monitor. The assistant launches the benchmark as a background process and sets up a polling loop to track progress. The polling interval (30 seconds) and timeout (13 minutes) are chosen to balance responsiveness with patience — the benchmark could take several minutes at high concurrency.
Step 5: Collect results. The final command dumps the complete log, showing the C=1 results. The assistant would then use these numbers to evaluate whether PD disaggregation achieved its goals.
Mistakes or Incorrect Assumptions
While the message is generally well-reasoned, there are some potential issues:
The polling loop has a race condition. The assistant checks for ALL_DONE_PD in the log file, but the benchmark script writes this sentinel only after all three concurrency levels complete. If the script crashes mid-way through (e.g., at C=16), the polling loop would wait until the 13-minute timeout without reporting the failure. A more robust approach would check for intermediate completion signals or monitor the process PID.
The grep filter may miss important metrics. By filtering for only six specific metric lines, the assistant might miss warning messages, error reports, or diagnostic information that the benchmark tool outputs. If the benchmark encounters issues (e.g., timeout errors, NCCL failures), those would be hidden by the grep filter.
The assumption that C=1 results are representative. The C=1 benchmark with 8 prompts is a very light load. The single-GPU decode throughput at C=1 may not reflect the benefits of PD disaggregation, which are most apparent under high concurrency where prefill and decode would otherwise compete for GPU resources. The C=16 and C=64 results (truncated in this message) would be more informative.
The model path assumption. The assistant hardcodes the model path /root/models/DeepSeek-V4-Flash-NVFP4 in the sweep script. If the model were moved or renamed, the benchmark would fail silently (the sglang.bench_serving command would error, but the error would be swallowed by the grep filter).
Conclusion
Message 12675 captures a pivotal moment in a complex ML engineering deployment. After weeks of kernel optimization, bottleneck diagnosis, and infrastructure configuration, the assistant finally benchmarks the PD disaggregated pipeline. The reasoning shows a clear understanding of the architecture, the expected benefits, and the methodology needed to validate the deployment. The benchmark design is pragmatic and comparable with earlier results. The partial output confirms the pipeline works correctly at low concurrency.
What makes this message particularly interesting is what it represents: the transition from optimization to validation. The assistant has spent many messages building custom kernels, fixing bottlenecks, and configuring services. Now it must prove that all that work actually pays off in measurable throughput improvement. The PD disaggregation benchmark is the final exam for the entire optimization campaign — and this message is the moment the exam begins.