Benchmarking Prefill-Decode Disaggregation: The First Performance Measurement

Introduction

In the middle of an intense optimization campaign for DeepSeek-V4-Flash on Blackwell (sm_120) GPUs, message [msg 12390] captures a pivotal moment: the launch of the first systematic performance benchmark after successfully deploying prefill-decode (PD) disaggregation. The message is deceptively simple—a single bash command that writes a shell script and launches it in the background on a remote server. But this benchmark represents the transition from "does it work?" to "how well does it work?"—a critical inflection point in any deployment effort.

The assistant has just achieved the headline deliverable of the segment: a fully functional PD disaggregation pipeline with prefill running on GPU0-3 (NUMA0) and decode on GPU4-7 (NUMA1), KV cache transfers mediated by NIXL/UCX, and a router on port 8000 coordinating traffic. Now comes the sobering reality check of measuring actual throughput.

The Message Content

The message contains a single tool call—a bash command executed via SSH on the remote machine 10.1.230.171. It writes a benchmark script to /root/run_pd_bench.sh and launches it in the background with nohup, capturing output to /root/dsv4_bench/pd_tp4.log:

timeout 40 ssh -o StrictHostKeyChecking=no root@10.1.230.171 '
cd /root
cat > /root/run_pd_bench.sh <<"EOF"
#!/bin/bash
M=/root/models/DeepSeek-V4-Flash
for spec in "1:6" "8:24" "16:32"; do
  C=${spec%%:*}; N=${spec##*:}
  echo "===== PD concurrency=$C (n=$N) ====="
  /root/venv_sglang211/bin/python -m sglang.bench_serving \
    --backend sglang --host 127.0.0.1 --port 8000 --model $M \
    --dataset-name random --random-input-len 256 --random-output-len 256 \
    --num-prompts $N --max-concurrency $C 2>&1 | \
    grep -iE "Successful requests|Benchmark duration|Output token throughput|Total token throughput|Mean TPOT|Median TPOT|Mean TTFT|Median TTFT|Mean E2E"
done
echo "ALL_DONE"
EOF
nohup bash /run_pd_bench.sh > /root/dsv4_bench/pd_tp4.log 2>&1 &
echo "bench PID $!"'

The benchmark script iterates over three concurrency configurations: 1 concurrent request with 6 total prompts, 8 concurrent with 24 prompts, and 16 concurrent with 32 prompts. Each run uses random input sequences of 256 tokens and requests 256 output tokens, measuring throughput, time-per-output-token (TPOT), time-to-first-token (TTFT), and end-to-end latency.

Why This Message Was Written: The Reasoning and Motivation

The immediate trigger for this benchmark is the successful deployment of PD disaggregation in the preceding messages. The assistant had just verified end-to-end generation through the router ([msg 12388]), confirming that the disaggregated pipeline functioned correctly—prefill on one set of GPUs, decode on another, KV transfer working. But "working" and "performing well" are two very different things.

The deeper motivation stems from the performance crisis that had already become apparent during the single-node deployment earlier in the segment. In chunk 0 of this segment, the assistant had measured single-node throughput at approximately 10 tok/s at batch size 1 and 25 tok/s at concurrency 16—far below the user's target of ~1000 tok/s. A GPU profile had traced 63% of decode time to a single kernel: _tiled_sparse_decode_kernel, the sm_120 Triton fallback for sparse MLA attention. This kernel launches only 64 blocks on ~170 SMs, creating a severe occupancy bottleneck.

PD disaggregation was supposed to help by isolating prefill and decode onto separate GPU sets, preventing prefill from blocking decode and vice versa. But the assistant clearly understood that disaggregation alone could not fix the fundamental kernel bottleneck. The benchmark was needed to quantify exactly how much PD helped (or didn't help) in practice, and to establish a baseline for further optimization.

There is also a methodological motivation here. Throughout this session, the assistant has demonstrated a strong commitment to measurement-driven optimization. Every configuration change—CUDA graphs, NCCL tuning, tilelang fusion, expert parallelism—was benchmarked. This message continues that pattern: before attempting any further optimization, the assistant first establishes a clean baseline under the new PD architecture.

How Decisions Were Made

Several design decisions are visible in the benchmark script, each reflecting careful reasoning:

Concurrency levels (1, 8, 16): The assistant chose three points that span the relevant range. Concurrency 1 measures pure decode latency without interference—the best-case scenario for a single user. Concurrency 8 is a moderate load that might show PD's advantage in parallelizing prefill and decode. Concurrency 16 pushes toward the server's capacity limits. The choice of 16 as the maximum likely reflects the memory constraints: each decode server was configured with --mem-fraction-static 0.70 and --cuda-graph-max-bs 32, but the MTP verifier (mentioned later in chunk 1) was consuming enough memory to effectively halve the batch capacity.

Prompt counts (6, 24, 32): The assistant allocated roughly 2× the concurrency level as the total number of prompts. This is a standard practice in serving benchmarks: you need enough prompts to reach steady-state behavior but not so many that the benchmark takes excessively long. With 256 output tokens and ~10 tok/s decode speed, each prompt takes ~25 seconds, so 32 prompts at concurrency 16 would complete in about 50 seconds—a reasonable duration.

Input/output length (256/256): The choice of 256 input tokens and 256 output tokens is significant. The assistant deliberately chose a relatively short context to keep benchmark runtime manageable given the slow decode. But this choice also has implications: PD disaggregation's benefits are more pronounced with longer prefill inputs (where prefill time dominates), so the 256/256 split might understate PD's advantage. The assistant likely prioritized getting baseline numbers quickly over comprehensive characterization.

Metrics filtered: The grep pattern selects "Successful requests", "Benchmark duration", "Output token throughput", "Total token throughput", "Mean TPOT", "Median TPOT", "Mean TTFT", "Median TTFT", and "Mean E2E". This is a carefully curated set that captures both throughput (tokens per second) and latency (TPOT, TTFT, E2E). The assistant is looking for two things: (1) whether PD improves throughput over single-node, and (2) whether latency is acceptable. The inclusion of both mean and median for TPOT and TTFT suggests awareness that latency distributions might be multimodal under PD (some requests get fast decode, others wait for prefill to complete).

Background execution: The use of nohup and output redirection to a log file reflects practical experience with long-running benchmarks over SSH. A 50-second benchmark could easily be killed by SSH timeout or connection interruption. By running in the background and writing to a file, the assistant can check results later without keeping the SSH session alive.

Assumptions Embedded in the Benchmark

The benchmark makes several assumptions that are worth examining:

Assumption 1: The router correctly distributes load. The benchmark targets port 8000 (the router), not the individual prefill or decode servers. This assumes the router properly implements PD-aware scheduling—sending prefill requests to the prefill server and decode requests to the decode server, with KV transfer orchestrated between them. If the router has bugs or misconfiguration, the benchmark results would reflect router performance, not PD performance.

Assumption 2: Random input/output lengths are representative. The benchmark uses random input sequences of exactly 256 tokens and requests exactly 256 output tokens. Real workloads have variable input lengths, different output lengths, and different token distributions. The benchmark assumes that the relative performance at 256/256 is representative of general behavior.

Assumption 3: The system reaches steady state. With only 2× concurrency worth of prompts, the benchmark assumes that the system reaches steady-state behavior within those prompts. For concurrency 16 with 32 prompts, this means each of the 16 slots handles exactly 2 requests. This might not be enough to warm up CUDA graphs, fill the radix cache, or trigger garbage collection—all of which affect long-running performance.

Assumption 4: The benchmark itself doesn't introduce overhead. Running sglang.bench_serving on the same machine as the servers creates potential interference. The benchmark client competes for CPU, memory bandwidth, and network resources with the SGLang servers. On a machine with many CPU cores, this might be negligible, but it's still an assumption worth noting.

Assumption 5: TPOT and TTFT are the right metrics. The assistant focuses on token-generation latency and throughput. But for PD disaggregation, there are other important metrics: KV transfer latency (how long does NIXL/UCX take to move KV cache between GPU sets?), prefill-decode imbalance (is one server waiting for the other?), and router scheduling overhead. These are not captured by the selected metrics.

Mistakes and Incorrect Assumptions

Several potential issues can be identified in retrospect:

The benchmark may not capture PD's key advantage. PD disaggregation primarily helps when prefill and decode compete for the same GPU resources. With 256-token inputs, prefill is very fast (a few milliseconds), so the benefit of isolating it on separate GPUs is minimal. The real advantage of PD appears with long-context prefills (thousands of tokens) where prefill can consume significant GPU time. By choosing short inputs, the assistant may be benchmarking PD in the regime where it helps least.

The concurrency range may be too narrow. Given that the decode server was configured with --cuda-graph-max-bs 32, the assistant could have tested higher concurrency levels (e.g., 24 or 32). The choice to stop at 16 might miss the throughput peak. However, this was likely a practical decision—higher concurrency would require more prompts and longer benchmark time, and the assistant was already concerned about slow decode.

The benchmark doesn't isolate the disaggregation overhead. There's no comparison to a single-node baseline under the same conditions. The assistant had single-node numbers from earlier (~10 tok/s at bs=1, ~25 tok/s at C=16), but those used different settings (different batch sizes, different input lengths). A proper comparison would require re-running the single-node benchmark with the same 256/256 configuration. Without this, it's impossible to attribute any performance difference to PD versus other factors.

Background execution risks missing errors. By running the benchmark in the background and filtering output through grep, the assistant might miss important error messages or warnings. If the benchmark crashes midway through (e.g., OOM at concurrency 16), the log file would contain partial results that might be misinterpreted as complete.

Input Knowledge Required

To understand and write this message, the assistant needed:

  1. Knowledge of the SGLang benchmark tool: The sglang.bench_serving module and its command-line interface, including the --backend, --host, --port, --model, --dataset-name, --random-input-len, --random-output-len, --num-prompts, and --max-concurrency flags.
  2. Knowledge of the deployment topology: That the PD router is on port 8000, the model path is /root/models/DeepSeek-V4-Flash, and the servers are running on the same machine.
  3. Knowledge of typical serving benchmarks: The standard metrics (throughput, TPOT, TTFT, E2E) and the convention of using 2× prompts relative to concurrency.
  4. Knowledge of the model's expected performance: The assistant knew from earlier profiling that decode was running at ~10 tok/s, which informed the choice of 256 output tokens (to keep benchmark duration reasonable).
  5. Knowledge of bash scripting: The parameter expansion syntax (${spec%%:*} and ${spec##*:}) for splitting the concurrency:num_prompts pairs, nohup for background execution, grep for filtering output.

Output Knowledge Created

This message produces a log file at /root/dsv4_bench/pd_tp4.log containing benchmark results for three concurrency levels. The specific numbers are not visible in this message (they would appear in subsequent messages), but the structure of the output is determined by the grep filter: throughput metrics (output token throughput, total token throughput), latency metrics (TPOT, TTFT, E2E with mean and median), and request counts.

This benchmark output serves as the baseline for all subsequent optimization work in the segment. It quantifies the starting point before the assistant attempts MTP speculative decoding, NVFP4 quantization, and other optimizations described later in chunk 1. Without this baseline, it would be impossible to measure whether any optimization actually improves performance.

The Thinking Process Visible in the Message

The message reveals several layers of reasoning:

Practicality over perfection: The assistant chose a benchmark that is "good enough" rather than comprehensive. Three concurrency levels, one input/output length, background execution. This reflects an understanding that the first benchmark is a reconnaissance mission, not a final characterization. The goal is to get directional numbers quickly, then iterate.

Measurement hygiene: The assistant carefully separates benchmark output from server logs by writing to a dedicated file (/root/dsv4_bench/pd_tp4.log). The grep filter extracts only the essential metrics, making the results easy to parse. This attention to data cleanliness is characteristic of the assistant's methodology throughout the session.

Awareness of time constraints: The 40-second SSH timeout suggests the assistant expected the script-writing to be fast, but the benchmark itself would run longer (hence nohup). The assistant is working within the constraints of the SSH session while planning for a benchmark that outlives it.

Systematic iteration: The benchmark is structured as a loop over configurations, not separate commands. This allows the assistant to add more configurations later by editing the script rather than rewriting it. The ALL_DONE marker at the end provides a clear signal that the script completed successfully.

Conclusion

Message [msg 12390] is a textbook example of measurement-driven engineering. It appears at the precise moment when the assistant needs to transition from "does it work?" to "how well does it work?"—a transition that separates hobbyist deployment from professional systems engineering. The benchmark script is carefully designed to capture the metrics that matter (throughput, TPOT, TTFT) at the concurrency levels that are relevant (1, 8, 16), using a methodology that is reproducible and practical.

The results of this benchmark would go on to inform the entire optimization campaign that follows: the MTP speculative decoding experiments, the NVFP4 quantization switch, and ultimately the discovery that the sm_120 fallback kernels are the fundamental bottleneck. Without this baseline measurement, none of those subsequent decisions would have a solid empirical foundation. In the story of deploying DeepSeek-V4-Flash on Blackwell GPUs, message [msg 12390] is the moment when the team stopped guessing and started measuring.