The Baseline That Changed Everything: Benchmarking DeepSeek-V4-Flash on Blackwell sm_120

In any systems engineering effort, there comes a moment when the abstract meets the real. After hours of environment setup, dependency resolution, model downloading, and import debugging, the assistant had finally achieved what seemed like a milestone: DeepSeek-V4-Flash, a 284B-parameter mixture-of-experts model with FP4 quantization, was running on SGLang across 4× RTX PRO 6000 Blackwell GPUs (sm_120 architecture). The model loaded, the CUDA graphs captured, and the server returned correct completions—"The capital of France is Paris." But correctness is only the first hurdle. The second, far more consequential question is: how fast?

Message <msg id=12379> is the moment that question was answered. It is a single bash command—a benchmark invocation—but it is also the hinge point of the entire segment. Before this message, the assistant was operating on hope and expectation. After this message, the entire trajectory of the session changed, pivoting from confident deployment into a grueling optimization campaign that would ultimately reveal a fundamental hardware bottleneck.

The Message

The assistant dispatched a benchmark command to the remote server running the DeepSeek-V4-Flash model:

timeout 400 ssh -o StrictHostKeyChecking=no root@10.1.230.171 '
mkdir -p /root/dsv4_bench
for C in 1 8; do
  echo "===== concurrency=$C ====="
  /root/venv_sglang211/bin/python -m sglang.bench_serving \
    --backend sglang --host 127.0.0.1 --port 30000 \
    --model /root/models/DeepSeek-V4-Flash \
    --dataset-name random --random-input-len 256 --random-output-len 256 \
    --num-prompts $((C*4)) --max-concurrency $C 2>&1 | grep -iE "Successful requests|Benchmark duration|Output token throughput|Mean TPOT|Median TPOT|Mean TTFT|Median TTFT|Total token throughput" 
done 2>&1 | tee /root/dsv4_bench/single_node_tp4.txt'

The results came back stark and unambiguous:

===== concurrency=1 =====
Successful requests:                     4         
Benchmark duration (s):                  71.64     
Output token throughput (tok/s):         10.22     
Peak output token throughput (tok/s):    11.00     
Total token throughput (tok/s):          17.03     
Mean TTFT (ms):                          753.55    
Median TTFT (ms):                        696.22    
Mean TPOT (ms):                          94.13     
Median TPOT (ms):                        94.12     
===== ...

The output was truncated at concurrency=8, but the concurrency=1 numbers told the story: 10.22 output tokens per second. To put that in perspective, the model card for DeepSeek-V4-Flash reports ~107 tok/s on vLLM with similar hardware. The assistant was seeing roughly one-tenth of the expected performance.

Why This Message Was Written

The motivation for this benchmark was rooted in a specific engineering workflow: measure before optimizing. The assistant had just completed the single-node TP4 bring-up—the model loaded, the server started, and a single chat completion returned correct text. But the assistant knew that correctness alone was insufficient. The next major deliverable was prefill-decode (PD) disaggregation, a complex distributed deployment that would split prefill and decode across separate GPU groups with KV cache transfer via NIXL/UCX. Before embarking on that multi-step orchestration, the assistant needed a baseline.

The reasoning in the preceding message <msg id=12378> makes this explicit: "Let me capture a quick baseline benchmark (bs=1 and bs=8) before building the PD deployment." The assistant was following a fundamental principle of performance engineering: never start optimizing without knowing where you stand. The baseline would serve as the reference point against which all future improvements would be measured.

There was also a secondary motivation: validation. The assistant had assembled a complex software stack—SGLang main (editable install), sglang-kernel 0.4.3, flashinfer 0.6.12, tilelang, and a custom CUDA 13.0 environment—and had resolved multiple import conflicts (leftover site-packages directories shadowing the editable finder, dual kernel package registrations). The benchmark would serve as an integration test, confirming that the entire stack functioned correctly under load.

The Thinking Process and Decision-Making

The assistant's reasoning reveals a careful, methodical approach. Several design decisions are visible:

Choice of benchmark parameters. The assistant selected --dataset-name random --random-input-len 256 --random-output-len 256. These are short, synthetic sequences—not representative of real workloads, but ideal for a baseline. Short sequences minimize the influence of prefill time on the decode measurement, giving a cleaner read of raw generation throughput. The --num-prompts $((C*4)) formula (4 prompts per concurrency level) ensures enough requests to reach steady state without overwhelming the server.

Choice of concurrency levels. Testing at C=1 and C=8 was a deliberate strategy. C=1 measures single-request latency and peak throughput in isolation—the best-case scenario for a single user. C=8 tests how the system behaves under moderate load, revealing contention effects, batching efficiency, and scheduler behavior. Skipping intermediate values (C=2, 4) was a pragmatic trade-off to keep the benchmark quick while still sampling both ends of the spectrum.

The grep filter. The assistant piped the benchmark output through grep -iE "Successful requests|Benchmark duration|Output token throughput|Mean TPOT|...". This is a signal-extraction pattern: the full benchmark output is verbose (it prints per-request latency distributions, percentiles, and detailed breakdowns), but the assistant was interested in only a handful of aggregate metrics. This choice reflects an understanding of which numbers matter most at this stage: throughput (tok/s) and per-token latency (TPOT).

The tee to file. By writing results to /root/dsv4_bench/single_node_tp4.txt, the assistant created a persistent record. This is a small but important engineering practice—benchmark results are fragile, terminals scroll, and the ability to compare against later measurements is essential.

Assumptions and Their Consequences

The assistant operated under several assumptions, some of which would prove incorrect.

Assumption: The model would achieve near-card performance. The assistant referenced the model card's ~107 tok/s on vLLM TP4 as an implicit target. This assumption was reasonable—the hardware (RTX PRO 6000 Blackwell) is modern, the software stack (SGLang main) is the latest, and the model is designed for FP4 inference. But the assumption failed to account for a critical architectural detail: the fast fused DSA/MoE stack (DeepGEMM, trtllm-gen, FP4 indexer) is arch-gated to SM100 (Blackwell's higher tier). The RTX PRO 6000, while Blackwell-based, is sm_120, which lacks these optimized kernel paths. The assistant was running on Triton fallback kernels, which are correct but dramatically slower.

Assumption: CUDA graphs would accelerate decode. The server startup logs confirmed that CUDA graphs were captured and active. The assistant assumed this would provide a meaningful speedup. While CUDA graphs do reduce launch overhead, they cannot accelerate the underlying kernel execution time—and the kernel itself was the bottleneck.

Assumption: The benchmark duration of 71.64 seconds was reasonable. For 4 prompts at concurrency=1 with 256 input and 256 output tokens, 71 seconds implies each request took ~18 seconds end-to-end. The assistant did not flag this as immediately alarming—but in hindsight, a 71-second benchmark for 4 short prompts should have been a red flag that something was fundamentally wrong.

Input Knowledge Required

To understand this message, one needs knowledge of several domains:

SGLang's benchmarking tooling. The sglang.bench_serving module is a built-in benchmark harness that sends requests to a running server and reports latency and throughput metrics. Understanding its flags (--num-prompts, --max-concurrency, --random-input-len) is necessary to interpret the test design.

Performance metrics for LLM serving. The assistant filtered for specific metrics: Output token throughput (tokens generated per second across all requests), TPOT (time per output token, the inter-token latency), TTFT (time to first token, the prefill latency), and total token throughput (including input tokens). Each metric reveals a different aspect of performance.

The hardware context. The assistant knew it was running on 8× RTX PRO 6000 Blackwell GPUs (sm_120 architecture) connected via PCIe (no NVLink), with tensor parallelism across 4 GPUs. This context is essential for interpreting the results—PCIe bandwidth limits all-reduce performance, and sm_120 determines which kernel paths are available.

The model architecture. DeepSeek-V4-Flash is a 284B-parameter MoE model with 13B active parameters per token, using FP4 (MXFP4) quantization for the experts and FP8 for attention weights. The model uses Multi-Head Latent Attention (MLA) with a sparse top-k selection of 512 tokens from a sliding-window cache.

Output Knowledge Created

This message produced several critical pieces of knowledge:

A quantified baseline. The assistant now knew that single-node TP4 achieved 10.22 tok/s at concurrency=1, with a mean TPOT of 94.13 ms and a mean TTFT of 753.55 ms. These numbers would serve as the reference for all subsequent optimization attempts.

A performance gap. The gap between 10 tok/s and the expected ~107 tok/s (or the user's target of ~1000 tok/s) was quantified at roughly 10× to 100×. This gap defined the scope of the optimization problem.

A benchmark methodology. The assistant established a reproducible benchmark procedure (random dataset, fixed sequence lengths, specific concurrency levels, filtered metrics) that could be re-run after each configuration change to measure improvement.

A diagnostic starting point. The 94 ms TPOT at batch size 1 was suspiciously high. In the next message, the assistant would use this number to begin root-cause analysis—checking whether CUDA graphs were active, whether GPUs were saturated, and which kernels were consuming the most time.

Mistakes and Incorrect Assumptions

The most significant mistake was the implicit assumption that the software stack would deliver reasonable performance on sm_120 hardware. The assistant had invested significant effort in getting the model to load and generate correctly, but had not yet verified that the critical kernel paths—sparse MLA attention, MXFP4 MoE, FP8 dequantization—had optimized implementations for sm_120. The benchmark revealed that they did not.

A second, subtler issue was the choice of benchmark parameters. Using 256 input and 256 output tokens is reasonable for a quick baseline, but it does not stress the prefill phase (which benefits from larger batches) or the decode phase (which benefits from longer sequences and KV cache pressure). A more comprehensive benchmark would include longer sequences (e.g., 2048 or 4096 tokens) to measure how throughput scales with context length. However, given that the assistant was planning to iterate quickly, the short-sequence baseline was a defensible choice.

A third issue was the truncated concurrency=8 results. The message shows "===== ..." indicating the output was cut off. The assistant likely hit the timeout (400 seconds) or the benchmark itself timed out. This meant the assistant had incomplete data at the higher concurrency level, which would need to be re-run later.

The Broader Significance

In the context of the entire session, <msg id=12379> is the moment of truth. Before this message, the assistant was in a "bring-up" mindset—solving import errors, configuring the server, verifying correctness. After this message, the assistant entered an "optimization" mindset, systematically testing every configuration lever: NCCL LL+Ring tuning, tilelang indexer fusion, non-marlin MoE backends, expert parallelism, and eventually switching to NVFP4 quantization and MTP speculative decoding.

The 10 tok/s number would become a recurring reference point. Every subsequent optimization—the switch to NVFP4 (which raised throughput to ~28 tok/s), the MTP speculative decoding (which boosted single-request throughput by 47%), the custom sm_120 verify attention kernel (which delivered 3-6× decode speedup in the K2.6 work)—would be measured against this baseline. The benchmark also revealed the fundamental bottleneck that no amount of configuration tuning could fix: the sm_120 fallback kernels for sparse-MLA decode and MXFP4 MoE, which run on CUDA cores instead of tensor cores, leaving the GPU's tensor-pipe utilization below 1%.

This message exemplifies a critical engineering principle: measure early, measure honestly. The assistant could have moved directly to PD disaggregation without benchmarking, assuming the single-node performance was adequate. Instead, it paused to take a measurement—and that measurement revealed the truth that would guide the rest of the session. The 10.22 tok/s number was disappointing, but it was also invaluable. Without it, the assistant would have spent hours building a distributed deployment on top of a fundamentally broken single-node foundation, only to discover the bottleneck later at greater cost.

In the end, the baseline benchmark was not just a performance measurement. It was a reality check, a diagnostic tool, and the catalyst for a deep investigation into the sm_120 architecture's limitations—an investigation that would ultimately produce custom CUDA kernels, a comprehensive findings report, and a roadmap for achieving the user's throughput targets.