The Decisive Profile: Capturing Steady-State Kernel Timing for DeepSeek-V4-Flash on Blackwell

In the high-stakes world of LLM inference optimization on novel hardware, there comes a moment when all the easy hypotheses have been exhausted and only one path remains: look at what the GPU is actually doing, kernel by kernel. This is the story of that moment — a single message in a long optimization campaign for the DeepSeek-V4-Flash-NVFP4 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs (sm_120 architecture), where the assistant, after ruling out communication bottlenecks, measurement errors, and memory bandwidth limits, finally reaches for the torch profiler to capture the ground truth.

The Bottleneck That Wouldn't Budge

The context leading into this message is a carefully constructed diagnostic chain. The assistant had been systematically optimizing the deployment of DeepSeek-V4-Flash, a Mixture-of-Experts model quantized to NVFP4 (4-bit floating point), on a server with eight Blackwell GPUs connected via PCIe (no NVLink). The throughput had plateaued at approximately 33 tokens per second regardless of concurrency — a hard ceiling that resisted explanation.

The user had provided three critical observations in the preceding messages ([msg 12529], [msg 12530], [msg 12531]): first, that early output appeared slower than later output, raising the question of whether "thinking tokens" were being measured incorrectly; second, that PCIe communication at C=64 was only ~200 MB/s, far below the 64 GB/s capacity of PCIe Gen5; and third, that the cufall profiler showed SM cycles pegged near 100% while power draw sat at only 340 W — barely more than half the 600 W TDP of the RTX 6000 Blackwell.

The assistant's reasoning in the preceding message ([msg 12532]) wove these three threads into a coherent diagnosis. The early-slow effect was not a measurement bug but the prefill ramp: at C=64, the median time-to-first-token was 13.9 seconds because all 96 prompts needed to be prefilled before decode could stabilize. The low PCIe traffic confirmed the bottleneck was not inter-GPU communication — the NCCL all-reduce for tensor parallelism consumed only 1–2% of GPU time. And the cufall data was the smoking gun: SMs resident and active nearly 100% of the time, yet drawing only 340 W, meant the GPU cores were stalled on memory latency, running SIMT fallback kernels instead of feeding the tensor-core or FMA pipelines. The model was ~30× below the memory-bandwidth roofline of 1.9 TB/s.

But which specific kernels owned that ~30 ms per request of marginal cost? The prior profile had been taken under different conditions (before the NVFP4 quantization switch), and the MoE kernels may have shifted. A fresh profile at steady-state decode was needed — and that is exactly what this message delivers.

The Profiling Orchestration

The message contains a single bash command executed over SSH on the remote server:

[bash] timeout 120 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'bash /root/prof_steady.sh 2>&1; echo "--- decode steps during profile ---"; grep -E "Decode batch" /root/dsv4_nvfp4c64.log | tail -3'

The actual work, however, was done by the script prof_steady.sh that the assistant had written in the previous message. Let us examine that script's design, because its parameters encode a deep understanding of the system's behavior.

The script launches sglang.bench_serving with 220 prompts, random input and output lengths of 256 tokens, and a maximum concurrency of 32. The choice of C=32 (concurrency level 32) is deliberate: it is high enough to saturate the GPU's decode capacity without triggering the OOM crashes that had plagued the C=64 runs ([msg 12524], [msg 12525]). The assistant had already established that the CUDA graph decoder was configured with max_bs=64, so C=32 would stay comfortably within the graph-captured path.

The 38-second sleep before triggering the profiler is equally deliberate. The assistant knew from the prior benchmark runs that the prefill ramp at C=64 took approximately 14 seconds of median TTFT; at C=32 with 220 prompts, the ramp would be shorter but still significant. Thirty-eight seconds provides a generous margin for all prompts to complete prefill and enter the steady-state decode phase, where each decode step processes a full batch of 32 concurrent requests.

The profiler itself is triggered via the SGLang HTTP endpoint /start_profile with num_steps=12 and both CPU and GPU activities enabled. Twelve decode steps at ~550 ms per step (the measured step time at C=16) would yield approximately 6.6 seconds of profiling data — enough for statistical significance without excessive overhead. The profiler captures kernel launch timings, durations, and metadata from the CUDA runtime, producing a trace file that can be parsed to identify the top kernels by total GPU time.

What the Output Reveals

The output of the command tells a nuanced story. The profiler responds "Start profiling." and the script completes with "DONE_PROF", confirming the orchestration worked. But the decode logs appended at the end reveal something important about the state of the server during profiling:

[2026-06-17 21:38:30 TP0] Decode batch, #running-req: 1, #full token: 256, ... gen throughput (token/s): 0.91
[2026-06-17 21:39:05 TP0] Decode batch, #running-req: 31, #full token: 9728, ... gen throughput (token/s): 24.21

The first log entry, at 21:38:30, shows only 1 running request with a throughput of 0.91 tok/s — this is the tail end of the prefill ramp, where most requests are still being prefilled and decode is barely started. The second entry, at 21:39:05, shows 31 running requests with throughput climbing to 24.21 tok/s. This is the system approaching steady state but not yet fully there. The profiler was triggered at approximately 21:38:38 (38 seconds after the load started), which means it captured a mix of late-ramp and early-steady-state behavior.

This is a subtle but important point. The assistant assumed that 38 seconds would be sufficient for full steady state, but the data suggests the ramp was still in progress. The profile therefore captures a transitional period rather than pure steady-state decode. This does not invalidate the results — the dominant kernels at C=31 are likely the same as at C=32 — but it introduces some noise. The throughput of 24.21 tok/s at 31 requests is below the expected ~33 tok/s ceiling, confirming the system was still ramping.

The Deeper Significance

This message represents a pivotal methodological shift in the optimization campaign. Up to this point, the assistant had been reasoning from aggregate metrics: throughput, step time, power draw, PCIe bandwidth. These are necessary but insufficient for kernel-level optimization. The torch profile would provide the granular data needed to identify whether the bottleneck was in the sparse MLA attention kernel, the MoE gating and expert computation, the RMS norm, or some other operation.

The assistant's reasoning in the preceding message reveals the stakes: "the key question now is whether the cutlass FP4 group-mm is actually being used at decode time or falling back to a slower path, and whether attention has become the dominant bottleneck instead of MoE." A fresh profile at steady C=32 would settle this by showing the top kernels ranked by GPU time, along with their memory throughput and occupancy characteristics.

Moreover, the design of the profiling orchestration itself — a self-contained script that launches load, waits, profiles, and cleans up — demonstrates a production-oriented mindset. The assistant could have run these steps interactively, but chose to encode them in a reusable script that could be re-run with different parameters. This is the hallmark of systematic engineering: turning ad-hoc investigation into repeatable measurement.

Assumptions and Their Implications

Several assumptions underpin this profiling run, and examining them reveals the assistant's mental model of the system.

The first assumption is that 12 decode steps provide a representative sample. At ~550 ms per step, 12 steps cover about 6.6 seconds of decode time, which at 31 concurrent requests represents roughly 372 token-generation events. This is statistically meaningful, but it assumes the kernel mix is stable across decode steps — that there is no phase variation due to, say, varying numbers of active experts or changing KV cache access patterns.

The second assumption is that the profiler overhead does not significantly perturb the timing. The SGLang profiler hooks into the PyTorch CUDA profiling infrastructure, which adds synchronization points and can alter kernel launch timing. The assistant does not account for this perturbation, but the alternative — profiling without the tool — would provide no data at all.

The third assumption is that C=32 is representative of the bottleneck at all concurrency levels. The assistant had already established that the marginal cost per request was approximately 30 ms regardless of batch size (step ≈ 52 + 30·N ms), suggesting a serial kernel that does not parallelize across batch. If this is true, then the kernel breakdown at C=32 should be identical to C=64, just scaled. But this assumption would be tested by the profile itself.

The Knowledge Produced

This message creates several pieces of actionable knowledge. First, it confirms that the profiling infrastructure works correctly — the SGLang profiler endpoint responds, the trace files are written, and the orchestration script executes without error. Second, it provides a timestamped record of server state during profiling, showing that the system was at 31 concurrent requests with 24.21 tok/s throughput. Third, and most importantly, it produces four new trace files (one per TP rank) totaling approximately 44 MB of compressed profiling data, ready for parsing.

The next message in the conversation ([msg 12534]) confirms the traces were created successfully, listing four new .trace.json.gz files from the profiling run. These traces would ultimately reveal the kernel-level breakdown that guided the next phase of optimization — the discovery that the "glue" operations (elementwise copies, reductions, and the indexer bmm) were dominating GPU time, leading to the dramatic ~17× throughput breakthrough when the indexer's O(max_context) bottleneck was identified and fixed.

Conclusion

This single message — a bash command executing a profiling orchestration script — is a microcosm of the entire optimization methodology. It demonstrates how the assistant synthesized multiple diagnostic signals (prefill ramp, PCIe bandwidth, SM activity, power draw) into a precise measurement strategy. It shows the transition from aggregate reasoning to kernel-level investigation, from hypothesis to data. And it reveals the engineering discipline of building repeatable, self-contained measurement tools rather than relying on ad-hoc interactive commands.

The profiling run captured in this message would prove to be the turning point in the campaign. The traces it produced would reveal that the bottleneck was not in the attention or MoE kernels at all, but in the indexer's O(max_context) computation — a finding that would unlock a 17× throughput improvement and transform the deployment from a disappointing 33 tok/s to a respectable 531 tok/s at C=64. But before that breakthrough, there was this moment: the assistant, faced with a stubborn bottleneck and a GPU running at half power, chose to look deeper.