The Hard Ceiling: When Prefill-Decode Disaggregation Meets a Kernel Bottleneck

Introduction

In the long arc of deploying cutting-edge language models on novel hardware, there are moments of triumph—when a complex orchestration system first works end-to-end—and moments of reckoning, when the numbers come back and reveal a structural gap between aspiration and reality. Message [msg 12392] in this opencode session is a reckoning. After successfully deploying DeepSeek-V4-Flash with prefill-decode (PD) disaggregation across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant reviews the benchmark results and confronts an uncomfortable truth: the disaggregation system works perfectly, but it cannot improve throughput because the decode side is fundamentally bottlenecked by sm_120 fallback kernels that run on CUDA cores instead of tensor cores.

This message is a turning point. It is not the message where a breakthrough occurs or a new optimization is discovered. It is the message where the assistant steps back from the configuration knobs, looks at the aggregate data, and correctly identifies that no amount of architectural rearrangement—not PD disaggregation, not NUMA pinning, not NIXL/UCX transport—can compensate for a kernel that launches only 64 blocks on a GPU with 170 streaming multiprocessors. It is a diagnostic verdict, delivered with clarity and precision, that sets the stage for the deeper kernel-engineering work that follows in subsequent chunks.

The Context: A Complex Deployment Comes Together

To understand why this message matters, we must first appreciate what led to it. The assistant had been working for hours—across multiple segments and chunks—to bring up DeepSeek-V4-Flash on a machine with eight RTX PRO 6000 GPUs (sm_120 architecture, Blackwell generation). The model itself is a massive Mixture-of-Experts (MoE) architecture with sparse Multi-Head Latent Attention (MLA), and the hardware is the cutting edge of NVIDIA's consumer/professional GPU line, but with a critical limitation: many of the optimized fused kernels (DeepGEMM, trtllm-gen, FP4 indexers) are architecture-gated to SM100 (the datacenter Blackwell) and do not run on sm_120.

The assistant had already navigated a series of challenging deployments. In earlier segments, it had:

The Message: Reading the Benchmark Results

In message [msg 12392], the assistant receives the benchmark results that were launched in the background in [msg 12390]. The benchmark tested three concurrency levels: c=1 (6 requests), c=8 (24 requests), and c=16 (32 requests), using random 256-input/256-output length prompts.

The assistant's reasoning block reveals what the data showed:

Looking at the performance data across different concurrency levels, I'm seeing that throughput plateaus around 25 tokens per second once we hit higher concurrency, with time-to-first-token degrading significantly as we scale from single to multi-request scenarios.

This is the critical finding. Throughput does not scale with concurrency—it flatlines at ~25 tok/s regardless of whether there are 1, 8, or 16 concurrent requests. And TTFT, which PD is supposed to improve by isolating prefill from decode contention, actually degrades at higher concurrency.

The assistant immediately identifies the root cause:

The decode GPUs are the real constraint here—they're running on slower hardware with a fallback path that can't keep up, so even though PD isolates prefill from decode, it doesn't actually improve throughput since decode itself is the bottleneck.

This is a crucial diagnostic insight. PD disaggregation helps when decode is fast but gets interrupted by prefill bursts. It does not help when decode is so slow that it is the system's throughput. In this case, the decode side is running at its maximum capacity—constrained by the sm_120 fallback attention kernel—and cannot go faster regardless of how much prefill isolation it receives.

The assistant then makes the direct comparison:

Comparing single-node TP4 to PD's dual TP4 setup, both hit roughly the same aggregate throughput of 24-25 tokens per second, but PD uses twice as many GPUs to get there.

This is a sobering result. PD consumed 8 GPUs instead of 4 to achieve the same throughput. The disaggregation overhead (KV transfer, additional communication) did not make things worse, but it also did not help. The decode bottleneck is the absolute ceiling.

The Reasoning Process: A Diagnostic Mindset

What makes this message particularly valuable is the clarity of the assistant's reasoning. The thinking process visible in the "Agent Reasoning" block shows a systematic diagnostic approach:

  1. Observe the aggregate behavior: Throughput plateaus, TTFT degrades. These are symptoms.
  2. Identify the bottleneck domain: The decode GPUs, not the prefill GPUs or the network. This is inferred from the fact that PD doesn't help—if prefill were the bottleneck, isolating it would improve throughput.
  3. Connect to prior knowledge: The sm_120 fallback kernel pathology was already identified in earlier profiling. The assistant connects the benchmark results to this known bottleneck rather than searching for new ones.
  4. Quantify the gap: PD uses twice the GPUs for the same throughput. This is a concrete, measurable inefficiency.
  5. Draw the structural conclusion: PD would shine if decode had headroom. It cannot help when decode is saturated. This reasoning is notable for what it does not do. It does not blame configuration, does not suggest tuning more knobs, does not propose switching backends or adjusting batch sizes. The assistant recognizes that the bottleneck is structural—it is in the kernel, not in the orchestration—and no amount of system-level rearrangement can fix it.

Assumptions and Their Validation

Several assumptions underpin this message, and the assistant implicitly validates them:

Assumption 1: The benchmark is representative. The assistant assumes that the random-input-length benchmark at 256/256 tokens captures the real performance characteristics of the system. This is a reasonable assumption for a first-pass evaluation, though real workloads with varying input/output lengths and prompt patterns could behave differently.

Assumption 2: PD is correctly configured. The assistant assumes that the PD deployment is working correctly—that prefill and decode are properly isolated, that KV transfer via NIXL/UCX is not introducing overhead, and that the router is correctly balancing requests. The verification steps in the bash command (checking GPU memory, port health, NUMA binding) confirm this assumption.

Assumption 3: The single-node comparison is valid. The assistant compares PD's 8-GPU throughput to the earlier single-node 4-GPU throughput. This assumes the earlier benchmark was run under comparable conditions (same model, same input lengths, same software version). Given that the assistant ran both benchmarks, this is a safe assumption.

Assumption 4: The bottleneck is in the attention kernel, not elsewhere. The assistant connects the aggregate throughput plateau to the earlier profiling finding that _tiled_sparse_decode_kernel consumes 63% of decode time. This is a well-supported inference, though the message does not re-run the profiler to confirm under PD conditions.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Knowledge of PD disaggregation: Understanding that prefill-decode disaggregation separates the two phases of LLM inference onto different GPUs, with KV cache transferred between them. The benefit comes from preventing prefill from stealing GPU time from decode.
  2. Knowledge of the sm_120 architecture: The RTX PRO 6000 Blackwell GPUs have sm_120 compute capability, which is a consumer/professional variant of the Blackwell architecture. Many optimized kernels target sm_100 (datacenter Blackwell) and fall back to slower Triton or CUDA implementations on sm_120.
  3. Knowledge of the attention kernel pathology: The _tiled_sparse_decode_kernel launches only 64 blocks (one per attention head) on a GPU with ~170 SMs, meaning ~60% of SMs are idle during the most expensive decode operation. This is a classic occupancy problem.
  4. Knowledge of the earlier profiling results: The 63% decode-time attribution and the 40× gap to the 1000 tok/s target were established in chunk 0 of this segment.
  5. Knowledge of the deployment topology: Prefill on GPU0-3/NUMA0, decode on GPU4-7/NUMA1, with NIXL/UCX transport and a router on port 8000.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. PD disaggregation does not improve throughput for DeepSeek-V4-Flash on sm_120. This is a negative result, but a valuable one. It saves future experimentation time by ruling out PD as a performance lever for this specific hardware-model combination.
  2. The decode bottleneck is structural, not configurable. The assistant establishes that no amount of system-level tuning (TP, EP, NCCL, CUDA graphs, etc.) can overcome the sm_120 fallback kernel limitation.
  3. PD consumes 2× GPUs for 0× throughput gain. This is a concrete efficiency metric that quantifies the cost of the bottleneck.
  4. The path forward is kernel engineering. By ruling out PD and configuration tuning, the message implicitly points to the only remaining lever: custom CUDA kernels for the sm_120 attention and MoE paths, similar to the 3-6× gains achieved for K2.6 in earlier segments.
  5. A validated deployment recipe. The bash command at the end of the message provides a verified recipe for PD deployment on this hardware: the launch scripts, port assignments, NUMA bindings, and health checks are all documented and confirmed working.

The Verification Step: Confirming the Deployment

After the diagnostic reasoning, the assistant runs a verification command that checks three things:

  1. GPU memory allocation: Prefill GPUs (0-3) show 71,876 MiB each; decode GPUs (4-7) show 80,226 MiB each. The higher memory on decode is expected because decode holds the full KV cache for ongoing sequences.
  2. Component health: All three ports (30000 prefill, 30001 decode, 8000 router) return healthy responses.
  3. NUMA binding: The prefill process is bound to CPUs 0-31 (NUMA0) and the decode process to CPUs 32-63 (NUMA1), confirming that the intended resource isolation is in effect. This verification is important because it rules out deployment misconfiguration as the cause of the poor throughput. The system is correctly set up—the bottleneck is in the kernel, not the orchestration.

Mistakes and Incorrect Assumptions

The message is largely accurate, but there are a few points worth examining:

The throughput ceiling may be slightly overstated. The assistant reports ~25 tok/s as the plateau, but this is at 256 output tokens with random input. Real workloads with shorter outputs or different prompt distributions could see different throughput. However, the relative comparison (PD vs single-node) would likely hold.

The assumption that PD cannot help under any circumstances may be premature. While PD does not improve peak throughput, it could still improve tail latency or TTFT under bursty request patterns where prefill spikes would otherwise delay decode. The benchmark used steady-state concurrency, which does not test this scenario. However, the assistant's reasoning is about throughput, not latency, and on that metric the conclusion is sound.

The message does not quantify the PD overhead. The KV transfer via NIXL/UCX has a cost (both latency and bandwidth), and the assistant does not measure this separately. It is possible that PD is slightly worse than single-node at low concurrency due to transfer overhead, and the ~25 tok/s plateau represents a balance between this overhead and the isolation benefit. A microbenchmark of transfer time would be informative but is not necessary for the throughput conclusion.

The Broader Significance

This message is a case study in the limits of system-level optimization. The assistant has done everything right: deployed the model, configured PD disaggregation, pinned processes to NUMA nodes, verified component health, run benchmarks at multiple concurrency levels. And yet the throughput is the same as the simpler single-node deployment.

The lesson is that when the bottleneck is in a single kernel that consumes 63% of GPU time and runs at 37% SM occupancy, no amount of architectural rearrangement at the system level can help. The only path forward is to rewrite that kernel—to attack the 38% of decode time spent in sparse attention with a custom implementation that achieves full occupancy and tensor-core utilization.

This is exactly what the assistant did in earlier segments for the K2.6 model, achieving 3-6× decode speedup with a custom sm_120 verify attention kernel. The message implicitly sets up the same playbook for DeepSeek-V4-Flash: the next step is to build a split-K tensor-core sparse-attention kernel that can break through the ~25 tok/s ceiling.

Conclusion

Message [msg 12392] is the moment in the session where the assistant transitions from system-level optimization to kernel-level diagnosis. It is a message of clarity, not of breakthrough—the assistant reads the benchmark data, correctly identifies the structural bottleneck, and documents the finding with precision. The PD disaggregation deployment is verified as working correctly, but the throughput ceiling is confirmed as a kernel problem, not an orchestration problem.

This message matters because it saves future effort. Without it, one might continue tuning PD parameters, trying different router policies, or adjusting batch sizes—all of which would produce negligible gains. By establishing that PD cannot improve throughput when decode is saturated, the assistant focuses attention on the real bottleneck: the sm_120 fallback attention kernel that must be rewritten to achieve the 300-600 tok/s that the hardware is capable of delivering.

In the broader narrative of the session, this message is the pivot point. It closes the chapter on configuration and opens the chapter on kernel engineering—the same playbook that delivered dramatic gains for K2.6 and will need to be repeated for DeepSeek-V4-Flash.