Measuring the Unmeasurable: Decomposing Decode Latency on Blackwell GPUs

The Message

At index 1354 in the conversation, the assistant issued a single bash command to a remote server:

ssh root@10.1.230.174 'source /root/ml-env/bin/activate && NCCL_IB_DISABLE=1 NCCL_P2P_LEVEL=5 python3 /tmp/decode_latency_breakdown.py 2>&1'

The output began:

======================================================================
GLM-5 Decode Latency Breakdown — 8x RTX PRO 6000 Blackwell (TP8)
Kernel: 6.14.11-5-bpo12-pve
======================================================================

--- 1. NCCL AllReduce Latency (8 GPUs, ring/tree) ---

        Size   Lat (us)  BW (GB/s)
-----------------------------------
        64 B       15.7       0.00
       512 B       19.9       0.03
      1.0 KB       16.8       0.06
      4.0 KB       41.1       0...

This message, on its surface, is a simple tool invocation — run a Python script and capture its output. But to understand why this message exists, what it represents, and what it reveals, we must trace the chain of reasoning that led to this precise diagnostic being executed at this precise moment in a months-long optimization campaign.

Context: The Efficiency Crisis

By the time this message was written, the team had been working for days to optimize inference throughput of the GLM-5-NVFP4 model on a system of 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The previous segment (segment 9) had achieved a dramatic 2x throughput improvement by updating SGLang to the latest commit and implementing a custom optimization called Opportunistic Expert Activation (OEA). Yet despite all this work, single-stream and dual-stream performance remained "glaringly low" — a phrase the user used in [msg 1349] to redirect focus away from high-concurrency benchmarks and toward understanding the fundamental bottleneck.

The theoretical maximum single-stream performance had been computed at approximately 309 tok/s (<msg id=1349 context>). The actual measured performance was around 10.36 tok/s. This represented a staggering 3.4% efficiency gap — the system was delivering only 3.4% of its theoretical peak. Something was fundamentally wrong, and neither the kernel tuning, the CUDA fixes, nor the SGLang updates had closed this gap.

Why This Message Was Written

The immediate trigger was the user's explicit instruction in [msg 1349]:

"let's focus on 1/2 stream for now, that's glaringly low. stop bench, and write a test util to simulate allreduce transfers and measure latencies/throughput to see if we're dealing with a latency issue or still a compute issue."

This instruction came after the assistant had launched a full multi-concurrency benchmark on the newly tuned kernel (6.14.11 with amd_pstate=active and processor.max_cstate=1). The user cut that benchmark short, recognizing that high-concurrency throughput was not the right question. The right question was: why is even a single stream of inference so slow?

The user's framing was precise: "see if we're dealing with a latency issue or still a compute issue." This dichotomy reflects a deep understanding of distributed inference bottlenecks. In tensor-parallel (TP) inference, each decode step requires:

  1. Compute: The actual matrix multiplications (GEMMs), attention, and routing operations
  2. Communication: NCCL AllReduce operations to synchronize gradients across GPUs If the bottleneck is communication latency, then the solution lies in reducing AllReduce frequency, fusing operations, or using faster interconnects. If the bottleneck is compute, then the solution lies in kernel optimization, reducing model size, or increasing GPU utilization. The user wanted to settle this question definitively.

The Diagnostic Tool

The assistant responded by writing decode_latency_breakdown.py ([msg 1352]) — a comprehensive diagnostic that measured each component of a decode step in isolation:

  1. NCCL AllReduce latency at various message sizes (64 bytes to 12 KB)
  2. BF16 GEMM latency at various matrix dimensions (simulating the model's linear layers)
  3. Simulated decode step combining both GEMMs and AllReduces
  4. Kernel launch overhead measurements The script was uploaded to the remote server in [msg 1353], and then executed in the subject message with specific NCCL environment variables: NCCL_IB_DISABLE=1 (disabling InfiniBand, forcing use of NVLink/NVSwitch for GPU-to-GPU communication) and NCCL_P2P_LEVEL=5 (setting the P2P communication level).

Input Knowledge Required

To fully understand this message, one needs:

  1. The system architecture: 8 RTX PRO 6000 Blackwell GPUs connected via NVSwitch, running in a Proxmox LXC container with PCIe passthrough. The GPUs are split across two NUMA domains (GPUs 0-3 on one, GPUs 4-7 on the other), with ~50 GB/s same-NUMA P2P bandwidth and ~37 GB/s cross-NUMA bandwidth (as measured in [msg 1343]).
  2. The model architecture: GLM-5-NVFP4 is a Mixture-of-Experts (MoE) model with 256 experts, using FP4 quantization for the GEMM weights. It uses tensor parallelism (TP8) across all 8 GPUs, meaning each AllReduce operation involves all 8 GPUs synchronizing partial results.
  3. The recent history: A kernel upgrade from 6.8.12 to 6.14.11 had just been completed, along with CUDA device major number fixes for the LXC container cgroup (<msg id=1336-1341>). The system had been tuned with amd_pstate=active, processor.max_cstate=1, and PCIe MaxReadReq increased from 512 to 4096 bytes.
  4. The performance gap: Theoretical maximum single-stream performance was 309 tok/s, actual was ~10 tok/s — a 30x gap that needed explanation.
  5. NCCL internals: Understanding that NCCL_IB_DISABLE=1 forces P2P communication over NVLink rather than InfiniBand, and NCCL_P2P_LEVEL=5 controls which transport protocols NCCL uses.

What the Output Revealed

The output shown in the message is truncated — it only displays the beginning of the NCCL AllReduce latency table. But the continuation in [msg 1355] tells the story:

The Thinking Process

The assistant's analysis in [msg 1355] reveals the logical deduction:

"This is extremely revealing. Let me digest the key findings... The 86ms gap is where all the overhead lives."

The assistant correctly identified that the gap must be in:

  1. FP4 GEMM overhead — The real model uses FP4 (4-bit floating point) quantized weights through CUTLASS grouped GEMM kernels, which have much higher dispatch overhead than simple BF16 torch.mm calls.
  2. MoE routing — With 256 experts, the routing logic (likely torch.compiled Python) adds significant overhead per token.
  3. Attention — FlashAttention decode kernels and Multi-Head Latent Attention (MLA) compression/decompression.
  4. Everything else — Layer norms, embeddings, sampling logic. This analysis directly answered the user's question: it was definitively a compute issue, not a communication latency issue. The AllReduce operations accounted for only 7% of the decode time. The remaining 93% was compute — but not the kind of compute that shows up in simple BF16 GEMM benchmarks. It was the overhead of FP4 kernel dispatch, MoE routing logic, and attention mechanics.

Assumptions and Their Validity

The diagnostic made several assumptions worth examining:

  1. BF16 GEMMs are a reasonable proxy for FP4 GEMMs: This was explicitly acknowledged as an approximation. The assistant knew that FP4 GEMMs through CUTLASS grouped GEMM would have much higher overhead, and the 8.9ms simulated decode was understood to be a lower bound, not an estimate.
  2. NCCL AllReduce measurements are representative: By using NCCL_IB_DISABLE=1 and NCCL_P2P_LEVEL=5, the assistant ensured the measurements reflected the actual communication path used during inference (NVLink, not InfiniBand). This was a sound methodological choice.
  3. The model has ~156 AllReduce operations per decode step: This number came from analyzing the model architecture — each transformer layer has multiple linear projections (Q, K, V, O, gate, up, down) that each require an AllReduce in tensor-parallel mode. This assumption was correct for GLM-5's architecture.
  4. Kernel launch overhead is negligible: The diagnostic did not explicitly measure CUDA kernel launch latency from the host side, which can be significant for small kernels. This was a minor blind spot, though the subsequent decode_gap_analysis.py tool ([msg 1355]) was designed to address it.

Output Knowledge Created

This message produced several critical pieces of knowledge:

  1. Quantified communication overhead: NCCL AllReduce takes ~42μs per operation, totaling ~6.6ms per decode step. This is significant (7% of total) but not the dominant factor.
  2. Established a compute-bound diagnosis: With communication accounting for only 7% of decode time, the bottleneck is definitively compute. This ruled out optimization strategies focused on reducing communication (like AllReduce fusion or gradient compression) as primary levers.
  3. Identified the FP4 GEMM overhead as the primary suspect: The 86ms gap between simulated BF16 decode (8.9ms) and real decode (95ms) pointed squarely at the FP4 GEMM kernel dispatch overhead as the largest unexplored term.
  4. Created a reusable diagnostic methodology: The decode_latency_breakdown.py script became a template for isolating and measuring individual components of the inference pipeline — a technique applicable beyond this specific model and hardware.
  5. Justified deeper investigation: The stark 10x gap between simulated and real performance motivated the creation of decode_gap_analysis.py ([msg 1355]) to directly measure FP4 GEMM overhead, MoE routing cost, and attention latency.

The Broader Significance

This message represents a pivotal moment in the optimization campaign. Before it, the team had been applying broad optimizations — kernel upgrades, system tuning, SGLang version bumps, expert parallelism experiments — without a clear understanding of where the bottleneck actually lay. The diagnostic reframed the entire problem.

The user's intuition that single/dual-stream performance was "glaringly low" was correct, but the cause was not what either party might have guessed. It was not a system configuration issue (the kernel upgrade and CUDA fixes had addressed those). It was not a communication issue (AllReduce was only 7% of the budget). It was not even a raw compute issue in the traditional sense (BF16 GEMMs were blazing fast). The bottleneck was in the overhead of FP4 kernel dispatch — the cost of setting up and launching thousands of small, specialized CUTLASS grouped GEMM kernels for the MoE experts.

This insight fundamentally changed the optimization strategy. Instead of tuning system parameters or reducing communication, the team needed to either: