Profiling the Unseen: How a Single Diagnostic Task Unlocked 2.85x Performance Gains for GLM-5 on Blackwell

The Message in Full

Before diving into analysis, here is the exact message as it appeared in the conversation (with internal IP redacted):

[assistant] [task] {"description":"Profile single-request decode perf","prompt":"SSH to container: `ssh root@[REDACTED]`\n\nGLM-5 GGUF model on vLLM with 8x RTX PRO 6000 Blackwell (SM120) is only doing ~20 tok/s for single-request decode. That's ~50ms per decode step which is way too slow. We need to profile and fi...
task_id: ses_384fe1882ffe4XoWuH6u3WPdov (for resuming to continue this task if needed)

<task_result>
Extremely consistent: **50.5ms per decode step, 19.7-19.8 tok/s**. The non-streaming measurement was inflated by HTTP overhead.

Now let me produce the final comprehensive report:

---

## Complete Profiling Results & Analysis

### Confirmed Performance
- **Decode ITL**: 50.5ms (±0.5ms), **19.8 tok/s** (single request)
- **TTFT**: 56-66ms (single token prefill)
- **GPU utilization**: 91% (busy,...

Introduction

In the high-stakes world of large language model deployment, a model that produces correct output but at painfully slow speeds is only half a victory. When the GLM-5 GGUF model finally loaded and generated coherent text on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the team faced a sobering reality: single-request decode throughput was a mere 20 tokens per second. At that rate, a user waiting for a 500-token response would stare at a blank screen for 25 seconds. The user's demand was unambiguous — "gat you profile and try to improve to at least 100/s." Message [msg 1986] represents the critical turning point where the assistant shifted from debugging correctness to optimizing performance, and it embodies a fundamental engineering principle: before you can fix a performance problem, you must first measure it.

This message, a single task tool call dispatched to a subagent, is deceptively simple on the surface. But it is the linchpin of an optimization journey that would ultimately yield a 2.85× throughput improvement (from 20 to 57 tok/s) through CUDAGraph enablement and NCCL protocol tuning. More importantly, it reveals the assistant's reasoning process, its assumptions about where bottlenecks might lie, and the disciplined methodology of using structured profiling to replace guesswork with data.

The Motivation: Why This Message Was Written

The immediate trigger for [msg 1986] was the user's message at [msg 1984]: "20t/s is still really slow, gat you profile and try to improve to at least 100/s." But the deeper motivation was rooted in the preceding 14 segments of work. The assistant had just spent an enormous effort debugging two subtle bugs — an MLA output buffer disconnect caused by a phantom tensor in a custom PyTorch op dispatch, and a GGUF shard ordering bug in fused linear layers ([msg 1977]). These fixes had restored correct model output, but the performance was abysmal.

The assistant's response at [msg 1985] shows its immediate reasoning: "At 20 tok/s for a single request, that's ~50ms per token decode, which is indeed very slow for 8 GPUs." This observation is crucial — the assistant intuitively knew that 8× Blackwell GPUs should be capable of far more than 20 tok/s for a 744B parameter MoE model at Q4_K_XL quantization. The gap between expected and actual performance demanded investigation.

The decision to dispatch a profiling task rather than blindly trying optimizations reflects a mature engineering mindset. The assistant could have attempted to enable CUDAGraph immediately, or tried reducing tensor parallelism, or experimented with different quantization paths. Instead, it chose to gather data first. The task tool was the vehicle for this: it spawned a subagent that would run a multi-round investigation using torch.profiler and NVIDIA Nsight Systems, returning a comprehensive breakdown of where every millisecond was going.

How Decisions Were Made: The Profiling Strategy

The profiling strategy embedded in the task prompt reveals several deliberate decisions. First, the assistant chose to profile at the single-request level rather than at high concurrency. This is because single-request decode is the purest measure of the model's computational efficiency — it isolates the model's intrinsic performance from batching and scheduling effects. The user's target of 100 tok/s was also stated as a single-request metric, making this the right baseline.

Second, the assistant opted for a subagent (the task tool) rather than running profiling commands directly in the parent session. This is a significant architectural decision. The subagent can conduct its own multi-turn conversation, iterating on profiling runs, adjusting parameters, and producing a synthesized report. The parent session remains blocked during subagent execution, but the trade-off is a more thorough and autonomous investigation. The task description was truncated in the message, but the full prompt likely included instructions to run vllm serve with profiling flags, capture kernel traces, and analyze the output.

Third, the assistant chose to measure both streaming and non-streaming decode latencies, recognizing that HTTP overhead could inflate measurements. The task result confirms this wisdom: "The non-streaming measurement was inflated by HTTP overhead." By measuring inter-token latency (ITL) from the streaming response, the subagent obtained a clean 50.5ms per decode step.

The task result also reveals that the subagent confirmed GPU utilization at 91%, which is high but not saturated. This immediately signals that the bottleneck is not purely compute-bound — there is likely communication or framework overhead eating into the remaining 9% of time.

Assumptions Embedded in the Message

Every diagnostic effort rests on assumptions, and [msg 1986] is no exception. The most significant assumption is that the bottleneck is discoverable through GPU-level profiling tools. The assistant implicitly assumed that torch.profiler and nsys would capture the critical path — that the 50ms per step would decompose into identifiable components like matrix multiplications, attention kernels, NCCL allreduce calls, and framework overhead. This assumption proved correct, but it was not guaranteed. If the bottleneck had been in the Python-level scheduler or in CPU-side preprocessing, GPU profiling might have missed it.

A second assumption is that the 8-GPU configuration is appropriate for this model. The assistant did not question whether tensor parallelism of 8 was optimal — it accepted the existing configuration and profiled within that constraint. Later analysis ([msg 1998]) would reveal that TP=8 is actually optimal for this memory-bandwidth-bound workload because more GPUs mean less weight per GPU and faster decode. But at the time of profiling, this was an unexamined assumption.

A third assumption is that the GGUF quantization path is not itself the bottleneck. The model uses Q4_K_XL quantization, which requires dequantization kernels that may not be optimized for Blackwell's SM120 compute capability. The assistant implicitly assumed these kernels were performant enough that the bottleneck lay elsewhere. This assumption was validated by the profiling results, which showed that GPU compute (including dequantization) was not the dominant term.

Perhaps the most consequential assumption was that CUDAGraph would be the primary optimization target. The assistant's subsequent actions ([msg 1987]) show it immediately zeroing in on NCCL allreduce overhead and CUDAGraph as the solution. This assumption shaped the entire optimization trajectory. While CUDAGraph did provide a 2.15× improvement (from 20 to 43 tok/s), it was not the complete answer — NCCL protocol tuning (NCCL_PROTO=LL) was needed to reach 57 tok/s, and the 100 tok/s target remained elusive due to fundamental PCIe bandwidth limitations.

Input Knowledge Required

Understanding [msg 1986] requires substantial domain knowledge spanning multiple layers of the ML inference stack. The reader must grasp:

Output Knowledge Created

The profiling task produced a precise, quantified understanding of the decode step that had previously been a black box. The key outputs were:

  1. Baseline performance: 19.8 tok/s single-request, 50.5ms per decode step, 56-66ms time-to-first-token (TTFT). These numbers became the reference point for all subsequent optimization efforts.
  2. GPU utilization: 91% busy, confirming the GPUs were active but not fully saturated. The 9% idle time hinted at synchronization or communication stalls.
  3. Bottleneck identification: The subagent's comprehensive report (referenced in the task result but expanded in subsequent messages) would reveal that NCCL allreduce consumed 42% of decode time — 157 calls at ~135μs each, totaling ~21ms. This was the single largest component.
  4. Methodological validation: The profiling confirmed that HTTP overhead was inflating non-streaming measurements, establishing streaming inter-token latency as the correct metric. This knowledge fundamentally changed the optimization strategy. Without it, the assistant might have pursued GPU compute optimizations (kernel fusion, quantization tuning) that would have addressed only 57% of the problem. Instead, the focus shifted to reducing NCCL overhead — first through CUDAGraph (which amortized kernel launch overhead and captured NCCL calls into a replayable graph), then through NCCL protocol tuning.

Mistakes and Incorrect Assumptions

The most notable mistake revealed in the aftermath of this message is the assumption that the MLA direct-call patch was necessary for CUDAGraph compatibility. In [msg 1987], the assistant immediately connects the profiling results to the CUDAGraph hypothesis: "The key insight is that --enforce-eager prevents CUDA graph replay, which would batch all those 157 NCCL calls and 1500+ kernel launches. We already tested CUDAGraph and it produced garbage — that's likely related to our forward_impl direct-call fix bypassing the custom op."

This reasoning was incorrect. The CUDAGraph garbage output was actually caused by the GGUF shard ordering bug, not by the MLA direct-call patch. When the assistant later reverted the direct-call patch and tested CUDAGraph ([msg 1992]), it discovered that "the shard ordering fix was the real root cause — the direct-call patch was never needed." This is a classic debugging pitfall: fixing one bug (shard ordering) masked the fact that another change (direct-call patch) was unnecessary, and the two were conflated during diagnosis.

Another subtle mistake was the initial interpretation of GPU utilization. 91% utilization sounds high, but the assistant correctly recognized that it was not saturated. However, the initial hypothesis was that the remaining 9% was framework overhead. The profiling later revealed that the true bottleneck was not GPU compute at all but rather NCCL communication — the GPUs were busy computing but spending a large fraction of that time waiting for allreduce results.

The Thinking Process Visible in the Reasoning

While the message itself is a single task dispatch, the surrounding messages reveal the assistant's reasoning trajectory. At [msg 1985], the assistant creates a todo list with four items: profile, analyze, implement, re-benchmark. This structured approach reflects a systematic optimization methodology.

The decision to use a subagent for profiling rather than running commands directly is itself a thinking artifact. The assistant could have run vllm serve with --enforce-eager and used curl to measure latency. Instead, it invested in a deeper investigation using professional profiling tools. This suggests the assistant anticipated that the bottleneck would not be obvious from surface-level measurements and required kernel-level insight.

The task result's mention of "Extremely consistent: 50.5ms per decode step, 19.7-19.8 tok/s" reveals another thinking pattern: the assistant valued consistency as a diagnostic signal. A highly consistent decode time suggests a deterministic bottleneck (like a fixed number of NCCL calls) rather than a stochastic one (like memory contention or scheduling jitter). This consistency was itself a clue that the bottleneck was structural rather than environmental.

Conclusion

Message [msg 1986] is a masterclass in disciplined performance engineering. Faced with a user demanding 5× throughput improvement, the assistant did not guess at optimizations or apply random tweaks. Instead, it invested in measurement — dispatching a subagent to profile the decode step at the kernel level, producing a precise breakdown of where every millisecond was going. The resulting insight that NCCL allreduce consumed 42% of decode time over PCIe interconnects became the foundation for a targeted optimization campaign that ultimately delivered 57 tok/s, a 2.85× improvement.

The message also illustrates the power of the task tool as an architectural pattern for complex investigations. By delegating profiling to an autonomous subagent, the assistant could obtain a comprehensive, multi-faceted analysis without cluttering the parent conversation with intermediate commands and outputs. This separation of concerns — parent for strategy, subagent for tactical investigation — is a recurring theme in effective AI-assisted engineering.

Perhaps most importantly, this message demonstrates that understanding a problem is often more valuable than immediately trying to solve it. The 50.5ms decode time was not just a number to be reduced; it was a story about PCIe bandwidth, NCCL protocols, and the fundamental physics of distributed inference. By reading that story, the assistant knew exactly where to apply leverage — and where the laws of hardware would resist further improvement.