The 69% Bottleneck: How a Torch Profiler Trace Reshaped the GLM-5 Deployment Strategy

Introduction

In the high-stakes world of large language model inference optimization, the difference between weeks of fruitless tuning and a decisive breakthrough often comes down to a single question: where is the time actually going? This article chronicles a pivotal sequence in an intensive optimization campaign for the GLM-5 model — a 744-billion-parameter Mixture-of-Experts (MoE) architecture — running on eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The sequence spans the moment when a static gap analysis reached its limits, through a failed attempt to profile with NVIDIA Nsight Systems, to a successful torch profiler trace that revealed the smoking gun: 69% of decode time was consumed by a single aten::copy_ operation casting the KV cache from FP8 to BF16 on every layer, every step.

This discovery fundamentally reshaped the optimization strategy, leading first to a gather-then-cast patch that achieved a 29% throughput improvement, and ultimately to the user's decision to abandon the NVFP4 quantization path entirely in favor of unsloth's GGUF quantization and vLLM deployment. The story of how this bottleneck was found — and how the assistant navigated the treacherous waters of profiling a complex multi-GPU inference server — is a masterclass in systematic performance debugging.

The 86ms Gap: Setting the Stage

The optimization campaign had been running for days across multiple sessions. The GLM-5-NVFP4 model, with its 744 billion parameters spread across 256 experts (8 activated per token), was deployed using SGLang with tensor parallelism across 8 GPUs (TP8). Despite extensive tuning — kernel upgrades, CPU governor optimization, PCIe configuration, NCCL tuning, FlashInfer MoE autotune, and experiments with expert parallelism, CUDA graphs, and MSCCLPP allreduce — single-stream decode throughput remained stubbornly stuck at approximately 10.5 tokens per second, corresponding to a time-per-output-token (TPOT) of roughly 95 milliseconds.

Theoretical analysis had established a stark baseline: a simulated decode step using BF16 GEMMs and NCCL AllReduce completed in just 8.9 milliseconds. The difference — 86 milliseconds — was a vast, unexplained gap representing 3.4% efficiency against the theoretical maximum of 309 tok/s.

The assistant had already built and deployed a custom gap analysis script (decode_gap_analysis.py) to decompose this gap into measurable components. The results were illuminating but incomplete. The script measured MoE routing overhead (2.4ms total), token permutation (1.6ms), RMSNorm (3.4ms), CPU dispatch overhead (5.3ms), and FP4 GEMMs (2.3ms). Combined with AllReduce (6.6ms), these accounted for only ~22 milliseconds of the 95ms TPOT. That left ~73 milliseconds completely unexplained — nearly 77% of the total decode time was a black box.

The static analysis had reached its limit. The assistant needed to observe the actual inference path under real conditions, capturing every CUDA kernel launch, every memory operation, every synchronization event. The hunt for the bottleneck was about to enter a new phase.

The Failed Nsight Systems Campaign

The assistant's first instinct was correct: use NVIDIA Nsight Systems (nsys), the industry-standard GPU profiling tool, to capture a full trace of a single inference request. The plan was methodical and leveraged existing infrastructure. SGLang had built-in NVTX (NVIDIA Tools Extension) layer-wise markers accessible via the --enable-layerwise-nvtx-marker flag. The assistant verified that nsys was available (version 2024.6.2), confirmed the flag's existence in the SGLang source code, and designed a careful workflow: launch the server under nsys with --capture-range=none for delayed capture, then trigger profiling during a single inference request.

The assistant wrote a launch script (nsys_profile_server.sh) and a profiling orchestration script (profile_decode.py), uploaded them to the remote server, and launched the server under nsys supervision. The model began loading — a 405 GB behemoth spread across 83 safetensor checkpoint shards — and eventually reached a state where it could handle prefills. But something went wrong. The detokenizer heartbeat began failing. The health endpoint returned HTTP 503 errors. The scheduler processes became defunct zombies.

The assistant's diagnosis was perceptive: "The nsys overhead + --enable-layerwise-nvtx-marker seems to have made the scheduler/detokenizer hang." This was a classic profiling pitfall. Nsight Systems, when configured to trace every CUDA API call and NVTX event across eight GPU worker processes, introduced enough instrumentation overhead to break the server's timing-sensitive scheduling loop. The server was not merely slow — it was functionally broken under the profiler.

The assistant killed the zombie processes, verified that all eight GPUs were clear (each reporting the baseline 3 MiB memory usage), and regrouped. The nsys approach had failed, but it had taught a valuable lesson: for a complex, multi-process, latency-sensitive serving system, external profiling tools can perturb the system enough to invalidate the measurement.

The Pivot to Torch Profiler

After the nsys failure, the assistant made a critical strategic decision. Instead of wrapping the entire server in an external profiler, it would inject torch.profiler instrumentation directly into SGLang's model runner — modifying the source code of the inference engine itself to capture a targeted profile of the forward pass. This was a fundamentally different philosophy: rather than observing the system from the outside, the assistant would instrument it from within.

The reasoning was sound. Torch profiler is a first-party PyTorch tool with deep integration into CUDA kernel tracing. It can capture kernel names, durations, input shapes, and FLOP counts with minimal overhead when configured correctly. By instrumenting only the forward_decode method of the model runner, the assistant could capture precisely the operation of interest — a single decode step — without the noise of HTTP server overhead, scheduler logic, or detokenization. The instrumentation could be triggered conditionally via an environment variable (SGLANG_PROFILE_DECODE=1), making it easy to enable and disable without permanent code changes.

The patch itself was a model of surgical precision. The assistant located the forward_decode method at line 2276 of /root/sglang/python/sglang/srt/model_executor/model_runner.py and replaced it with a version that included profiler state management. The profiler was configured to:

The Smoking Gun: 69% in a Single Operation

The moment of truth arrived when the assistant fetched the profiler summary and performed a deep analysis. The results were stunning:

| Component | Per-step Time | % of TPOT | |---|---|---| | aten::copy_ (dtype cast) | 64.6 ms | 69.3% | | NCCL AllReduce | 7.7 ms | 8.2% | | aten::mm (GEMM) | 6.1 ms | 6.6% | | CUTLASS GemmUniversal | 3.0 ms | 3.3% | | cuBLAS gemvx | 2.8 ms | 3.0% | | Torch-Compiled (MoE routing) | 3.7 ms | 4.0% | | FlashInfer MLA decode | 0.8 ms | 0.9% | | FusedAddRMSNorm | 0.5 ms | 0.5% |

The aten::copy_ operation, backed by the unrolled_elementwise_kernel, consumed 64.6 milliseconds per decode step — more than eight times the next largest contributor. The assistant performed a critical sanity check: total CUDA time of 2.797 seconds across 30 steps equals 93.2 ms/step, which matched the independently measured 95ms TPOT almost perfectly. The profiler had captured the right thing.

The pattern was unmistakable: 2,340 calls across 30 steps = 78 calls per step. GLM-5 has 78 layers. Something was performing a dtype conversion once per layer, every decode step.

Tracing the Bottleneck: From Hypothesis to Discovery

The assistant's first hypothesis was that the aten::copy_ was FP4 weight dequantization — the FP4 weights being cast to BF16 on every forward pass. This was a reasonable assumption given the model's exotic quantization format. The assistant searched the FP4 quantization module (modelopt_quant.py) for .to(), copy_, and dtype conversion calls, expecting to find the source.

But the grep revealed something unexpected: the FP4 GEMM kernel (cutlass_fp4_gemm) accepted an out_dtype parameter and produced BF16 output directly, without an explicit copy step. The FP4 weights were being dequantized on-the-fly within the tensor core operations, not through a separate aten::copy_ call. The FP4 hypothesis was wrong.

The real culprit was discovered through deeper analysis of the Chrome trace and code inspection. The aten::copy_ was coming from the FlashInfer MLA attention backend — specifically, the KV cache being cast from FP8 to BF16 on every layer for the entire 495,552-token pool. The code at flashinfer_mla_backend.py:639-640 was executing:

k_buffer = forward_batch.token_to_kv_pool.get_key_buffer(layer.layer_id).to(q.dtype)

This cast the entire KV cache pool — 285.6 million elements per layer, approximately 857 MB of data — from FP8 to BF16 before the attention computation. With 78 layers, that was 66.8 GB of data movement per decode step, consuming 64.6 milliseconds of GPU time. The KV cache was stored in FP8 to save memory, but the FlashInfer attention kernel required BF16 inputs, and the conversion was being done naively — casting the entire pool rather than just the active KV entries.

The Gather-Then-Cast Patch: A 29% Improvement

Once the KV cache cast was identified as the bottleneck, the assistant designed and implemented a targeted fix. The core insight was simple: the existing code was casting the entire KV cache pool (495,552 tokens) from FP8 to BF16 on every layer, every decode step. But the attention computation only needed the KV entries for the active tokens in the current batch — a tiny fraction of the total pool.

The gather-then-cast patch changed the logic to:

  1. Gather only the KV entries corresponding to active token positions from the FP8 pool
  2. Cast only those gathered entries from FP8 to BF16
  3. Pass the smaller, cast buffer to the attention kernel This reduced the data movement from ~857 MB per layer (the entire pool) to approximately the number of active tokens × KV head dimension — a reduction of several orders of magnitude for single-stream inference. The patch modified four code paths in flashinfer_mla_backend.py: call_begin_forward (to save indices and plan with sequential indices), forward_decode (to perform the gather-then-cast), forward_extend paged path (same treatment for prefill), and the indices updater initialization. The deployment via scp and remote Python execution succeeded with seven "OK" confirmations. The validation came in two stages. First, a health check confirmed the server started successfully — "READY at attempt 12" after approximately 60 seconds. Then, the benchmark results arrived: 13.5–13.6 tok/s with TPOT of approximately 74ms, compared to the baseline of ~10.5 tok/s (95.6ms TPOT) — a 29% improvement. However, the assistant immediately noted that the theoretical savings from eliminating the 64.6ms cast was not fully realized. The actual improvement was only about 21ms, roughly one-third of the theoretical maximum. The assistant identified three hypotheses for this gap: the gather operation itself takes time (random gather of FP8 data), the cast of the gathered subset still has nonzero cost, and the torch.arange() call per step adds overhead. This intellectual honesty — acknowledging that the fix didn't fully deliver on its promise — is characteristic of rigorous engineering.

Alternative Attention Backends: The Dead Ends

Before settling on the gather-then-cast approach, the assistant systematically evaluated alternative attention backends that might natively support FP8 KV cache, eliminating the cast entirely. Two candidates were tested:

trtllm_mla: This backend, based on TensorRT-LLM's MLA implementation, crashed immediately because it required qk_nope_head_dim == 128 while GLM-5 uses 192. The dimension mismatch was fundamental — GLM-5's MLA architecture uses a different head dimension configuration than what the TensorRT-LLM backend expected, and there was no configuration parameter to override this.

cutlass_mla: This backend, a CUTLASS-based MLA implementation, crashed with an assertion error about page size. The error suggested that the backend expected a specific KV cache page size that was incompatible with GLM-5's configuration.

Both failures reinforced that the assistant was locked into the FlashInfer MLA backend — the only backend compatible with GLM-5's specific architectural parameters. This meant the KV cache cast was not a bug to be fixed by switching backends but an inherent property of the model-hardware-software stack.

The Strategic Pivot: Abandoning NVFP4

Despite the 29% improvement, the fundamental architectural limitation remained. The NVFP4 quantization format, combined with the FlashInfer MLA backend's lack of native FP8 KV cache support on SM120 (Blackwell) GPUs, imposed a tax that could not be fully amortized. The user made the decisive call: "Maybe let's just consider abandoning this quant." The user pointed to unsloth's GLM-5 GGUF quantizations on Hugging Face, listing eight 4-bit variants ranging from 403 GB (IQ4_XS) to 473 GB (Q4_1), and asked for the one with the best precision for coding and agent tasks that fit the system constraints.

The assistant's response was a masterclass in concise strategic communication: "Good call" followed by a precise articulation of why the NVFP4 path had failed — "the NVFP4 quant's flashinfer MLA backend has a fundamental FP8 KV cache casting bottleneck that wastes 69% of decode time, and none of the alternative backends are compatible with GLM-5's architecture." The assistant then laid out a structured todo list: research GGUF options, kill the SGLang server, remove the old NVFP4 model, and evaluate serving engines.

The evaluation of GGUF options compared seven 4-bit formats across quality, size, and suitability for coding and agent tasks. The user selected UD-Q4_K_XL (431 GB) — Unsloth Dynamic 2.0 quantization, which claims superior accuracy by keeping important layers at higher precision. This choice represented a balance between quality (the Unsloth Dynamic 2.0 improvements) and footprint (smaller than Q4_K_M at 456 GB).

The Serving Engine Correction

The assistant's initial recommendation for serving the GGUF model was llama.cpp — the most mature GGUF runtime with native multi-GPU tensor-split support. But the user rejected this categorically: "Consider vllm/tensorrt, llama.cpp no bc it's not an inference engine."

This correction was a pivotal moment. The user drew a sharp distinction between an inference runtime (llama.cpp) and a production serving engine (vLLM, TensorRT-LLM). The former can load and run models, but the latter provides continuous batching, paged attention, concurrent request handling, scheduling, and production-grade API endpoints — all essential for serving coding agents at scale.

The assistant accepted the correction immediately, articulating why the user was right: "Good point — llama.cpp is an inference runtime, not a proper serving engine with continuous batching, paged attention, and concurrent request handling at scale." The assistant then launched web searches to investigate vLLM's GGUF support for GLM-5, specifically targeting MoE architecture and tensor parallelism.

This moment reveals a critical dynamic in human-AI collaboration: the assistant excels at low-level technical optimization — profiling kernels, patching attention backends, computing memory budgets — but can miss higher-level architectural and operational concerns that a human practitioner recognizes instinctively. The user's correction was not about a specific technical detail but about the category of tool appropriate for the deployment context.

The 10-Shard Problem

The vLLM + GGUF path then hit a concrete obstacle. The assistant discovered that the unsloth GGUF was split into 10 shard files (00001-of-00010 through 00010-of-00010). vLLM's GGUF documentation explicitly states that it "only supports loading single-file GGUF models." This was a direct conflict between the model quantizer's output format and the inference engine's input requirements.

The assistant identified the solution: use gguf-split --merge to combine the shards into a single monolithic file. But this introduced new risks: the merge operation requires temporary double-storage (the 10 shards plus the merged file must coexist), and vLLM's experimental GGUF loader might not handle a 431 GB MoE model correctly even after merging. The assistant launched parallel web searches to check if vLLM had updated its multi-file GGUF support or if SGLang had added GGUF support as a contingency.

The storage arithmetic was precise. After deleting the NVFP4 model (405 GB), the system would have approximately 1,241 GB free on /shared — enough to accommodate the 431 GB download plus the 431 GB merged file during the merge operation, with 379 GB of headroom.

Clearing the Decks

The cleanup phase was executed decisively. The assistant killed the SGLang server and deleted the NVFP4 model files with a single rm -rf command, freeing 1.2 TB on the shared volume. This was a moment of closure — weeks of optimization work, the gather-then-cast patch, the alternative backend experiments, the server tuning — all erased with a single command. But it was also a moment of forward momentum: the assistant confirmed the feasibility of the merge workflow and positioned the system for the next phase: downloading the GGUF, merging the shards, and deploying with vLLM.

The message is a transitional artifact, bridging two major phases of the project. It closes the NVFP4 chapter with decisive action and opens the GGUF chapter with a clear plan. The assistant's careful reasoning about storage constraints, toolchain compatibility, and workflow sequencing demonstrates a methodical approach to infrastructure engineering that is essential when operating at the scale of 744-billion-parameter models across eight GPUs.

Lessons in Diagnostic Methodology

This segment exemplifies several critical principles of performance debugging at scale:

1. Static analysis has limits. The gap analysis script measured individual components in isolation but could not capture the KV cache cast because it only manifests during actual inference with a populated KV cache. The ~73ms unexplained remainder was a clear signal that a different methodology was needed.

2. Profiling tools can perturb the system. The nsys approach failed because its overhead broke the server's timing-sensitive scheduling. The observer effect is real in performance engineering — the act of measuring can change the behavior being measured.

3. Surgical instrumentation beats system-wide tracing. The torch profiler approach succeeded because it was scoped to exactly the function of interest, gated behind an environment variable, and configured with appropriate warmup and capture windows. It captured precisely the data needed without destabilizing the server.

4. The most obvious suspect is not always the culprit. The FP4 dequantization hypothesis was elegant and satisfying — it explained the 78 calls per step, the massive data movement, and the per-layer pattern. But it was wrong. The real bottleneck was a mundane dtype conversion in the attention backend, hiding in plain sight.

5. Profiler data must be cross-validated. The assistant verified that the profiler's per-step time (93.2ms) matched the independently measured TPOT (95ms), confirming the profiler was not introducing significant overhead or missing major operations.

6. Architectural bottlenecks cannot be patched away. The gather-then-cast patch achieved a respectable 29% improvement, but it could not eliminate the fundamental architectural limitation: the FlashInfer MLA backend on SM120 required BF16 KV cache, and the NVFP4 format stored it in FP8. This mismatch was not a bug to be fixed but a property of the hardware-software stack. The user's decision to abandon NVFP4 was a recognition that some bottlenecks are architectural, not tunable.

7. The serving engine is as important as the model format. The user's rejection of llama.cpp in favor of vLLM or TensorRT-LLM highlighted that production serving requires more than just model compatibility. Continuous batching, paged attention, concurrent request handling, and production-grade APIs are essential for real-world deployment. The assistant's initial assumption that GGUF implied llama.cpp was a natural but incorrect coupling.

8. Storage planning is a first-class engineering concern. The discovery of the 10-shard problem and the subsequent storage arithmetic (1,241 GB available, 862 GB peak requirement during merge) demonstrates that deploying large models requires careful resource planning beyond just GPU memory. Disk space, temporary storage, and toolchain compatibility are all critical.

The Broader Significance: A Template for Performance Debugging

The diagnostic sequence chronicled in this segment — from static gap analysis through failed profiling attempt to successful instrumentation and bottleneck discovery — provides a template for systematic performance debugging that applies well beyond this specific project.

The methodology is reproducible. When faced with a performance gap that resists explanation:

  1. Start with static analysis to rule out obvious suspects and establish a baseline
  2. Recognize when static analysis has reached its limit — the ~73ms unexplained remainder was the signal to escalate
  3. Attempt the most powerful profiling tool first (nsys), but be prepared for it to fail
  4. Pivot to surgical instrumentation when external tools prove too disruptive
  5. Cross-validate profiler data against independently measured metrics
  6. Form and test hypotheses systematically, using code search and subagent delegation to accelerate the process
  7. Be willing to abandon incorrect hypotheses — the FP4 dequantization theory was elegant but wrong, and recognizing this quickly saved hours of wasted effort The tools matter, but methodology matters more. The torch profiler trace was the instrument that delivered the breakthrough, but it was the assistant's methodical approach — the willingness to fail, pivot, and iterate — that made the discovery possible. The nsys failure was not a setback; it was data that informed the next approach. The bottleneck was hiding in plain sight. The KV cache cast was not an exotic operation or a subtle interaction effect. It was a simple dtype conversion that happened to be catastrophically expensive because it operated on the entire token pool. It had been invisible to the gap analysis because that script measured individual operations in isolation, not the full inference path. Only a complete profiler trace of the actual running system could reveal it.

Conclusion

The diagnostic journey documented in this segment transformed the GLM-5-NVFP4 optimization campaign. The discovery that 69% of decode time was consumed by a single aten::copy_ operation — the KV cache being cast from FP8 to BF16 on every layer for the entire 495K-token pool — was the smoking gun that explained the 86ms decode gap. The gather-then-cast patch achieved a 29% improvement, but the fundamental architectural limitation led the user to abandon the NVFP4 quantization path entirely, pivoting to unsloth's GGUF quantization and vLLM deployment.

This sequence exemplifies the power of systematic, profiler-driven performance debugging. When optimization efforts stall, the answer is not to try more random changes — it is to measure precisely what is happening, at the right level of granularity, with the right tool for the job. The torch profiler trace did not just identify a bottleneck; it changed the entire trajectory of the project, saving weeks of effort that would have been spent optimizing the wrong components.

In the end, the most important lesson from this segment is that the bottleneck you expect is rarely the bottleneck you get. The FP4 GEMM kernels, the MoE routing, the NCCL communication — all were secondary. The real culprit was a mundane dtype conversion that had been hiding in plain sight, waiting for the right measurement tool to expose it. And when the measurement came, it did not just reveal a bug to fix — it revealed a fundamental architectural limitation that required a complete strategic pivot, from NVFP4 to GGUF, from SGLang to vLLM, from optimization within a framework to choosing a fundamentally different framework altogether.

References

[1] "The Diagnostic Pivot: How a Torch Profiler Trace Exposed the 69% KV Cache Cast Bottleneck in GLM-5 Inference" — Chunk article detailing the profiler trace, bottleneck discovery, and gather-then-cast patch

[2] "The Pivot: From KV Cache Cast Bottleneck to GGUF Deployment — A Synthesis of the GLM-5 Optimization Journey" — Chunk article covering the strategic pivot to GGUF, serving engine evaluation, and cleanup