From Bottleneck to Breakthrough: The Optimization Journey of GLM-5-NVFP4 on Blackwell GPUs

Introduction

In the high-stakes world of large language model (LLM) deployment, the path from a working system to a highly optimized one is rarely linear. It is a journey of hypothesis testing, kernel-level analysis, configuration tuning, and systematic documentation — a process that reveals as much about the hardware as it does about the model. This article examines a pivotal segment of an opencode coding session focused on optimizing the GLM-5-NVFP4 model on eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture). The segment captures a critical inflection point: the moment when the team confirmed the model was compute-bound, deeply analyzed why the GPUs were underutilized, and began systematically exploring optimization strategies.

The three message articles from this chunk — [1], [2], and [3] — each examine a different facet of a subagent session that read and summarized a FINDINGS.md document. But the story they tell extends far beyond a simple file-reading exercise. It is a story about how structured knowledge extraction, rigorous benchmarking, and creative problem-solving converged to unlock significant performance gains on novel hardware.

The Hardware: Blackwell's Promise and Constraints

The system under optimization was formidable on paper: eight NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96GB of VRAM, totaling 768GB of GPU memory. The architecture was SM120 — NVIDIA's Blackwell microarchitecture — representing the cutting edge of GPU compute. The system was equipped with ~1.5TB of system memory and a dual-socket CPU with two NUMA nodes.

But the hardware also came with significant constraints. The GPUs were connected via PCIe-only interconnect — no NVLink. GPU pairs were connected through PIX (direct PCIe switch), with cross-NUMA connections routed through the system fabric (SYS). This topology had profound implications for distributed inference: custom allreduce optimizations were disabled for configurations with more than two PCIe-only GPUs, meaning that inter-GPU communication would be a bottleneck under certain parallelism strategies.

The SM120 architecture itself introduced new challenges. The shared memory limit was 101KB (or 99KB depending on configuration), which was significantly less than what some kernel configurations required. Default MoE (Mixture of Experts) kernel configurations from SGLang exceeded this limit, causing failures. The CUTLASS kernels, which are critical for FP4 matrix multiplication, plateaued at ~1,300 TFLOPS (about 70% of dense peak) but only for very large matrices. During actual decode operations, where per-expert batch sizes ranged from 16 to 64 tokens, the kernels achieved merely 0.8 to 55 TFLOPS — a staggering 0.02% to 3% of peak theoretical performance.

Perhaps most striking was the power utilization data: during inference, the GPUs drew only ~235W out of their 600W TDP. The hardware was dramatically underutilized, and the optimization challenge was to close this gap.

The Compute-Bound Discovery: TP4+PP2 vs. TP8

The first major finding documented in this segment was the confirmation that the model was compute-bound rather than communication-bound. The team benchmarked two parallelism configurations:

The Kernel-Level Investigation: Why GPUs Were Starved

With the compute-bound hypothesis confirmed, the team dug deeper into why the GPUs were underperforming. The investigation, documented across multiple research agents and summarized in the chunk, revealed several layers of constraints:

Shared Memory Limitations: The SM120 architecture's 99KB shared memory limit prevented the use of larger CUTLASS tile configurations. Configurations like M128×N256 and M256×N128 failed to initialize because they required more shared memory than available. This forced the use of smaller tile sizes, which reduced computational efficiency.

Small Batch Decode: During actual inference, the per-expert batch sizes were tiny — 16 to 64 tokens. The CUTLASS kernels are designed for large matrices; at these small sizes, they achieved only 0.02% to 3% of peak theoretical performance. This was the root cause of the compute-bound behavior: the kernels were fundamentally mismatched to the workload.

Power Underutilization: The GPUs drew only 235W out of 600W TDP during inference. This suggested that the kernels were not keeping the compute units busy — there were frequent stalls waiting for data movement or synchronization.

These findings painted a clear picture: the optimization challenge was not about reducing communication overhead but about increasing the computational efficiency of the FP4 GEMM kernels at the batch sizes that actually occurred during inference.

Systematic Tuning: The 28% Breakthrough

Despite the kernel-level constraints, the team achieved significant gains through systematic server parameter tuning. By raising --max-running-requests to 2048 and setting --num-continuous-decode-steps to 8, the assistant achieved a 28% throughput improvement at 2048 concurrency, reaching 2,095 output tok/s and 4,151 total tok/s.

This improvement came from better utilization of the available memory bandwidth through increased batching. The PCIe-only topology required high concurrency (256+) to saturate memory bandwidth, and the tuning pushed the system to operate in this regime more effectively. The 250× throughput improvement from concurrency 1 (18 tok/s) to concurrency 512 (4,500 tok/s) demonstrated just how critical batching was for this hardware configuration.

The Research Frontier: Multiple Optimization Approaches

The segment documented a wide-ranging exploration of optimization strategies, each investigated by dedicated research agents:

  1. Expert Parallelism: Distributing experts across GPUs differently to improve load balancing and reduce communication.
  2. Piecewise CUDA Graphs: Breaking the computation graph into segments that could be optimized independently, potentially reducing kernel launch overhead.
  3. MSCCLPP Allreduce: Exploring whether Microsoft's Collective Communication Library could improve allreduce performance, even though the model was compute-bound.
  4. Single-Batch Overlap: Overlapping computation with communication for single-batch requests to reduce latency.
  5. L2 Cache Pinning: Pinning frequently accessed data in the L2 cache to reduce memory access latency.
  6. Persistent Grouped GEMM Kernels: Keeping kernels resident on the GPU between iterations to reduce launch overhead.
  7. FP4 Structured Sparsity: Exploiting sparsity patterns in the FP4 weights to reduce computation. Each of these approaches was documented in a separate glb5improvement-xx.md file, starting with piecewise CUDA graphs. The systematic documentation approach ensured that findings were captured and could be revisited as the optimization work progressed.

The Knowledge Extraction Pattern: Subagent as Analyst

The three message articles [1], [2], and [3] examine a subagent session that played a crucial role in this optimization workflow. The subagent was tasked with reading FINDINGS.md and extracting specific information about TP4+PP2 configuration results, MoE tuning parameters, batch overlap and allreduce optimization techniques, and the best-performing configuration found.

As [1] explores in depth, this seemingly simple request embodied a sophisticated understanding of the optimization landscape. The four search criteria were not random — they represented the key decision points in the optimization process. The TP4+PP2 results answered the fundamental question of parallelism strategy. The MoE tuning parameters addressed the immediate practical challenge of getting the model to run on novel hardware. The batch overlap and allreduce query reflected an exploration of whether communication optimizations could yield gains. And the best-performing configuration query sought the bottom-line answer that would guide deployment decisions.

Article [2] examines the subagent's response — a masterful structured summary that organized the findings into four clear sections with tables, code blocks, and actionable recommendations. The summary transformed a dense, multi-faceted research document into curated knowledge that could inform immediate decisions. The subagent demonstrated sophisticated thinking by acknowledging what was not in the document (the absence of batch overlap techniques) and by cross-referencing findings across sections (connecting the PCIe-only topology to the need for high concurrency).

Article [3] focuses on the reading step itself — the moment when the assistant read the file before analyzing it. This seemingly trivial step embodies the fundamental pattern of "read first, then reason" that underpins effective AI-assisted development. In the round-based architecture of the opencode session, the assistant could not both read and analyze in the same message; the reading step was a deliberate pause that ensured the subsequent analysis would be grounded in actual data rather than potentially stale memory.

The MoE Kernel Tuning Challenge

One of the most concrete outcomes of the optimization work was the resolution of MoE kernel configuration issues. SGLang had no pre-tuned MoE kernel configurations for the NVIDIA_RTX_PRO_6000_Blackwell_Server_Edition GPU. Without custom configurations, the system fell back to defaults that exceeded the SM120 architecture's 101KB shared memory limit.

The tuning procedure, documented in FINDINGS.md and extracted by the subagent, involved running tuning_fused_moe_triton.py with specific model and dtype parameters. The process tested 1,920 kernel configurations across batch sizes from 1 to 4,096 and took approximately 45 minutes. The key parameters established by PR #16975 were:

From Analysis to Documentation

The segment concluded with the assistant synthesizing the findings into a ranked optimization plan and beginning documentation as glb5improvement-xx.md files. This documentation step is often overlooked in accounts of optimization work, but it is critical for several reasons:

  1. Reproducibility: The documented configurations and procedures ensure that gains can be reproduced and built upon.
  2. Knowledge Preservation: In a fast-moving optimization campaign, findings can be lost or forgotten. Documentation creates a permanent record.
  3. Decision Tracking: The ranked optimization plan captures not just what was tried, but what was learned and what should be tried next.
  4. Collaboration: The documented findings can be shared with other team members working on related problems. The first file in the series, glb5improvement-01-piecewise-cuda-graphs.md, began the process of translating experimental findings into actionable improvement proposals.

Conclusion

The optimization journey captured in this segment of the opencode session is a testament to the depth and rigor required to deploy large language models on novel hardware. From the initial confirmation that the model was compute-bound to the kernel-level analysis of FP4 GEMM efficiency, from the 28% throughput improvement through parameter tuning to the systematic exploration of multiple optimization approaches, the work demonstrates the iterative, hypothesis-driven nature of ML inference optimization.

The three message articles [1], [2], and [3] provide a window into a specific moment in this journey — the subagent session that extracted and synthesized findings from FINDINGS.md. But the story they tell is universal: in the complex world of LLM deployment, the difference between a system that delivers 1,800 tokens per second and one that achieves 5,154 tokens per second often comes down to the quality of the questions asked, the rigor of the analysis performed, and the discipline of the documentation maintained.

The GLM-5-NVFP4 model on Blackwell GPUs remains a work in progress. The kernel-level constraints — small batch decode, shared memory limits, power underutilization — are fundamental architectural challenges that may require deeper changes to the serving stack. But the systematic approach documented in this segment provides a roadmap for addressing these challenges, one experiment at a time.