Chunk 6.0
In this chunk, the assistant significantly improved the inference throughput for GLM-5-NVFP4 on 8x RTX PRO 6000 GPUs by addressing several key bottlenecks. The primary fix was enabling FlashInfer CUTLASS MoE autotune for SM120 by patching `model_runner.py` to include `flashinfer_cutlass` in the autotune list, and raising `--max-running-requests` from 64 to 1024. Combined with `--disable-cuda-graph` and `--disable-radix-cache`, total token throughput jumped from ~880 tok/s to ~1,950 at 256 concurrency, ~2,800 at 512, and ~3,740 at 1024 concurrency, with peak output reaching nearly 4,000 tok/s. However, GPU power draw remained around 250W out of a 600W TDP, indicating the hardware was still underutilized. The assistant then investigated the root cause of the low power and identified that FlashInfer's allreduce fusion was disabled on SM120 because the underlying TRT-LLM communication kernels only supported SM90/SM100. Attempts to patch the flashinfer kernel to add SM120 support (by modifying the architecture gate and version check) allowed the server to start, but the fusion performed poorly, dropping throughput to 236 tok/s and power to 125W, suggesting synchronization issues on SM120. The assistant reverted those changes and instead explored NCCL tuning and `--num-continuous-decode-steps 4`, neither of which yielded meaningful gains. The chunk concludes with the assistant trying the `flashinfer_trtllm` MoE backend as an alternative path to better GPU utilization. The overarching theme is the challenge of adapting a large MoE inference stack designed for datacenter Blackwell (SM100) to consumer Blackwell (SM120) hardware, which has different constraints like smaller shared memory and missing kernel support for advanced features like allreduce fusion. The assistant's approach of forking and patching both sglang and flashinfer code demonstrates a willingness to modify upstream dependencies to unlock performance, but the allreduce fusion attempt highlights the risk of using architecture-specific synchronization primitives without thorough validation. The next steps will likely involve either finding a working allreduce fusion approach for SM120 or accepting the PCIe bottleneck and focusing on other optimizations like TP4+PP2 or deeper MoE kernel tuning.
Message Articles
- The State of the Union: A Critical Inflection Point in Blackwell Inference Optimization
- The Proceed Signal: How a Three-Line Message Became the Turning Point in a High-Performance ML Deployment
- The Strategic Pivot: Planning the Next Phase of GLM-5 Inference Optimization
- The Idle Giants: A Diagnostic Pivot in the GLM-5 Inference Optimization Journey
- The Pivot Point: From Deployment to Optimization in Large-Scale MoE Inference
- The Pivot Point: Diagnostics, Benchmarking, and Knowledge Gathering in Message 655
- The Plateau at 912 Tok/s: A Diagnostic Benchmarking Pivot in GLM-5 Inference Optimization
- The Critical Observation That Changed the Trajectory of GLM-5 Inference Tuning
- The Turning Point: Diagnosing GPU Underutilization and Initiating MoE Kernel Tuning for GLM-5 on Blackwell
- The Pivot Point: MoE Kernel Tuning Begins for GLM-5 on SM120 Blackwell
- The Missing Config: Diagnosing MoE Kernel Performance on SM120 Blackwell
- The Diagnostic Pivot: Tracing MoE Kernel Configurations in SGLang on SM120 Blackwell
- The Pivot: How a Missing Log Message Saved Hours of Misguided Optimization
- A Pivot to Prior Art: The User's Terse Directive to Leverage Existing MoE Tuning Artifacts
- The Pivot to Prior Art: How a Directory Listing Unlocked MoE Kernel Tuning for GLM-5 on Blackwell
- Reading the Artifacts of Prior Work: How a Directory Listing Shaped the Next Phase of MoE Kernel Tuning
- Mining Prior Art: How a Single Message Unlocked 4× Performance Through Knowledge Transfer
- Mining Prior Art: How the Assistant Leveraged Kimi K2 Research to Inform GLM-5 MoE Tuning
- Mining Prior Art: How the Assistant Used Historical Benchmarks to Diagnose MoE Kernel Bottlenecks
- The Comparative Epiphany: Mining Prior Success to Unlock GLM-5 Inference Performance
- The Turning Point: Diagnosing Concurrency as the Hidden Bottleneck in GLM-5-NVFP4 Inference
- The Fork-and-Patch Moment: Enabling FlashInfer CUTLASS MoE Autotune on SM120
- Reading the Code Before the Patch: A Critical Information-Gathering Step in MoE Kernel Tuning
- Reading the Autotune Code: A Pivotal Moment of Investigation Before a Risky Patch
- The Autotune Gamble: Patching SGLang to Unlock FlashInfer CUTLASS MoE on SM120
- The Moment of Discovery: Verifying a Failed Patch in the FlashInfer Autotune Pipeline
- The Sed That Failed: Debugging Multiline Patterns in the Quest for SM120 MoE Autotune
- The Critical Uncomment: Enabling FlashInfer CUTLASS MoE Autotune for Blackwell SM120
- The Verification That Unlocked 4× Throughput: A Single sed Command in the FlashInfer Autotune Pipeline
- The Patch That Unlocked 4x Throughput: Enabling FlashInfer CUTLASS MoE Autotune on SM120 Blackwell
- The Kill Signal: A Pivotal Transition in ML Inference Optimization
- The Verification Before the Breakthrough: A Quiet Moment in the GLM-5 Optimization Journey
- The Pivot: How a Single Server Restart Unlocked 4x Throughput on Blackwell GPUs
- The Moment of Truth: Checking the Server Log After a Pivotal Optimization Restart
- The Waiting Game: Monitoring FlashInfer Autotune After Patching SGLang's MoE Backend
- The Waiting Game: Watching a Patched Inference Server Load on 8 Blackwell GPUs
- The Diagnostic Pause: Reading Between the Lines of a Server Startup
- The Diagnostic Grep: Reading Server Logs to Validate an Inference Optimization Hypothesis
- The Art of Waiting: Monitoring Server Initialization in a Distributed ML Inference Stack
- The Art of Watching Paint Dry: Why a Simple Status Check Reveals the Soul of ML Engineering
- The Pivot Point: When FlashInfer Autotune Finally Fires on 8 Blackwell GPUs
- The Autotune Awakens: Enabling FlashInfer CUTLASS MoE Kernel Tuning on SM120 Blackwell
- The Turning Point: How One Server Restart Unlocked 4x Throughput on Blackwell GPUs
- The Moment of Truth: Benchmarking a Patched Inference Stack
- The Validation Point: How a 689 tok/s Disappointment Became a 1,950 tok/s Breakthrough
- The Breakthrough and the Crash: A Moment of Triumph and Setback in Blackwell Inference Optimization
- The Crash That Wasn't OOM: Debugging a PrefillMetadata AttributeError in SGLang
- The Autotuner's Secret: What a Skipped Tactic Revealed About Blackwell GPU Inference
- The Diagnostic Pivot: Distinguishing Resource Exhaustion from Software Bugs in High-Concurrency ML Inference
- The Quiet Verification: Why a Single Bash Command Can Speak Volumes in ML Infrastructure Debugging
- The Pivot: Balancing Throughput and Stability in GLM-5-NVFP4 Inference
- The 165-Second Wait: How a Simple Server-Readiness Check Unlocked 3,740 tok/s for GLM-5-NVFP4
- Benchmarking After the Autotune: Validating GLM-5-NVFP4 Performance on 8× Blackwell GPUs
- The Moment of Truth: Benchmarking GLM-5-NVFP4 at Scale
- The Moment the Server Fell Silent: A Diagnostic Pivot in High-Concurrency LLM Inference
- The Page Table That Wasn't There: Debugging a PrefillMetadata AttributeError in SGLang's NSA Backend
- Debugging the NSA Attention Bug: Chasing a Missing Attribute in SGLang's FlashInfer Path
- The $sed Detective: Tracing a GPU Inference Crash Through Source Code Archaeology
- Tracing a Ghost Attribute: Debugging the `page_table_1_flattened` Crash in NSA Attention
- Debugging the NSA Attention Crash: A Case Study in Inference Engine Diagnostics
- The Art of the Workaround: Disabling Radix Cache to Circumvent an NSA Attention Bug
- The 75-Second Wait: A Pivot Point in ML Infrastructure Debugging
- The Benchmark After the Workaround: Validating Stability at 1,846 tok/s
- The 512 Concurrency Benchmark: A Pivotal Data Point in Scaling GLM-5-NVFP4 on Blackwell
- The 3,740 tok/s Milestone: Pushing GLM-5-NVFP4 Inference to Its Limit on 8 Blackwell GPUs
- Breaking Through the Throughput Ceiling: 3,740 tok/s on GLM-5-NVFP4 with 8 Blackwell GPUs
- The 87-Watt Revelation: When 3,740 tok/s Masks a Deeper Bottleneck
- The Power Draw Revelation: Diagnosing GPU Underutilization in Blackwell MoE Inference
- The Moment of Validation: GPU Utilization Confirmed at 100%
- The Victory Lap: Synthesizing Performance Gains in a 744B-Parameter MoE Inference Stack
- The Clarifying Question That Reshaped a Benchmark Narrative
- The Clarification That Changed Everything: Total vs. Output Token Throughput in the GLM-5-NVFP4 Benchmark
- The Clarification That Changed Everything: Deconstructing a Benchmark Metric Correction in ML Inference Optimization
- The SM120 Wake-Up Call: When GPU Utilization Lies and Hardware Nuance Matters
- The 600W Misunderstanding: Correcting Assumptions About GPU Power in a Blackwell Inference Stack
- The 600W Wake-Up Call: Correcting Hardware Assumptions in the Quest for Blackwell Inference Performance
- The 600W Wake-Up Call: How a Power Correction Reshaped an ML Inference Debugging Session
- The 600W Revelation: How a Simple Power Query Reshaped an Inference Optimization Campaign
- The 600W Discovery: A Pivotal Moment in GPU Inference Debugging
- The Turning Point: How a Single Task Call Uncovered the GPU Utilization Mystery in GLM-5-NVFP4 Inference
- The Turning Point: Diagnosing GPU Underutilization in GLM-5-NVFP4 Inference on SM120 Blackwell
- Probing the Code: How Three Bash Commands Uncovered SM120's Missing Allreduce Fusion
- The Missing Architecture Gate: How a Single Line of Code Was Starving Eight GPUs
- Reading the Code: Diagnosing SM120 Allreduce Fusion Gaps in SGLang
- The Smoking Gun: Tracing a 58% GPU Utilization Gap to a Missing Architecture Check
- The Missing File: A Debugging Detour in the Quest for SM120 Allreduce Fusion
- The Hunt for `is_sm120_supported`: A Single Bash Command That Reveals the Architecture Gap in SGLang
- The Grep That Unlocked Blackwell: Tracing SM120 Support Through SGLang's Codebase
- The Missing Module: A Diagnostic Failure That Reveals the Architecture of Debugging
- A Single Line of Confirmation: Verifying SM120 Support in the SGLang Inference Stack
- Patching the Allreduce Fusion Gap: Enabling SM120 Support in SGLang's Communicator
- The Sed Command That Unlocked 42% Idle GPU Power: Patching Allreduce Fusion for SM120 Blackwell
- The Sed Command That Unlocked (and Broke) Blackwell Inference
- The Verification That Unlocks Blackwell RTX: Patching SGLang's Allreduce Fusion for SM120
- The Anatomy of a Single-Line Fix: Enabling SM120 Allreduce Fusion in SGLang
- The Sed Command That Unlocked 42% Idle GPU Power: Patching SGLang's SM120 Allreduce Fusion Gate
- Patching the Architecture Gate: Adding SM120 Support to SGLang's MoE Runner Backend
- The Sed Command That Unlocked Blackwell: Enabling SM120 Support in SGLang's MoE Runner
- The Verification That Unlocks Blackwell Consumer GPUs: A Deep Dive Into Patching SGLang for SM120
- Patching the Gates: Enabling FlashInfer Allreduce Fusion for Blackwell SM120 GPUs
- The Turning Point: Enabling SM120 Allreduce Fusion in SGLang
- The Critical Restart: Enabling Allreduce Fusion for Blackwell GPUs in SGLang
- The Polling Watchdog: Validating Kernel-Level Patches for Blackwell SM120 Inference
- The Weight of a Single Word: How "crashed" Forced a Fundamental Reckoning with Hardware Compatibility
- When Patching Meets Reality: The SM120 Allreduce Fusion Crash