Chunk 16.0

This chunk covers the final debugging, performance optimization, and productionalization of the GLM-5 GGUF model on vLLM with 8x Blackwell GPUs. The primary task was diagnosing why the model produced incoherent text despite loading successfully. After systematically ruling out weight loading, tensor parallelism sharding, and configuration issues, the root cause was isolated to two bugs in vLLM's Triton MLA attention backend: an output buffer disconnect caused by a custom PyTorch op creating a phantom tensor, and a shard ordering bug in the GGUF dequantization layer for fused projections. Fixing these bugs restored correct model output. With correct output achieved, the focus shifted to performance optimization. Single-request decode was initially ~20 tok/s, with profiling revealing that ~42% of the time was spent on NCCL allreduce calls over PCIe. Enabling CUDAGraph doubled throughput to ~43 tok/s by batching kernel launches, and tuning `NCCL_PROTO=LL` further boosted it to ~57 tok/s. More advanced optimizations like custom allreduce and allreduce-RMS fusion were explored but found to be incompatible with the PCIe-only hardware topology, establishing a clear hardware bottleneck. The final task was productionalizing the optimized configuration into a systemd service (`vllm-glm5.service`). While the service was created and deployed, initial startup failed due to a conflict with a stale vLLM process from a previous session. The overarching themes of this chunk are deep, methodical debugging of a complex inference stack (vLLM, GGUF, Triton MLA), iterative performance optimization constrained by hardware topology (PCIe vs NVLink), and the successful adaptation of cutting-edge open-source software for a novel model (GLM-5) on novel hardware (Blackwell GPUs).

From Garbage to Production: Debugging, Optimizing, and Deploying GLM-5 on 8× Blackwell GPUs 2693 words

Message Articles

Subagent Sessions