Chunk 14.0

In this chunk, the assistant focused on deploying the final GGUF patches and debugging the `vllm serve` launch. The `gguf_loader.py` patch (with the corrected 3D kv_b reassembly logic) and the `weight_utils.py` patch (force-dequant for kv_b sentinel tensors) were successfully SCP'd to the container. A quick test validated the weight name mapping, revealing 27 unmapped tensors from the MTP/nextn layer (blk.78), which were confirmed to be safely skipped by the model's `load_weights` method. The initial launch attempts failed due to three distinct issues: the `maybe_override_with_speculators` function crashing on the unsupported `glm-dsa` architecture (fixed by patching the config), a `torch.bfloat16` dtype incompatibility with GGUF quantization (fixed by adding `--dtype float16`), and most critically, the lack of a valid attention backend for the Blackwell SM120 GPU. The primary blocker was the absence of an attention backend supporting the combination of SM120 compute capability, sparse MLA (DSA indexer), and `qk_nope_head_dim=192`. After confirming that the latest vLLM nightly lacked this support, the assistant and user chose to patch the existing Triton MLA backend to handle sparse attention. The assistant analyzed the sparse MLA architecture (SparseMLAAttentionImpl, FlashMLASparseBackend) and the Triton decode kernel, designing a `TritonMLASparseBackend` that reuses the existing kernel by treating the physical sparse indices as a virtual block table. This new backend was implemented, registered in the attention registry and CUDA priority list, and deployed to the container. With the new backend selected, a subsequent launch failed due to a `KeyError` in `weight_utils.py`. The root cause was a global string replacement (`name.replace("weight", "qweight")`) that corrupted parameter names containing "weight" as a substring (e.g., `weights_proj` became `qweight_types_proj`). The assistant fixed this by changing the logic to only replace the `.weight` suffix. After deploying this fix, the model loading progressed past the attention backend selection, successfully selecting `TRITON_MLA_SPARSE` and beginning to load the 402GB model onto the GPUs. The session concluded with the model loading process underway, marking a significant milestone in getting the GLM-5 GGUF model running on the Blackwell hardware.

The Gauntlet: Deploying GLM-5 GGUF on Blackwell GPUs Through a Cascade of Breakthroughs 2333 words

Message Articles

Subagent Sessions