The Final Deployment: Patching vLLM's GGUF Loader for GLM-5 on Blackwell GPUs

"Good — the GGUF file is there (402GB), GPUs are idle with 0MB used. Both patched files look complete. Let me now deploy the final gguf_loader.py and verify the weight_utils.py is already deployed."

This brief message, accompanied by two scp commands, marks a critical inflection point in a months-long effort to run the 744-billion-parameter GLM-5 model on a cluster of eight NVIDIA RTX PRO 6000 Blackwell GPUs. On its surface, the message is mundane — a file copy operation, a verification step. But beneath that simplicity lies the culmination of an intricate debugging and reverse-engineering effort that spanned multiple days, uncovered a latent bug in vLLM's GGUF support affecting both DeepSeek V2/V3 and GLM-5 architectures, and required the assistant to reconstruct the precise tensor transformation pipeline used by llama.cpp's convert_hf_to_gguf.py to split MLA key-value projection weights during model conversion.

The Moment of Truth

The subject message (msg id=1669) appears at a specific juncture in the conversation where all the preparatory work has been completed and the assistant is about to test whether the patches actually work. The preceding message (msg id=1668) verified that the 402GB merged GGUF file exists on the container and that all eight GPUs are idle with zero memory utilization. The assistant had also just re-read both patched files locally to confirm they were complete. Everything was in place for the final deployment.

The message contains exactly two tool calls — both scp commands that copy the patched source files from the local development machine to the inference container:

scp /home/theuser/glm-kimi-sm120-rtx6000bw/gguf_loader.py.patched \
    root@10.1.230.174:/root/ml-env/lib/python3.12/site-packages/vllm/model_executor/model_loader/gguf_loader.py

scp /home/theuser/glm-kimi-sm120-rtx6000bw/weight_utils.py.patched \
    root@10.1.230.174:/root/ml-env/lib/python3.12/site-packages/vllm/model_executor/model_loader/weight_utils.py

These two files represent the entire patch surface needed to make vLLM's GGUF loading pipeline support the GLM-5 architecture. The gguf_loader.py patch handles the architecture-specific mapping of GGUF tensor names to HuggingFace parameter names, including the critical reassembly of the split key-value projection tensors (attn_k_b and attn_v_b) back into the single kv_b_proj weight that the model expects. The weight_utils.py patch modifies the quantized weight iterator to force-dequantize these split tensors (which are stored as Q8_0 in the GGUF file) and yield them as float32 values, bypassing the normal quantized weight pipeline.

Why This Message Was Written

The message exists because of a specific workflow reality: the assistant had been developing and testing patches on a local machine (a KVM VM with the username theuser), while the actual inference would run on a separate LXC container at 10.1.230.174. The development cycle involved editing files locally, then deploying them to the container via scp for testing. This particular message represents the final iteration of that cycle.

The context leading up to this moment reveals a complex debugging journey. The assistant had discovered that vLLM's GGUF loader had a fundamental bug: the kv_b_proj weight — a key component of the Multi-Head Latent Attention (MLA) mechanism used by both DeepSeek V2/V3 and GLM-5 — was never loaded from GGUF files. The root cause was a mismatch between how llama.cpp's conversion script stores the weight (split into separate attn_k_b and attn_v_b tensors) and how vLLM's GGUF loader expected to find it (as a single attn_kv_b tensor). This bug had been reported in the vLLM issue tracker (issue #30641) but remained unfixed in the nightly build.

The assistant's patch design was elegant: it used "sentinel suffixes" (__k_b and __v_b) appended to the HuggingFace parameter name in the GGUF-to-HF name map, which allowed the weight iterator to detect these special tensors and handle them differently. The _reassemble_kv_b() static method in gguf_loader.py would intercept these sentinel-suffixed tensors, undo the transpose that llama.cpp applied to k_b, concatenate k_b and v_b along the head dimension, and reshape the result to match the original kv_b_proj shape of [28672, 512].

The Thinking Process Visible in the Message

The assistant's reasoning is evident in the structure of the message. The opening line — "Good — the GGUF file is there (402GB), GPUs are idle with 0MB used. Both patched files look complete." — reveals a deliberate verification step. The assistant is checking three conditions before proceeding:

  1. The model file exists: The 402GB merged GGUF file is present at /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf. This is non-trivial — the file was assembled from 10 split files using a custom-built llama-gguf-split tool, and any corruption or missing data would cause the load to fail.
  2. The GPUs are available: All eight RTX PRO 6000 Blackwell GPUs show 0 MiB of memory used, meaning no previous process is occupying them. This is important because the previous workstream (NVFP4 via SGLang) had left processes running that needed to be killed.
  3. The patches are complete: The assistant had just read both patched files and confirmed they contain the full modified source. The weight_utils.py patch was already deployed from a previous iteration, but the gguf_loader.py patch had been edited locally after its last deployment to fix the 3D tensor reassembly logic (the initial version assumed n_head_kv=1, but the actual GGUF file used n_head_kv=64). The decision to deploy both files simultaneously — even though weight_utils.py was already on the container — shows a deliberate "fresh start" approach. By overwriting both files, the assistant ensures they are in sync and that no stale version of either file could cause mysterious errors. This is a wise engineering practice when dealing with interdependent patches.

Assumptions Made

Several assumptions underpin this message, some explicit and some implicit:

The GGUF file is structurally valid. The assistant assumes that the merged 402GB file is internally consistent — that all tensor shapes match the expected dimensions, that the quantization types are correct, and that the architecture identifier (glm-dsa) is properly set. This assumption was partially validated by earlier inspection (1809 tensors, correct architecture key), but the full structural validity can only be confirmed when vLLM attempts to load the model.

The patches are correct. The assistant assumes that the _reassemble_kv_b() logic correctly reverses the transformation applied by llama.cpp's convert_hf_to_gguf.py. This is a non-trivial assumption because the conversion script has multiple versions, and the unsloth GGUF file was produced with a different version than what the assistant analyzed. The assistant had verified that the GGUF file uses n_head_kv=64 (not the n_head_kv=1 override that newer versions of the conversion script apply), and had adjusted the reassembly logic accordingly. But there could be other version-dependent differences in how the tensors were stored.

The MTP/nextn layers can be safely ignored. The GGUF file contains tensors for layer 78+ (the "nextn" prediction layers), and the assistant's earlier test revealed 27 unmapped tensors from these layers. The assumption is that the model's load_weights() method will simply skip these unmapped tensors, which is the standard behavior for vLLM's GGUF loader. This assumption proved correct in subsequent testing, but it was not yet validated at the time of this message.

The Blackwell GPUs will support the required operations. The RTX PRO 6000 Blackwell GPUs use the SM120 compute architecture, which is relatively new. The assistant assumes that vLLM's CUDA kernels (including the attention backend) will work correctly on this hardware. This assumption would later prove problematic — the next chunk of the conversation reveals that the initial launch attempt failed because no valid attention backend existed for SM120 with sparse MLA, requiring the assistant to implement a new TritonMLASparseBackend.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

The vLLM GGUF loading pipeline. This is a multi-stage process: the _get_gguf_weights_map() method builds a mapping from GGUF tensor names to HuggingFace parameter names; get_gguf_weight_type_map() determines quantization types; gguf_quant_weights_iterator() in weight_utils.py yields (name, tensor) pairs in two passes (first qweight_type entries, then actual weights); and finally model.load_weights() maps the HuggingFace names to vLLM's internal parameter names. The patch modifies the first and third stages.

The MLA (Multi-Head Latent Attention) architecture. GLM-5 uses a variant of DeepSeek's MLA mechanism, where the key and value projections are computed through a low-rank bottleneck. The kv_b_proj weight is a combined projection matrix of shape [n_head_kv * (qk_nope_head_dim + v_head_dim), kv_lora_rank] — in this case [28672, 512]. During GGUF conversion, llama.cpp splits this into separate k_b and v_b components and transposes k_b.

The GGUF tensor storage convention. GGUF stores tensor shapes in reverse order relative to the data layout. A GGUF shape of [192, 512, 64] means the actual tensor in memory has shape [64, 512, 192]. This reversal must be accounted for when reading tensor metadata.

The unsloth quantization scheme. The UD-Q4_K_XL quantization uses a 4-bit block quantization with XL-sized blocks for higher accuracy. The expert weights are stored as 3D merged tensors (e.g., ffn_gate_exps.weight with shape [6144, 2048, 256]), which is a non-standard format that the patch must handle.

Output Knowledge Created

This message creates several important outputs:

A deployable patch set. The two scp commands create a consistent, deployed patch on the container. The gguf_loader.py at /root/ml-env/lib/python3.12/site-packages/vllm/model_executor/model_loader/gguf_loader.py now contains the GLM-DSA architecture support, the kv_b reassembly logic, and the expert weight mappings. The weight_utils.py at the corresponding path now contains the sentinel suffix detection and force-dequantization logic.

A verified deployment state. The message confirms that the GGUF file exists (402GB), the GPUs are idle (0MB used), and both patches are deployed. This state is the precondition for the next step: actually running vllm serve and debugging whatever errors emerge.

A record of the patch boundary. The message implicitly defines which files were modified and which were not. The deepseek_v2.py model file (which contains the GlmMoeDsaForCausalLM class) was not patched — it already had the necessary stub. The gguf.py quantization config was also not patched. This boundary definition is valuable for understanding what the patch covers and what it doesn't.

Mistakes and Incorrect Assumptions

The most significant assumption that proved incorrect was the belief that deploying these patches would be sufficient to get the model loading. In the subsequent chunk (chunk 0 of segment 14), the assistant encountered multiple additional issues:

  1. The maybe_override_with_speculators function crashed because it didn't support the glm-dsa architecture, requiring a config patch.
  2. A torch.bfloat16 dtype incompatibility with GGUF quantization required adding --dtype float16 to the vLLM serve command.
  3. The most critical issue: no valid attention backend existed for the combination of SM120 compute capability, sparse MLA (DSA indexer), and qk_nope_head_dim=192. This required implementing an entirely new TritonMLASparseBackend from scratch.
  4. A KeyError in weight_utils.py caused by a global string replacement (name.replace("weight", "qweight")) that corrupted parameter names containing "weight" as a substring (e.g., weights_proj became qweight_types_proj). This was a bug in the patch itself that needed fixing. These issues were not foreseeable at the time of this message. The attention backend problem, in particular, was a hardware compatibility issue that could only be discovered by actually attempting to load the model on the Blackwell GPUs. The weight_utils.py string replacement bug was a latent defect in the patch that the assistant had written — a subtle interaction between the global string replacement logic and the specific naming conventions of the GLM-5 architecture.

The Broader Significance

This message represents the transition from development to deployment in a complex systems integration effort. The assistant had spent multiple rounds analyzing the vLLM GGUF loading pipeline, reverse-engineering the llama.cpp conversion script, inspecting the actual GGUF tensor shapes, writing patches, and iterating on the reassembly logic. This message is the moment when all that work is committed to the target system.

The two scp commands are deceptively simple. They carry the weight of dozens of earlier messages: the discovery of the kv_b_proj loading bug, the verification that the unsloth GGUF used n_head_kv=64 (not 1), the analysis of the 3D tensor shapes, the design of the sentinel suffix approach, and the careful implementation of the _reassemble_kv_b() method. Each line of the patched files represents a specific decision informed by careful investigation.

In the broader narrative of the conversation, this message is the culmination of the GGUF patching workstream. The subsequent rounds would reveal additional challenges (attention backend, string replacement bug), but those were separate issues — the core GGUF loading fix was complete and correct. The model would eventually load, thanks to the foundation laid in this message.