Segment 13
In this sub-session, the assistant resumed work on deploying GLM-5 GGUF with vLLM. After finding the GGUF model download had failed, it restarted the download using a Python script with huggingface_hub.snapshot_download for reliability. While the download proceeded, the assistant finalized and applied critical patches to vLLM's gguf_loader.py and weight_utils.py, adding support for the glm_moe_dsa architecture by manually mapping expert weights and the split attn_k_b/attn_v_b tensors back to the single kv_b_proj weight. This patch also fixed a latent bug in DeepSeek V2/V3 GGUF support. Once the download completed, the assistant built the llama-gguf-split tool from llama.cpp source and merged the 10 split GGUF files into a single 402GB file. Upon inspecting the merged file's metadata, the assistant discovered that the attn_k_b and attn_v_b tensors used an older shape representation (n_head_kv=64), prompting a revision of the kv_b reassembly logic to use a simple transpose-and-concatenate approach instead of the more complex MQA reversal. By the end of the session, all prerequisites for running the GLM-5 GGUF model were complete: the model was downloaded, merged, and the vLLM source code was patched.
The Architecture of a Patch: Reverse-Engineering GLM-5's GGUF Attention Weights for vLLM