Segment 12

In this sub-session, the assistant executed the pivot to GGUF UD-Q4_K_XL quantization on vLLM. After freeing disk space and starting the 431 GB GGUF download, the assistant discovered a critical blocker: vLLM's GGUF support depends on transformers for architecture metadata parsing, and neither transformers (v5.2.0) nor gguf-py (0.17.1) included the glm-dsa architecture used by GLM-5. The user explicitly rejected alternative paths and directed the assistant to add GGUF support to vLLM. The assistant then conducted parallel deep-dive research into transformers GGUF config mapping, vLLM's GGUFModelLoader, and the GLM-5 GGUF tensor structure via the latest gguf-py from llama.cpp HEAD. Key findings included that the blocker is solely in transformers, vLLM already has manual weight mappings for DeepSeek architectures, and the gguf-py library from llama.cpp already defines LLM_ARCH_GLM_DSA with a complete tensor name map. The assistant installed vLLM nightly (0.16.0rc2.dev313), upgraded transformers to 5.3.0.dev0 from git HEAD, installed gguf-py from llama.cpp source (adding glm-dsa architecture support), and began writing a comprehensive patch for vLLM's gguf_loader.py to handle glm_moe_dsa model_type with expert weight sideloading, KV split reassembly, and indexer/nextn tensor mapping. The GGUF download was started in the background.

Pivot to GGUF deployment on vLLMInstall vLLM nightly and upgrade transformersInstall gguf-py with glm-dsa supportResearch GGUF architecture support in transformers/vLLMWrite patch for vLLM gguf_loader.py for GLM-5Download 431 GB GGUF split files

The GGUF Frontier: Engineering vLLM Support for GLM-5 from First Principles 3350 words

Chunks