The Missing Architecture: Diagnosing GGUF Support Gaps in the GLM-5 Deployment Pipeline

Introduction

In the sprawling effort to deploy the GLM-5 model on a multi-GPU Ubuntu 24.04 system, message [msg 1536] represents a quiet but pivotal diagnostic moment. After weeks of battling flash-attn build failures, NVFP4 quantization bottlenecks, and KV cache cast overhead, the assistant had pivoted decisively toward a new strategy: deploying GLM-5 via GGUF quantization (UD-Q4_K_XL) on vLLM. The user had explicitly chosen option E — "add this gguf support to vllm" — rejecting all alternative paths. But before any patch could be written, the assistant needed to understand the full dependency chain. This message, a single SSH command listing all supported architectures in the installed gguf-py library, is the moment where the assistant discovers that even the foundational Python library for GGUF parsing lacks support for the glm-dsa architecture used by GLM-5. It is a moment of calibration — a reality check that reveals the true scope of the patching effort required.

The Strategic Pivot to GGUF

To understand why this message matters, one must appreciate the context that led to it. The assistant had spent the better part of a session (Segment 12) trying to deploy GLM-5 using the NVFP4 quantization path on SGLang. That path had yielded 13.5 tok/s single-stream performance, but the user decided to abandon NVFP4 after the assistant identified the KV cache FP8-to-BF16 cast as a fundamental bottleneck consuming 69% of decode time. The new plan was to use Unsloth's UD-Q4_K_XL GGUF quantization on vLLM — a path that promised lower memory footprint (431 GB vs 860 GB for FP8) and potentially better performance.

However, the assistant quickly discovered a critical blocker: vLLM's GGUF support depends on the transformers library for parsing GGUF metadata into HuggingFace config objects, and transformers v5.2.0 did not include the deepseek2 or glm_moe_dsa architecture in its GGUF_CONFIG_MAPPING. Multiple GitHub issues confirmed that every attempt to run DeepSeek GGUF models on vLLM failed with the error ValueError: GGUF model with architecture deepseek2 is not supported yet. The user's directive was clear: patch the code to make it work.

The assistant launched three parallel research tasks to understand the full picture: (1) the transformers GGUF config mapping system, (2) vLLM's GGUFModelLoader, and (3) the GLM-5 GGUF tensor structure. The research revealed that vLLM already had manual weight mappings for DeepSeek architectures — the blocker was solely in transformers. But there was another layer to the dependency chain: the gguf-py library, which defines the architecture constants and tensor name maps that both transformers and vLLM depend on.

The Diagnostic Command

The message itself is deceptively simple. The assistant runs:

ssh root@[REDACTED] '~/ml-env/bin/python3 -c "
import gguf
for key, value in sorted(gguf.MODEL_ARCH_NAMES.items()):
    print(f\"  {key}: {value}\")
"'

This is a straightforward Python one-liner that imports the gguf library and iterates over its MODEL_ARCH_NAMES dictionary — a mapping from integer constants to architecture name strings. The output is a numbered list of 57 architectures, from clip (1) through glm4 (57).

The command is notable for what it does not include: any error handling, any filtering, any analysis. It is a raw dump of the data, letting the assistant (and the user) see exactly what the installed gguf-py library knows about. The assistant could have written a more targeted query — checking only for glm-dsa or deepseek2 — but chose instead to dump the entire list. This is a deliberate diagnostic strategy: by seeing the full landscape, the assistant can verify that the library is correctly installed, check the version's completeness, and spot any unexpected entries that might hint at partial support.

What the Output Reveals

The output reveals a critical gap. The gguf-py library installed on the system (version 0.17.1) lists architectures up to glm4 (index 57), but glm-dsa is not present. This is the architecture name that llama.cpp uses for GLM-5 models — the LLM_ARCH_GLM_DSA constant defined in llama.cpp's gguf-py source. The installed version simply predates this addition.

This finding has profound implications for the patching effort. The assistant now knows that the dependency chain has three broken links:

  1. gguf-py (0.17.1): Does not define LLM_ARCH_GLM_DSA or its tensor name mappings. Without this, no higher-level library can interpret GLM-5 GGUF files.
  2. transformers (5.3.0.dev0): Does not have glm_dsa or deepseek2 in its GGUF_CONFIG_MAPPING. Even if gguf-py were updated, transformers would still fail to parse the GGUF metadata into an HF config.
  3. vLLM (0.16.0rc2.dev313): Has manual weight mappings for DeepSeek architectures but depends on transformers for the initial GGUF parsing. The assistant had previously assumed that only transformers needed patching — the research task on vLLM's GGUF loader had suggested that vLLM already supported DeepSeek architectures internally. But this message reveals that the problem is deeper: even if transformers were patched, the gguf-py library would still fail to recognize the glm-dsa architecture when vLLM calls gguf.GGUFReader to read tensor metadata.

The Dependency Chain Problem

This message illustrates a classic systems integration challenge: the GGUF loading pipeline is a chain of three libraries, each with its own architecture registry. gguf-py (from llama.cpp) defines the fundamental constants. transformers maps GGUF metadata keys to HuggingFace config parameters. vLLM maps HuggingFace tensor names to its own model implementation. For a new architecture like GLM-5's glm-dsa, all three layers must be updated.

The assistant's research had already revealed that llama.cpp's convert_hf_to_gguf.py defines a GlmMoeDsaModel class that inherits from DeepseekV2Model, and that the latest gguf-py source (from llama.cpp HEAD) includes LLM_ARCH_GLM_DSA with a complete tensor name map. But the installed version — 0.17.1 from PyPI — predates these changes. The solution is to install gguf-py from llama.cpp source rather than from PyPI.

Assumptions and Blind Spots

The assistant made a reasonable but incorrect assumption earlier in the session: that the gguf-py library from PyPI would be sufficient. In [msg 1533], the assistant checked gguf.MODEL_ARCH_NAMES for glm-dsa or deepseek2 and found only deepseek (54), deepseek2 (55), chatglm (56), and glm4 (57). This message confirms that finding definitively. The assumption that PyPI's gguf-py would be up-to-date with llama.cpp's latest developments was optimistic — the GLM-5 model was released relatively recently, and the glm-dsa architecture addition to llama.cpp may not have been packaged into a PyPI release yet.

Another subtle assumption visible in the assistant's approach is that the architecture name in gguf-py would match the GGUF file's architecture string exactly. The GGUF file for GLM-5 uses the architecture string glm-dsa (with a hyphen), while the Python constant would be LLM_ARCH_GLM_DSA mapping to the string glm-dsa. The assistant correctly anticipated this naming convention, but needed to verify that the constant existed at all.

Implications for the Patch

The output of this message directly shapes the assistant's next steps. The assistant now knows that it must:

  1. Install gguf-py from llama.cpp source to get the glm-dsa architecture definitions
  2. Patch transformers ggml.py to add glm-dsa to GGUF_CONFIG_MAPPING with the correct parameter mappings
  3. Patch vLLM's gguf_loader.py to handle GLM-5's specific tensor layout (split kv_b_proj, fused gate_up_proj, DSA indexer tensors) The message also confirms that the assistant's multi-pronged research strategy was correct. By checking all three libraries independently, the assistant avoided the mistake of patching only transformers and then discovering that gguf-py also needed updating. The systematic enumeration of all 57 architectures — rather than a targeted check for glm-dsa — provides a complete baseline that the assistant can refer back to when verifying that the updated gguf-py installation has added the missing architecture.

Conclusion

Message [msg 1536] is a quiet but essential diagnostic step in a complex software integration effort. It demonstrates the importance of verifying every layer of a dependency chain before writing patches, and it reveals the true scope of the work required to add GGUF support for GLM-5 to vLLM. The assistant's methodical approach — researching three codebases in parallel, then verifying each one's actual state against the installed versions — is a textbook example of how to approach a multi-library integration problem. The missing glm-dsa architecture in gguf-py is not just a data point; it is the key insight that redefines the patching strategy from a single-library fix to a coordinated update across three interdependent packages.