The Pivot Point: Confirming the `glm-dsa` Architecture in gguf-py

In the span of a single bash command and its output, an entire deployment strategy crystallizes. Message [msg 1540] appears, at first glance, to be one of the most trivial exchanges in a long and technically grueling coding session: a Python one-liner that iterates over a dictionary and prints a few filtered entries. But this message is anything but trivial. It is the moment of verification that unlocks a path forward after hours of uncertainty, the final piece of a puzzle that had threatened to derail the entire deployment of a 431 GB language model. To understand why this single command matters so deeply, one must trace the reasoning chain that led to it, the assumptions it validates, and the knowledge it creates.

The Context: A Pivot Under Pressure

The session had been a rollercoaster. The assistant and user had spent segments 7 through 11 of their conversation deeply invested in deploying the GLM-5 model using NVIDIA's NVFP4 quantization format via SGLang. After extensive benchmarking, kernel profiling, and optimization attempts — including a gather-then-cast patch that improved throughput by 29% — the user made a decisive call to abandon the NVFP4 path entirely ([msg 1517]). The new plan was to deploy GLM-5 using Unsloth's UD-Q4_K_XL GGUF quantization on vLLM, a combination that promised lower memory footprint (431 GB vs. 860 GB for FP8) and better compatibility with the available 768 GB of VRAM across eight RTX PRO 6000 Blackwell GPUs.

But this pivot immediately hit a wall. As the assistant discovered in [msg 1517], vLLM's GGUF support depends critically on the transformers library for parsing GGUF metadata into HuggingFace model configurations. The installed transformers v5.2.0 did not include the deepseek2 or glm_moe_dsa architectures in its GGUF_CONFIG_MAPPING. Multiple GitHub issues confirmed that every attempt to load DeepSeek or GLM GGUF models on vLLM failed with the same error: ValueError: GGUF model with architecture deepseek2 is not supported yet. The user's response was unambiguous: "E. add this gguf support to vllm" ([msg 1518]).

The Research Phase: Uncovering the Layers

The assistant launched three parallel research tasks ([msg 1520]) to understand the full scope of what needed to be patched. The results were illuminating. The vLLM GGUF loader (gguf_loader.py) already had manual weight mappings for DeepSeek architectures, including expert weight sideloading and e_score_correction_bias. The GlmMoeDsaForCausalLM class already existed as a stub in vLLM, inheriting from DeepseekV2ForCausalLM. The blocker was exclusively in transformers — the GGUF_CONFIG_MAPPING dictionary simply had no entry for the GLM/DeepSeek architecture family.

However, a critical ambiguity remained. What name did the GLM-5 GGUF file use for its architecture? The llama.cpp conversion script (convert_hf_to_gguf.py) defined a GlmMoeDsaModel class that inherited from DeepseekV2Model, but the GGUF architecture constant might be either deepseek2 (inherited) or glm-dsa (a separate entry in llama.cpp's LLM_ARCH enum). This distinction mattered enormously: if the GGUF file used deepseek2, the patch would need to add a deepseek2 entry to transformers' config mapping; if it used glm-dsa, the patch would need to add glm-dsa instead — and the gguf-py library itself would need to support that architecture constant.

The Dependency Chain Problem

The assistant systematically worked through the dependency chain. First, vLLM nightly (0.16.0rc2.dev313) was installed (<msg id=1522-1523>), which unfortunately downgraded transformers from 5.2.0 to 4.57.6. The assistant immediately corrected this by installing transformers from git HEAD, yielding version 5.3.0.dev0 (<msg id=1524-1525>). But even the bleeding-edge transformers lacked deepseek2 or glm-dsa support (<msg id=1526-1527>).

Then came the crucial check: what does the gguf-py library know? The installed version (0.17.1 from PyPI) listed 57 architectures, and none of them were glm_dsa or deepseek2 in the GLM sense (<msg id=1533-1536>). There was a deepseek (arch 54) and deepseek2 (arch 55), but these corresponded to older model families. The glm-dsa architecture was simply absent from the PyPI release.

This was a critical finding. The gguf-py library is the Python binding for the GGUF format specification. If it doesn't know about the glm-dsa architecture, it cannot parse the tensor names, metadata keys, or weight layouts of a GLM-5 GGUF file. Any patch to transformers or vLLM would be useless if the foundational gguf-py library couldn't even recognize the architecture.

The Solution: Installing from Source

The assistant's response was to install gguf-py directly from the llama.cpp repository's git source ([msg 1539]). This is a significant step: instead of relying on the PyPI release cadence, the assistant pulled the absolute latest version of the GGUF specification from the llama.cpp project's HEAD commit (10b26ee23a2d1b563a62db1ea4710cf8b723791a). This is where bleeding-edge architecture support lives, often weeks or months before it reaches PyPI.

The Subject Message: Verification

This brings us to [msg 1540]. The message, in its entirety, is a single bash command executed over SSH on the remote machine:

[assistant] [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 -c "
import gguf
for key, value in sorted(gguf.MODEL_ARCH_NAMES.items()):
    if \"glm\" in value.lower() or \"deepseek\" in value.lower() or \"dsa\" in value.lower():
        print(f\"  {key}: {value}\")
print()
print(\"Total architectures:\", len(gguf.MODEL_ARCH_NAMES))
"'
  68: deepseek
  69: deepseek2
  70: chatglm
  71: glm4
  72: glm4moe
  73: glm-dsa

Total architectures: 121

The Python one-liner iterates over the gguf.MODEL_ARCH_NAMES dictionary — a mapping from integer architecture IDs to their string names — filtering for any entry containing "glm", "deepseek", or "dsa" in a case-insensitive manner. The output reveals exactly what the assistant needed to see: glm-dsa is present as architecture ID 73, alongside related architectures like deepseek2 (69), chatglm (70), glm4 (71), and glm4moe (72). The total count of 121 architectures confirms that this is the bleeding-edge version of the library, far beyond the 57 architectures available in the PyPI release.

This output is the confirmation the assistant needed. The glm-dsa architecture is present as entry 73 out of 121 total architectures. The gguf-py library from llama.cpp HEAD now recognizes the GLM-5 architecture. The path forward is clear: the transformers GGUF_CONFIG_MAPPING needs a glm-dsa entry that maps GGUF metadata keys to HuggingFace config parameters, and the vLLM GGUF loader needs to handle the glm_moe_dsa model_type with its specific tensor transformations (split kv_b_proj, fused gate_up_proj for experts, DSA indexer tensors, and e_score_correction_bias).

Assumptions and Their Validation

Several assumptions underpinned this verification step. The first was that llama.cpp's git HEAD would contain the glm-dsa architecture definition. This was a reasonable assumption given that Unsloth had published GLM-5 GGUF files, implying the conversion script existed. The second assumption was that the architecture name would be glm-dsa rather than deepseek2 — the output confirms this, though it also shows deepseek2 exists as a separate entry (arch 69), suggesting the GLM-5 conversion intentionally uses a distinct architecture identifier. The third assumption was that installing from source would not break existing functionality — the jump from 57 to 121 total architectures indicates the llama.cpp HEAD has substantially more architectures than the PyPI release, but the existing ones remain intact.

The Knowledge Created

This message creates critical output knowledge. It confirms that the gguf-py library now supports glm-dsa, which means any patch written for transformers and vLLM can reference this architecture with confidence. It establishes the exact architecture name string (glm-dsa) that must appear in the GGUF_CONFIG_MAPPING dictionary. It also reveals the scale of the gguf-py library's growth — 121 architectures versus the 57 in the PyPI release — which hints at the rapid pace of development in the GGUF ecosystem.

For the assistant, this message also represents a subtle but important shift in strategy. Earlier in the session, the assistant had assumed that the GGUF file might use deepseek2 as its architecture name (since GlmMoeDsaModel inherits from DeepseekV2Model). This assumption led to checking for deepseek2 in the transformers config mapping and in the gguf-py architecture list. The discovery that glm-dsa is a separate, distinct architecture means the patch must be written specifically for this architecture, not as a subclass of the deepseek2 handling. This distinction matters for tensor name mappings: while the GLM-DSA architecture shares many structural similarities with DeepSeekV2 (MLA attention, MoE with shared experts), it has unique tensor names like attn_k_b and attn_v_b (split from the fused kv_b_proj) and DSA-specific indexer tensors.

The Broader Significance

In the larger narrative of this coding session, [msg 1540] is the moment when the GGUF deployment path becomes viable. Before this message, the assistant was operating on incomplete information — the architecture name was uncertain, the gguf-py support was unconfirmed, and the entire patching effort risked being built on a wrong foundation. After this message, the assistant can proceed with confidence to write the transformers and vLLM patches, knowing that the underlying GGUF specification supports the target architecture.

This message also illustrates a broader principle of systems engineering: when building on a complex dependency chain, you must verify each layer from the bottom up. The assistant could not patch transformers until it knew the correct architecture name. It could not know the architecture name until it checked the GGUF file's metadata. It could not parse that metadata until gguf-py supported the architecture. And it could not get that support from PyPI — it had to go to the source. Each layer of the stack had to be validated before the next layer could be addressed. Message [msg 1540] is the validation of the foundational layer, and everything else depends on it.

The Pivot Point: Confirming the glm-dsa Architecture in gguf-py