The Confirmation Blow: Discovering Transformers' GGUF Architecture Gap

In the long and winding journey to deploy the 744-billion-parameter GLM-5 model on a multi-GPU server, there comes a moment where a single command definitively closes the door on one path and forces a radical pivot. Message [msg 1515] is that moment. It is a brief, almost anticlimactic exchange—a single SSH command executed on a remote server, followed by a list of 20 architecture names. Yet this output carries devastating weight: it confirms that the transformers library, version 5.2.0, does not support the glm_moe_dsa architecture required to load GLM-5 via GGUF. This single discovery reshapes the entire deployment strategy and sets the stage for one of the most technically ambitious moves in the session: writing a custom patch to add GGUF support to vLLM.

The Message Itself

The message is terse and purely investigative:

[assistant] [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 -c "
from transformers.integrations.ggml import GGUF_CONFIG_MAPPING
for arch in sorted(GGUF_CONFIG_MAPPING.keys()):
    print(arch)
" 2>&1'
bloom
deci
falcon
gemma2
gemma3
general
gpt2
lfm2
llama
mamba
mistral
nemotron
phi3
qwen2
qwen2_moe
qwen3
qwen3_moe
stablelm
starcoder2
t5
tokenizer
umt5

There is no reasoning block, no commentary, no analysis attached to this message. It is a raw data-gathering operation—a direct query to the installed Python environment to enumerate the architectures that transformers recognizes for GGUF file loading. The assistant does not interpret the results within the message; that interpretation will come in subsequent messages. But for anyone following the conversation, the meaning is immediately clear: deepseek2, glm_moe_dsa, and any DeepSeek V3-derived architecture are absent from this list. The path forward through standard GGUF loading is blocked.

The Reasoning and Motivation: Why This Message Was Written

To understand why this message exists, we must trace the chain of reasoning that led to it. The assistant had been pursuing a GGUF-based deployment of GLM-5 using vLLM, following the user's explicit directive after the NVFP4 quantization path had proven untenable. The user had rejected all alternative approaches—reverting to sglang, using llama.cpp directly, or deploying the FP8 variant—and insisted on the GGUF + vLLM combination.

The assistant had already gathered significant circumstantial evidence that this path might be blocked. In the preceding messages ([msg 1509] through [msg 1514]), it had conducted a thorough investigation:

  1. GitHub issue research ([msg 1509]): Web searches revealed multiple reports of ValueError: GGUF model with architecture deepseek2 is not supported yet when attempting to load DeepSeek-based GGUF models in vLLM. These issues dated from mid-to-late 2025, and there was no clear evidence they had been resolved.
  2. Documentation review ([msg 1512]): The official Hugging Face transformers documentation listed supported GGUF architectures as "Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, and more"—with no mention of DeepSeek or GLM.
  3. Failed import attempt ([msg 1513]): An initial attempt to import GGUF_SUPPORTED_ARCHITECTURES from transformers.integrations.ggml failed with an ImportError, suggesting the API had changed or the constant didn't exist in that form.
  4. Discovery of the actual API ([msg 1514]): By listing the module's contents, the assistant found GGUF_CONFIG_MAPPING, GGUF_CONFIG_DEFAULTS_MAPPING, and GGUF_TOKENIZER_MAPPING—the actual API for architecture support detection. Message [msg 1515] is the culmination of this investigation. It uses the API discovered in the previous step to get the definitive, authoritative answer. The assistant could have speculated based on GitHub issues and documentation, but it chose to get ground truth from the actual installed library. This is a hallmark of rigorous debugging: verify the blocker on the actual system, not just in documentation.

The Assumptions at Play

Several assumptions underpin this message, both on the part of the assistant and implicitly on the part of the user:

Assumption 1: vLLM relies on transformers for GGUF architecture support. This is a critical architectural assumption. The assistant has been operating under the understanding that vLLM's GGUF loader delegates to transformers for parsing GGUF metadata and mapping architecture names to model configurations. If vLLM had its own independent GGUF architecture mapping, the transformers list would be irrelevant. The assistant's subsequent actions (writing a patch for vLLM's gguf_loader.py) confirm this assumption was correct—vLLM does indeed use transformers for this purpose.

Assumption 2: The installed transformers version (5.2.0) is representative. The assistant assumes that the GGUF_CONFIG_MAPPING dictionary in the installed version reflects the current state of GGUF support. This is a reasonable assumption, but it's worth noting that transformers 5.2.0 is a very recent version (released in early 2026), and if a newer version had added DeepSeek support, the assistant would have needed to upgrade. In fact, in subsequent messages, the assistant does upgrade transformers to 5.3.0.dev0 from git HEAD, and later discovers that even the development version lacks glm-dsa support.

Assumption 3: The architecture name in the GGUF file is the key to loading. The assistant assumes that the GGUF file's architecture metadata field (which for GLM-5 is glm_moe_dsa or glm-dsa) must match a key in GGUF_CONFIG_MAPPING. This is correct—the GGUF loading pipeline uses this mapping to determine which Hugging Face model configuration class to instantiate.

Assumption 4: The user is willing to accept a custom patching approach. This is an implicit assumption that will be validated in the next message, where the user explicitly directs the assistant to "add GGUF support to vLLM" (option E). The assistant's willingness to invest in this investigation suggests confidence that the user would prefer a technical solution over abandoning the GGUF path.

Mistakes and Incorrect Assumptions

While the message itself is factually correct (it accurately reports what the installed library supports), there is one notable limitation in the investigation at this point:

The assistant did not check whether vLLM has its own independent GGUF architecture mapping. The GGUF_CONFIG_MAPPING in transformers is used by the transformers library's own GGUF loading functionality. However, vLLM's GGUFModelLoader class (which the assistant will later examine in detail) has its own mechanism for mapping GGUF architectures to model implementations. In fact, vLLM's approach is quite different: it uses a model_type field from the Hugging Face config (not the GGUF architecture name directly) to select which model loader to use. The GGUF architecture name is primarily used to reconstruct the Hugging Face configuration from GGUF metadata.

This distinction becomes crucial later. The assistant discovers that the blocker is specifically in transformers' ability to parse the GGUF metadata into a Hugging Face AutoConfig object. vLLM's own model loading code already has manual weight mappings for DeepSeek architectures—it just can't get past the configuration parsing step. This means the fix needs to happen at the transformers level (or be bypassed), not in vLLM's model loader.

Another subtle mistake: the assistant did not verify that the architecture name stored in the actual GLM-5 GGUF files matches what it's searching for. The Unsloth page lists the architecture as glm-dsa, but the GGUF file might use a different internal name like glm_moe_dsa or deepseek2. The assistant later discovers this by examining the GGUF file directly using gguf-py from llama.cpp HEAD, which reveals that gguf-py (version from llama.cpp) already defines LLM_ARCH_GLM_DSA with a complete tensor name map. This is a critical piece of information that the assistant doesn't have at the time of message [msg 1515].

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Knowledge of the GGUF file format: GGUF (GPT-Generated Unified Format) is a single-file format for storing quantized models, originally developed for llama.cpp. It includes metadata fields like architecture name, tensor names, and quantization parameters.
  2. Understanding of the transformers library architecture: The transformers library by Hugging Face provides a GGUF integration module (transformers.integrations.ggml) that can parse GGUF files and reconstruct Hugging Face model configurations. The GGUF_CONFIG_MAPPING dictionary maps GGUF architecture names (like llama, qwen2) to Hugging Face configuration classes.
  3. Knowledge of the GLM-5 model architecture: GLM-5 is a 744B-parameter Mixture-of-Experts model with 256 experts, derived from the DeepSeek V3 architecture. It uses the glm_moe_dsa architecture type (or glm-dsa in GGUF metadata), which is a variant of the deepseek2 architecture with additional features like DSA (Dynamic Sparse Attention).
  4. Context of the deployment struggle: The assistant has spent many hours trying various deployment approaches—NVFP4 quantization with SGLang, FP8 with vLLM, and now GGUF with vLLM—each encountering blockers related to architecture support, kernel compatibility, or quantization format.
  5. The vLLM GGUF loading pipeline: vLLM's GGUF support is "highly experimental" and works by using transformers to parse the GGUF metadata into a Hugging Face configuration, then using that configuration to select the appropriate vLLM model loader. If transformers cannot parse the architecture, the entire pipeline fails.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. Definitive proof of the blocker: The output provides an authoritative, system-verified list of supported GGUF architectures. The absence of deepseek2, glm_moe_dsa, or any GLM variant is conclusive evidence that the standard GGUF loading path is blocked.
  2. The exact API surface: The message confirms that GGUF_CONFIG_MAPPING is the correct API to query for architecture support. This is useful for anyone debugging similar issues with other architectures.
  3. The scope of the problem: By listing all 20 supported architectures, the message makes it clear that this is not a minor omission—the GLM/DeepSeek architecture family is entirely absent from a list that includes many other architectures (including the relatively niche deci, lfm2, and nemotron).
  4. A baseline for patching: The list serves as a reference point. Any patch to add glm_moe_dsa support will need to add entries to this mapping, and the existing entries serve as templates for how to structure the new entries.
  5. Confirmation of the investigation direction: The message validates the assistant's earlier research (GitHub issues, documentation review) and confirms that the blocker is real and must be addressed directly rather than worked around.

The Thinking Process Visible in the Reasoning

While message [msg 1515] itself contains no explicit reasoning block, the thinking process is visible in the sequence of messages that led to it. The assistant's approach follows a clear investigative pattern:

Step 1: Formulate a hypothesis. Based on GitHub issues and documentation, the assistant hypothesizes that transformers does not support the glm_moe_dsa architecture for GGUF loading.

Step 2: Attempt direct verification. In [msg 1513], the assistant tries to import GGUF_SUPPORTED_ARCHITECTURES—a constant that seems like it should exist based on the naming convention. This fails, but the failure is informative: it reveals that the API is different from expected.

Step 3: Explore the API surface. In [msg 1514], the assistant uses dir() to list the module's contents, discovering GGUF_CONFIG_MAPPING, GGUF_CONFIG_DEFAULTS_MAPPING, and GGUF_TOKENIZER_MAPPING. This is a classic debugging technique: when the expected API doesn't exist, enumerate what does exist.

Step 4: Execute the definitive query. In [msg 1515], the assistant uses the discovered API to get the definitive list. The command is carefully constructed: it imports GGUF_CONFIG_MAPPING, iterates over its sorted keys, and prints each one. The 2>&1 redirect ensures stderr is captured alongside stdout, preventing error messages from being lost.

Step 5: Interpret and act. The assistant does not include interpretation in this message, but the subsequent messages show the conclusion: the blocker is confirmed, and the path forward is to either upgrade transformers (which doesn't help, as later discovered) or write a custom patch.

The thinking process also reveals a sophisticated understanding of the software stack. The assistant knows that vLLM's GGUF support is layered on top of transformers, and that transformers' GGUF integration is in the ggml module (named after the GGML library that preceded GGUF). It knows the difference between the GGUF architecture name (used for metadata parsing) and the Hugging Face model type (used for model loading). It knows that the architecture name must match a key in the mapping dictionary for the configuration to be reconstructed.

The Broader Significance

Message [msg 1515] is a turning point in the session. It transforms the GGUF deployment path from "potentially viable, needs investigation" to "definitively blocked, needs intervention." This sets the stage for the most technically ambitious phase of the entire session: writing a comprehensive patch for vLLM's gguf_loader.py to handle the glm_moe_dsa architecture.

The message also demonstrates a methodological principle that runs throughout the session: verify on the actual system, not in documentation. The assistant could have concluded from GitHub issues alone that the path was blocked, but it chose to get ground truth from the installed library. This is especially important in a fast-moving ecosystem like ML infrastructure, where documentation can be outdated and GitHub issues can be resolved without clear communication.

The list of 20 architectures is itself interesting. It reveals which model families the transformers team has prioritized for GGUF support: the Llama family (including derivatives like Mistral, Qwen, Phi3), the Falcon/StarCoder family, and a few outliers like Mamba (state-space models) and T5/UMT5 (encoder-decoder models). The absence of DeepSeek is notable given DeepSeek's prominence in the open-source LLM landscape, but it reflects the reality that GGUF support is still experimental and prioritized based on community demand and the complexity of the architecture.

Conclusion

Message [msg 1515] is a deceptively simple query that delivers a devastating answer. In 20 lines of architecture names, it confirms that the standard GGUF deployment path for GLM-5 is blocked, forcing a pivot to custom patching. The message is a masterclass in investigative debugging: formulate a hypothesis, discover the API, execute the definitive query, and let the data speak. It is a moment of clarity in a complex technical struggle—the moment when speculation becomes certainty, and the real work can begin.