The Moment of Verification: Confirming the GGUF Architecture Blocker for GLM-5 on vLLM

In the sprawling, multi-day effort to deploy the 744-billion-parameter GLM-5 model on a cluster of RTX PRO 6000 Blackwell GPUs, there came a decisive turning point. After weeks of wrestling with NVFP4 quantization, FP8-to-BF16 KV cache cast overhead, and SGLang server tuning, the team had pivoted hard toward a new approach: deploying GLM-5 via GGUF UD-Q4_K_XL quantization on vLLM. But this pivot ran headlong into a critical unknown. Would vLLM's GGUF loader actually support the glm-dsa architecture used by GLM-5? Message [msg 1513] captures the precise moment the assistant attempted to answer that question definitively — and hit an unexpected wall.

The Context: A Pivot Under Pressure

To understand message [msg 1513], one must appreciate the journey that led to it. The preceding segment (Segment 11) had been a painful exercise in bottleneck diagnosis. The assistant had painstakingly profiled the SGLang server, discovering that a staggering 69% of single-stream decode time was consumed by an FP8-to-BF16 cast of the KV cache — a fundamental data type conversion that was burning compute cycles with no benefit to model quality. A clever "gather-then-cast" patch had recovered 29% of performance, but the user ultimately decided to abandon the NVFP4 path entirely.

The new direction was GGUF quantization. Unsloth's UD-Q4_K_XL quantization promised better quality than standard Q4_K_M, and the model files were already available on Hugging Face as a set of 431 GB split files. The user's directive was unambiguous: deploy GLM-5 via GGUF on vLLM, and if that didn't work, add the necessary support to vLLM rather than falling back to alternatives like llama.cpp or FP8.

But as the assistant began laying the groundwork — checking disk space, installing vLLM nightly, and examining the GGUF documentation — a troubling pattern emerged. Multiple GitHub issues reported that DeepSeek-based GGUF models consistently failed with the error ValueError: GGUF model with architecture deepseek2 is not supported yet. Since GLM-5 is built on a DeepSeekV3-derived architecture called glm-dsa (or glm_moe_dsa in some contexts), this was a direct threat to the entire deployment plan.

The Message: A Deliberate Act of Verification

Message [msg 1513] is deceptively brief — just two sentences of commentary followed by a bash command and its error output. But within the narrative of the session, it represents a critical methodological choice: the assistant decided to verify the architecture support status by examining the actual source code rather than relying on documentation.

The message reads:

The transformers docs say supported architectures include "Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, and more." No mention of DeepSeek or GLM. Let me check the actual source code for the complete list:

>

`` [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 -c " from transformers.integrations.ggml import GGUF_SUPPORTED_ARCHITECTURES for arch in sorted(GGUF_SUPPORTED_ARCHITECTURES): print(arch) " 2>&1' Traceback (most recent call last): File "<string>", line 2, in <module> ImportError: cannot import name 'GGUF_SUPPORTED_ARCHITECTURES' from 'transformers.integrations.ggml' ``

The first sentence references documentation the assistant had just fetched from Hugging Face's transformers documentation page ([msg 1512]). That page listed supported architectures in a human-readable format, but the list was vague — "and more" could mean anything. The assistant needed to know, with certainty, whether deepseek2 or glm_moe_dsa or any GLM-related architecture was in the supported set.

The decision to check via source code rather than documentation is itself revealing. Documentation can be stale, incomplete, or misleading. The phrase "and more" is a hedge that could conceal either a small handful of obscure architectures or a large set that the docs author simply didn't bother to enumerate. By importing GGUF_SUPPORTED_ARCHITECTURES directly from the transformers package installed on the machine, the assistant would get the ground truth — whatever the installed version of transformers (5.2.0) actually supports.

The Assumption That Failed

The bash command reveals a specific assumption: that transformers.integrations.ggml would expose a constant called GGUF_SUPPORTED_ARCHITECTURES containing the complete list. This was a reasonable guess — the ggml.py file in transformers' integrations module is the primary location for GGUF-related code, and many Python packages expose supported-architecture lists as module-level constants.

But the assumption was wrong. The ImportError tells us that no such constant exists in that module. The architecture list might be:

Input Knowledge Required

To fully understand this message, the reader needs several pieces of context:

  1. The GGUF architecture problem: vLLM's GGUF loader delegates architecture identification to the transformers library. When a GGUF file is loaded, transformers reads the file's metadata to determine the architecture name (e.g., deepseek2, llama, glm_moe_dsa), then maps that to a Hugging Face model class. If the architecture isn't in the mapping, loading fails.
  2. The GLM-5 architecture: GLM-5 uses the glm-dsa architecture (sometimes written glm_moe_dsa), which is derived from DeepSeekV3. This is a Mixture-of-Experts architecture with 256 experts, Multi-Head Latent Attention (MLA), and several unique tensor structures.
  3. The transformers version: The environment has transformers 5.2.0 installed, which is very recent. The question is whether this version includes the glm-dsa architecture in its GGUF support.
  4. The vLLM GGUF dependency chain: vLLM doesn't parse GGUF architectures itself — it relies on transformers to do the initial parsing and model class selection. This means the blocker could be in either codebase, but transformers is the gatekeeper.
  5. Previous research: Messages [msg 1509] through [msg 1512] had already surfaced multiple GitHub issues confirming that deepseek2 GGUF models fail with vLLM. The assistant was now trying to confirm whether this was still true in the latest transformers.

Output Knowledge Created

Despite the error, this message produces several valuable pieces of knowledge:

  1. Negative confirmation: The constant GGUF_SUPPORTED_ARCHITECTURES does not exist in transformers.integrations.ggml at version 5.2.0. This means the architecture list is not trivially accessible.
  2. Methodological insight: The approach of importing a constant failed, but the error message reveals the exact file path (/root/ml-env/lib/python3.12/site-packages/transformers/integrations/ggml.py), which the assistant can now inspect directly.
  3. Refined search direction: The assistant now knows it needs to look deeper — either by examining the ggml.py file's contents directly, searching for architecture mapping dictionaries, or checking the transformers source code on GitHub.
  4. Confirmation of the blocker's nature: The fact that the architecture list isn't easily enumerable suggests that transformers' GGUF support is still evolving and may not have a clean, comprehensive mapping for all architectures. This reinforces the earlier suspicion that GLM-5 GGUF support will require significant work.

The Thinking Process

The assistant's reasoning in this message follows a clear pattern: document → verify → adapt. The documentation (transformers GGUF page) provided a partial answer — a list of architectures with an ambiguous "and more." Rather than accepting this uncertainty, the assistant formulated a test: import the architecture list directly and enumerate it.

The choice of GGUF_SUPPORTED_ARCHITECTURES as the import target suggests the assistant was familiar with common patterns in ML framework codebases. Many libraries (including vLLM itself) expose supported-feature lists as module-level constants with names like SUPPORTED_ARCHITECTURES or _ARCH_MAP. The specific name GGUF_SUPPORTED_ARCHITECTURES was likely inferred from the transformers documentation or from prior experience with the codebase.

When the import failed, the assistant did not panic or draw premature conclusions. The error simply redirected the investigation. In the messages that follow ([msg 1514] onward), the assistant would pivot to examining the ggml.py file directly, searching for architecture mapping dictionaries, and ultimately discovering that the real blocker was in transformers' GGUF_CONFIG_MAPPING — a dictionary that maps GGUF architecture names to Hugging Face model configuration classes. This dictionary, it turned out, did not include deepseek2 or any GLM variant.

Why This Message Matters

Message [msg 1513] is, in one sense, a failure — the command didn't produce the desired output. But in the broader arc of the session, it's a crucial diagnostic step. The assistant is systematically ruling out possibilities, testing assumptions, and gathering information. The error tells the assistant something important about the structure of the transformers codebase, which in turn shapes the next phase of the investigation.

This message also exemplifies a key principle of the opencode session's methodology: when documentation is ambiguous, go to the source code. The transformers docs said "and more" — but the assistant needed to know exactly what "more" meant. The attempt to enumerate the list programmatically was the right approach, even though the specific import path was wrong.

The message also marks a transition. Before this point, the assistant was still gathering information about whether GGUF support existed. After this point, the investigation becomes more active — the assistant will read source files, search for architecture mappings, and eventually write a custom patch for vLLM's gguf_loader.py. Message [msg 1513] is the last moment of passive research before the work of active engineering begins.

Conclusion

Message [msg 1513] captures a brief but pivotal moment in a complex deployment effort. A simple bash command, designed to enumerate supported GGUF architectures in the transformers library, fails with an ImportError — but that failure is itself a data point. It tells the assistant that the architecture support mapping is not trivially accessible, that the codebase structure is more complex than expected, and that the path to GLM-5 GGUF support on vLLM will require deeper intervention.

In the messages that follow, the assistant will rise to that challenge, ultimately writing a comprehensive patch for vLLM's GGUF loader that handles the glm_moe_dsa architecture's unique tensor structure — including split KV projections, fused gate-up expert weights, and the e_score_correction_bias parameter. But that work begins here, with a failed import and a determination to dig deeper.