The Architecture Mapping That Broke the Model: Debugging GLM-5's Dense-to-MoE Transition in GGUF

In the ongoing effort to deploy the massive GLM-5 model on 8× NVIDIA Blackwell GPUs using GGUF quantization and vLLM, the assistant had reached a frustrating impasse. After successfully patching vLLM's weight loading code to force-dequantize certain tensors and skip unknown parameters, the model finally loaded without errors and the server began serving requests. But the output was gibberish — flat log-probability distributions, repeated tokens like "IRS" and "BW" appearing in every position, and near-random generation. This message (global index 1915) captures a pivotal diagnostic moment: the assistant's investigation into whether the weight name mapping between GGUF tensor names and HuggingFace parameter names was correct for GLM-5's unusual architecture, specifically the transition from dense layers to Mixture-of-Experts (MoE) layers.

The Message: A Diagnostic Pivot

The message reads:

The GLM-5 path (line 155-192) maps ffn_gate_exps.weight → experts.0.gate_proj.weight for layers 3-77 (MoE layers). But layers 0-2 are dense (not MoE). Let me check what happens with layers 0-2's FFN weights:

The auto name map (gguf-py's get_tensor_name_map) for glm-dsa architecture should handle the dense layers automatically. But the model_type is set to "glm-dsa" which is gguf-py's key. Let me verify that this maps correctly:

[bash command to check gguf-py's name mapping for glm-dsa]

This is a short message — just two paragraphs of reasoning followed by a bash command — but it represents a critical shift in the debugging strategy. The assistant had been pursuing several hypotheses in parallel: that the GGUF dequantization kernel had SM120 compatibility issues (ruled out by a direct CPU-vs-GPU comparison test showing max diff of 0.00012), that the Triton MLA attention backend had Blackwell-specific bugs (still being investigated), and that the KV cache dtype configuration was wrong. But the logprobs from the model — with known continuation tokens like "2" after "1" scoring logprobs around -20 to -24 — strongly suggested that the model's first layers were producing garbage hidden states. This pointed to a weight loading problem rather than an attention or kernel issue.

Why This Message Was Written: The Reasoning and Motivation

The assistant's motivation for this message stems from a specific architectural insight about GLM-5. GLM-5 is a hybrid model: its first three layers (0-2) use standard dense feed-forward networks (FFN), while layers 3 through 77 use Mixture-of-Experts (MoE) with multiple experts per layer. This hybrid design is unusual — most MoE models like Mixtral or DeepSeek-V2 use MoE for all layers. The GGUF file format, which stores quantized weights in a model-agnostic way, uses different tensor names for dense FFN layers (ffn_gate, ffn_up, ffn_down) versus MoE expert layers (ffn_gate_exps, ffn_up_exps, ffn_down_exps).

The assistant had previously written a manual mapping in the patched gguf_loader.py that maps ffn_gate_exps.weight → experts.0.gate_proj.weight for layers 3-77. But this mapping explicitly only covered the MoE layers. The critical question was: what happens with layers 0-2's FFN weights? If the mapping for dense layers was broken, the first three layers of the model would receive incorrect or uninitialized weights, which would explain the garbage output — the model's very first computations would be corrupted.

The assistant's reasoning shows a clear understanding of the two-layer architecture of the GGUF loading pipeline. First, the patched gguf_loader.py provides a manual override mapping for the MoE-specific tensors. Second, the gguf-py library provides an automatic name map via get_tensor_name_map() for the glm-dsa architecture. The assistant assumed that this auto map would handle the dense layers (0-2) correctly, since those use standard tensor names that any FFN-supporting architecture should recognize. The message was written to verify this assumption.

The Thinking Process Visible in the Reasoning

The assistant's reasoning unfolds in a logical chain:

Observation: The manual mapping covers MoE layers 3-77, but not dense layers 0-2.
Hypothesis: The auto name map from gguf-py should handle the dense layers automatically.
Verification plan: Run a Python script that uses gguf-py's get_tensor_name_map() to check what HF names the GGUF tensor names map to for the glm-dsa architecture.
Execution: The bash command is carefully constructed to iterate through gguf-py's MODEL_ARCH_NAMES, find the glm-dsa entry, create a name map for 78 layers (the full model depth), and print the mapping for the first few layers. The bash command itself reveals the assistant's thoroughness. It uses head -40 to limit output, suggesting awareness that the full mapping would be enormous. It checks layers 0-3 specifically to see both dense and MoE layers. And it uses the actual gguf-py library installed in the environment, ensuring the test reflects reality.

Assumptions Made by the Assistant

This message rests on several key assumptions, some of which turn out to be incorrect:

Assumption 1: The gguf-py library's get_tensor_name_map() for glm-dsa correctly maps dense FFN tensor names. This is a reasonable assumption — gguf-py is the reference implementation for GGUF format handling, and the glm-dsa architecture was presumably added with complete tensor mappings. However, as the next message (msg id=1916) reveals, this assumption is catastrophically wrong: the mapping returns None for every single tensor, with 0 mapped and 1809 unmapped.

Assumption 2: The model_type variable in the patched gguf_loader.py is set to "glm-dsa" which matches gguf-py's architecture key. The assistant explicitly notes this: "But the model_type is set to "glm-dsa" which is gguf-py's key." This is correct — the architecture key 73 maps to "glm-dsa" in gguf-py.

Assumption 3: The auto name map is used as a fallback when the manual override doesn't cover a tensor. This is how the GGUF loader is designed to work — it first checks the manual override map, then falls back to the auto map from gguf-py. If the auto map returns None for dense layer tensors, those weights would simply be skipped or loaded incorrectly.

Assumption 4: The dense layers (0-2) use standard tensor names like ffn_gate, ffn_up, ffn_down that any architecture-supporting GGUF implementation should recognize. This is true in the GGUF file itself, but the mapping from GGUF names to HF names is architecture-specific and must be explicitly defined.

Mistakes and Incorrect Assumptions

The most significant mistake revealed by the subsequent message is the assumption that gguf-py's glm-dsa architecture mapping is complete. The result showing 0 mapped tensors out of 1809 is devastating — it means the auto name map is essentially non-functional for this architecture. The gguf-py library may have the glm-dsa architecture registered (it appears in MODEL_ARCH_NAMES), but the actual tensor-to-name mapping is empty or incomplete.

This is a subtle but critical bug. The architecture key exists, so get_tensor_name_map() doesn't raise an error, but the returned mapping has no entries for any of the actual tensors in the model. This means that all weight loading — for both dense and MoE layers — relies entirely on the manual override map in the patched gguf_loader.py. If the manual override doesn't cover every tensor, those tensors are silently dropped.

The assistant also implicitly assumes that the manual override map is complete for the tensors it does cover. But the override only handles MoE expert weights (ffn_gate_exps, ffn_up_exps, ffn_down_exps) and the kv_b_proj reassembly. It doesn't cover attention weights, norms, embeddings, or the dense FFN weights for layers 0-2. If the auto map returns None for all of these, then the model is being initialized with random or uninitialized weights for the vast majority of its parameters.

Input Knowledge Required to Understand This Message

To fully grasp this message, one needs:

GLM-5 architecture knowledge: Understanding that GLM-5 uses a hybrid design with 3 dense layers followed by 75 MoE layers. This is an unusual architecture that complicates weight mapping.
GGUF format knowledge: Understanding that GGUF stores tensors with architecture-specific names (like blk.0.ffn_gate.weight for dense layers and blk.3.ffn_gate_exps.weight for MoE layers), and that a name mapping step converts these to HuggingFace parameter names.
vLLM's GGUF loader architecture: Knowing that gguf_loader.py has two levels of name mapping — a manual override map (for architecture-specific quirks) and a fallback to gguf-py's auto map.
The previous debugging context: Understanding that the model loads without errors but produces garbage output, and that the assistant has already ruled out GGUF dequantization kernel issues and verified that individual tensor values look reasonable.
Tensor parallelism concepts: Knowing that vLLM shards weights across GPUs using ColumnParallelLinear and RowParallelLinear, and that weight shapes must match the TP-sharded expectations.

Output Knowledge Created by This Message

This message produces a critical diagnostic test. The bash command, when executed, will reveal whether the gguf-py auto name map for glm-dsa is functional. The result — showing 0 mapped tensors — fundamentally changes the debugging direction. Instead of investigating attention backends or KV cache configurations, the assistant now knows that the entire weight loading pipeline is broken because the name mapping is missing.

This finding also explains why the model output is garbage: if the auto map returns None for every tensor, then only the tensors explicitly covered by the manual override map are loaded correctly. Everything else — embeddings, attention weights, norms, output layers — gets default initialization. The model is essentially running with random weights for most of its parameters, which would indeed produce flat log-probability distributions and incoherent output.

The message also creates a new debugging task: either fix the gguf-py name mapping for glm-dsa to include all tensor mappings, or expand the manual override map in gguf_loader.py to cover every tensor in the model. The former is more sustainable but requires modifying the gguf-py library; the latter is more expedient but risks missing tensors.

The Broader Context: A Session of Incremental Discovery

This message sits within a longer debugging session that began with a complete failure to load the GGUF model (KeyError for qweight_type), progressed through force-dequantization patches and unknown-parameter skips, and then confronted the garbage output problem. The assistant's methodology is systematic: isolate each component (dequantization kernel, name mapping, attention backend, weight sharding) and test it independently.

The beauty of this particular message is its surgical precision. Rather than randomly tweaking parameters or re-running with different configurations, the assistant identifies a specific architectural edge case — the dense-to-MoE transition in GLM-5 — and designs a targeted test to verify whether the name mapping handles it correctly. This is the hallmark of experienced debugging: forming a hypothesis based on architectural understanding, then writing the minimal test to confirm or refute it.

The message also demonstrates the importance of understanding the full software stack. The assistant doesn't just look at vLLM's code; they trace through gguf-py's MODEL_ARCH_NAMES and get_tensor_name_map() to understand how the name mapping works at the library level. This cross-library understanding is essential when debugging issues that span multiple dependencies.

Conclusion

Message 1915 is a compact but pivotal diagnostic step in a complex debugging journey. It represents the moment when the assistant's investigation pivots from high-level hypotheses about attention backends and kernel compatibility to a concrete, testable question about weight name mapping. The assumption that gguf-py's auto name map would handle dense layers correctly is reasonable but turns out to be wrong — a finding that will redirect the entire debugging effort. In the broader narrative of deploying GLM-5 on Blackwell GPUs, this message captures the transition from "something is wrong with the model output" to "the weight mapping is fundamentally broken," a critical insight that brings the solution one step closer.