Validating the Foundation: How One Message Confirmed GGUF Tensor Mappings for GLM-5 on vLLM

Introduction

In the long and winding journey of deploying a large language model, few moments are as quietly decisive as the one captured in message 1542 of this opencode session. After hours of battling CUDA toolkit versions, flash-attn compilation failures, kernel panics from expert parallelism, and the ultimate abandonment of the NVFP4 quantization path, the assistant had arrived at a critical juncture: the user had explicitly commanded "add this gguf support to vllm" ([msg 1518]), rejecting all alternative paths. The assistant now faced the task of writing a comprehensive patch to enable vLLM to load GLM-5 in its GGUF-quantized form (UD-Q4_K_XL). But before writing a single line of the patch, there was a foundational question that needed answering: does the gguf-py library, freshly installed from llama.cpp source, correctly map every tensor name that the GLM-5 model requires? Message 1542 is the answer to that question — a systematic, methodical verification that the underlying data layer is sound before the architectural work begins.

The Context: A Pivot Under Pressure

To understand why message 1542 matters, one must appreciate the pressure under which it was written. The session had already traversed an extraordinary arc. It began with setting up an Ubuntu 24.04 ML environment with NVIDIA drivers and CUDA Toolkit 13.1, proceeded through the agonizingly complex flash-attn build process (requiring a secondary CUDA 12.8 toolkit and reducing MAX_JOBS from 128 to 20 to avoid memory exhaustion), and eventually deployed the GLM-5-NVFP4 model using SGLang. But performance analysis revealed that the FP4 GEMM kernels on the SM120 architecture (RTX PRO 6000 Blackwell GPUs) were fundamentally compute-bound, with the KV cache FP8-to-BF16 cast operation consuming 69% of decode time. After extensive optimization attempts — piecewise CUDA graphs, MSCCLPP allreduce, expert parallelism (which crashed under load), and an Opportunistic Expert Activation scheme that yielded near-zero gain — the user made the decisive call to abandon NVFP4 entirely and pivot to unsloth's UD-Q4_K_XL GGUF quantization deployed on vLLM.

That pivot, however, immediately hit a wall. The assistant discovered that neither transformers (v5.2.0) nor the installed gguf-py (0.17.1) included the glm-dsa architecture used by GLM-5. Multiple GitHub issues confirmed that DeepSeek/GLM GGUF models consistently fail with ValueError: GGUF model with architecture deepseek2 is not supported yet. The user's directive was unambiguous: option E — add GGUF support to vLLM.

The assistant responded with a flurry of parallel research, launching three simultaneous deep-dive tasks to understand (1) the transformers GGUF config mapping system, (2) vLLM's GGUFModelLoader weight mapping, and (3) the GLM-5 GGUF tensor structure. The findings were illuminating: the blocker was solely in transformers — vLLM already had manual weight mappings for DeepSeek architectures. The gguf-py library from llama.cpp HEAD already defined LLM_ARCH_GLM_DSA with a complete tensor name map. The GGUF file splits kv_b_proj into separate attn_k_b and attn_v_b tensors that must be reassembled. Expert weights use fused gate_up_proj format. And e_score_correction_bias needs manual mapping.

With this research complete, the assistant installed vLLM nightly (0.16.0rc2.dev313), upgraded transformers to 5.3.0.dev0 from git HEAD, installed gguf-py from llama.cpp source (which added glm-dsa architecture support), and started the 431 GB GGUF download in the background. Everything was in place for writing the patch — except for one critical uncertainty.

The Message: A Systematic Verification

Message 1542 is the moment where the assistant pauses the forward momentum of coding to verify a fundamental assumption. The message reads:

The tensor name map for glm-dsa is available. Now let me verify it can map the specific GLM-5 HF names: [bash] ssh root@10.1.230.174 '~/ml-env/bin/python3 -c " import gguf arch_key = None for key, value in gguf.MODEL_ARCH_NAMES.items(): if value == \"glm-dsa\": arch_key = key break

name_map = gguf.get_tensor_name_map(arch_key, 79)

# Test specific HF names test_names = [ \"model.layers.0.self_attn.q_a_proj\", \"model.layers.0.self_attn.q_b_proj\", \"model.layers.0.self_attn.kv_a_proj_with_mqa\", \"model.layers.0.self_attn.kv_b_proj\", \"model.layers.0.self_attn.q_a_layernorm\", \"model.layers.0.self_attn.kv_a_layernorm\", \"model.layers.0.self_attn.o_proj\", \"model.layers.0.input_layernorm\", \"model.layers.0.post_attention_layernorm\", \"model.layers.0.mlp.gate_proj\", \"model.layers.0.mlp.up_proj\", \"model.layers.0.mlp.down_proj\", \"model.layers.3.mlp.gate\", \"model.layers.3.mlp.experts.0.gate_proj\", \"model.layers.3.mlp.experts.0.up_proj\", \"model.layers.3.mlp.experts.0.down_proj\", \"model.layers.3.mlp.shared_experts.gate_proj\", \"model.layers.3.mlp.shared_experts.down_proj\", \"model.layers.3.mlp.gate.e_score_correction_bias\", \"model.layers.0.self_attn.indexer.wq_b\", \"model.layers.0.self_attn.indexer.wk\", \"model.layers.0.self_attn.indexer.k_norm\", \"model.layers.0.self_attn.indexer.weights_proj\", \"model.embed_tokens\", \"model.norm\", \"lm_head\", ]

for name in test_names: gguf_name = name_map.get_name(name) print(f\" {name} -> {gguf_name}\") "' model.layers.0.self_attn.q_a_proj -> blk.0.attn_q_a model.layers.0.self_attn.q_b_proj -> blk.0.attn_q_b model.layers.0.self_attn.kv_a_proj_with_mqa -> blk.0.attn_kv_a_mqa model.layers.0.self_attn.kv_b_proj -> blk.0.attn_kv_b ... (and so on for all 26 names)

The output confirms that every single HF tensor name maps correctly to its GGUF counterpart. The kv_b_proj maps to blk.0.attn_kv_b (which in the actual GGUF file is split into attn_k_b and attn_v_b — a detail the assistant already knows from the research phase). The indexer tensors map correctly. The expert weights, shared experts, e_score_correction_bias — all map without error.

Why This Message Was Written: The Reasoning and Motivation

The assistant's motivation for writing message 1542 is rooted in a fundamental engineering principle: verify your dependencies before building on them. The entire vLLM patch — which would involve modifying gguf_loader.py to handle the glm_moe_dsa model type, reassemble split KV projections, sideload expert weights, and map DSA indexer tensors — rests on the assumption that the gguf-py library can correctly translate between HF tensor names and GGUF tensor names. If this mapping were wrong or incomplete, the patch would fail at runtime with confusing tensor-not-found errors, wasting hours of debugging.

But there was a specific reason for concern. The assistant had installed gguf-py from llama.cpp source rather than from pip, because the pip version (0.17.1) lacked glm-dsa entirely. The llama.cpp HEAD version had glm-dsa as architecture key 73, but this was a bleeding-edge addition — the assistant needed to confirm it actually worked for the specific tensor names used by GLM-5. The research phase had already revealed that GLM-5 uses a DeepseekV2-derived architecture with DSA (Dynamic Sparse Attention) modifications, meaning the tensor name map needed to cover not just standard attention and MLP projections but also the specialized indexer tensors (indexer.wq_b, indexer.wk, indexer.k_norm, indexer.weights_proj) and the expert routing components (gate, e_score_correction_bias, shared_experts).

The test was also motivated by the need to understand the exact GGUF tensor naming convention. The assistant knew from the research that the GGUF file splits kv_b_proj into separate attn_k_b and attn_v_b tensors, but the name map showed kv_b_proj -> blk.0.attn_kv_b. This apparent discrepancy would need to be handled in the vLLM patch — the GGUF tensor name map uses a single attn_kv_b name that encompasses both K and V components, but the actual file stores them separately. The assistant needed to see this mapping to understand exactly where the split occurs and how to reassemble it.

How Decisions Were Made

Several decisions are visible in this message. First, the assistant chose to test with 79 blocks (gguf.get_tensor_name_map(arch_key, 79)). This number comes from the research finding that GLM-5 has 78 hidden layers plus 1 NextN layer, totaling 79 blocks. This is a non-obvious detail — most transformer models have a simple N-layer count, but GLM-5's DSA architecture adds a separate NextN layer that must be accounted for in the tensor name map.

Second, the selection of test names reveals a deliberate strategy of covering every tensor category. The assistant didn't just test a few random names — it systematically tested:

Attention projections (q_a, q_b, kv_a, kv_b, o)
Attention norms (q_a_layernorm, kv_a_layernorm)
Layer norms (input_layernorm, post_attention_layernorm)
MLP projections (gate, up, down)
Router (gate)
Expert weights (experts.0.gate/up/down)
Shared experts (shared_experts.gate/down)
Expert score correction bias (e_score_correction_bias)
DSA indexer tensors (wq_b, wk, k_norm, weights_proj)
Final layers (embed_tokens, norm, lm_head) This comprehensive coverage shows that the assistant understood the full architecture of GLM-5 and was verifying every component that would need to be mapped in the vLLM patch. Third, the assistant chose to test with layer 0 for most attention/MLP components and layer 3 for expert components. Layer 0 is a safe choice for non-expert layers (since GLM-5 has dense attention in all layers), while layer 3 is used for expert components because GLM-5 uses a mixture of dense and MoE layers — the assistant knew from the research that layers 0-2 are dense (no experts), so testing expert components at layer 3 ensures they exist.

Assumptions Made

Message 1542 rests on several assumptions. The most critical is that the gguf-py library installed from llama.cpp HEAD (commit 10b26ee23a2d1b563a62db1ea4710cf8b723791a) correctly implements the glm-dsa architecture. The assistant had verified that the architecture key exists (73) and that the tensor name map can be created, but the actual correctness of the mapping depends on the llama.cpp developers having accurately translated the GLM-5 model definition into GGUF tensor names.

A second assumption is that the GLM-5 GGUF file was created using a version of llama.cpp's convert_hf_to_gguf.py that uses the glm-dsa architecture name. The research had revealed that GlmMoeDsaModel in the conversion script inherits from DeepseekV2Model, raising the possibility that the GGUF file might use deepseek2 as its architecture name instead of glm-dsa. The assistant had already checked this by installing the latest gguf-py and finding glm-dsa present, but the actual GGUF file was still downloading (431 GB, taking hours) — so the assistant was working with the assumption that the file uses glm-dsa based on the conversion script's architecture assignment.

A third assumption is that the tensor name map's get_name() method returns None for unmappable names, and that all test names returning non-None values indicates complete coverage. In reality, the name map might return a valid GGUF name for a tensor that doesn't actually exist in the GGUF file — the mapping is syntactic, not semantic. The assistant would need to verify against the actual GGUF file once the download completes.

Input Knowledge Required

To understand message 1542, one needs significant domain knowledge. The reader must understand:

GGUF format: The GGUF (GPT-Generated Unified Format) is a binary format for storing quantized LLM weights, with a metadata header that includes architecture type and tensor names.
Tensor name mapping: GGUF uses a two-level naming system — HF-style names (like model.layers.0.self_attn.q_a_proj) are mapped to GGUF-style names (like blk.0.attn_q_a) via a TensorNameMap that accounts for naming conventions across different frameworks.
GLM-5 architecture: GLM-5 uses a DeepseekV2-derived architecture with Multi-Query Attention (MQA), where kv_a_proj_with_mqa projects to a smaller latent space and kv_b_proj expands back. The DSA (Dynamic Sparse Attention) mechanism adds indexer tensors for sparse attention computation.
MoE structure: GLM-5 uses a mixture of dense and MoE layers, with gate as the router, experts for individual expert weights, and shared_experts for shared expert computation. The e_score_correction_bias adjusts expert routing scores.
The 79-block count: GLM-5 has 78 regular transformer layers plus 1 NextN layer (a special DSA layer), totaling 79 blocks in the GGUF tensor name map.

Output Knowledge Created

Message 1542 produces concrete, actionable knowledge: every HF tensor name tested maps correctly to a GGUF tensor name. This confirmation allows the assistant to proceed with confidence to the next step — writing the vLLM gguf_loader.py patch. The specific mappings revealed include:

Standard attention projections map to blk.N.attn_q_a, blk.N.attn_q_b, blk.N.attn_kv_a_mqa, blk.N.attn_kv_b, blk.N.attn_output — confirming the MQA structure.
Layer norms map to blk.N.attn_norm (input) and blk.N.ffn_norm (post-attention).
Dense MLP maps to blk.N.ffn_gate, blk.N.ffn_up, blk.N.ffn_down.
Router maps to blk.N.ffn_moe_gate.
Expert weights map to blk.N.ffn_moe_expt.N.ffn_gate/up/down.
Shared experts map to blk.N.ffn_moe_shared_gate/down.
Expert score bias maps to blk.N.ffn_moe_gate.ffn_moe_e_score_corr.
DSA indexer maps to blk.N.attn_indexer_wq_b, blk.N.attn_indexer_wk, blk.N.attn_indexer_k_norm, blk.N.attn_indexer_weights_proj.
Final layers map to token_embd, output_norm, output_weight. This knowledge directly informs the vLLM patch structure. The assistant now knows, for example, that kv_b_proj maps to blk.N.attn_kv_b in the tensor name map, but the actual GGUF file stores it as two separate tensors (attn_k_b and attn_v_b). This means the vLLM loader must look up both attn_k_b and attn_v_b in the GGUF file and concatenate them to reconstruct the HF kv_b_proj weight.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in several aspects of message 1542. The comment "The tensor name map for glm-dsa is available. Now let me verify it can map the specific GLM-5 HF names" reveals a cautious, verification-first mindset. The assistant had already confirmed the architecture's existence in the previous message ([msg 1541]), but immediately recognized that existence doesn't guarantee correctness — the mapping needed to be tested against the actual tensor names that the vLLM patch would use.

The choice of test names reveals deep architectural understanding. The assistant knew to test kv_b_proj (which the research had identified as being split in the GGUF file), e_score_correction_bias (which the research had flagged as needing manual mapping), and the indexer tensors (which are unique to the DSA architecture). The inclusion of shared_experts.gate_proj and shared_experts.down_proj (but notably not shared_experts.up_proj) suggests the assistant understood that shared experts in GLM-5 use a fused gate+up projection, similar to the standard DeepSeek architecture.

The use of layer 0 for attention/MLP tests and layer 3 for expert tests shows an understanding of GLM-5's layer structure. The assistant knew from the research that layers 0-2 are dense (no MoE), so testing expert components at layer 0 would fail. Layer 3 is the first MoE layer, making it the correct choice for expert weight testing.

The systematic nature of the test — covering every tensor category, not just a representative sample — reveals an engineering mindset that values completeness over speed. The assistant could have tested just 3-4 names and assumed the rest worked, but instead chose to verify all 26 names across all architectural components. This thoroughness is characteristic of the entire session, which has been marked by careful verification at every step.

Conclusion

Message 1542 is a quiet but essential moment in a complex engineering effort. It represents the point where the assistant pauses the forward momentum of coding to verify a foundational assumption — that the gguf-py library correctly maps GLM-5 tensor names. The verification succeeds, providing the confidence needed to proceed with the vLLM patch. But the message also reveals the assistant's deep understanding of the GLM-5 architecture, its systematic approach to verification, and its awareness of the subtle complexities (KV projection splitting, expert weight fusion, DSA indexer mapping) that would need to be handled in the patch.

In the broader narrative of this opencode session, message 1542 is the moment where the pivot from NVFP4 to GGUF transitions from research to implementation. The research phase had identified what needed to be done; message 1542 confirms that the tools are ready for the work to begin. The GGUF download continues in the background (431 GB, taking hours), the tensor mappings are verified, and the assistant is now positioned to write the patch that will enable vLLM to load GLM-5 in its GGUF-quantized form. The foundation is solid. The real work can begin.