The Moment of Truth: Testing the Merged GGUF File

Introduction

In the long and winding journey of deploying the GLM-5 model on vLLM, there comes a moment when all the preparation, patching, and data wrangling must face reality. Message 1660 captures exactly that moment. After downloading 431 GB of split GGUF files across ten shards, merging them into a single 402 GB file, and patching vLLM's source code to support the novel glm_moe_dsa architecture, the assistant finally runs a diagnostic test to see whether the merged file is actually readable and whether its tensor structure matches expectations.

This message is a dry run — a low-risk inspection of the GGUF file's metadata before attempting a full model load that could crash or corrupt memory. It is the first concrete validation of an entire pipeline that began with a pivot from the NVFP4 quantization path to GGUF, and it reveals a critical detail that will force a revision of the weight reassembly logic.

The Context: What Led to This Moment

To understand why message 1660 matters, one must appreciate the chain of events that preceded it. The session had been an intense multi-day effort to deploy GLM-5, a massive Mixture-of-Experts model, on an 8-GPU machine. The original NVFP4 path had been abandoned after extensive profiling revealed that KV cache FP8-to-BF16 cast overhead consumed 69% of decode time ([msg 1641]). The user decided to switch to unsloth's UD-Q4_K_XL GGUF quantization and deploy on vLLM instead.

This pivot required a complete re-engineering of the deployment pipeline. The assistant discovered that neither HuggingFace Transformers nor gguf-py supported the glm-dsa architecture, meaning vLLM could not load the GGUF file out of the box. A comprehensive patch was written for vLLM's gguf_loader.py to map the GLM-5 architecture's unique weight layout — including manually routing expert weights and reassembling the split attn_k_b and attn_v_b tensors back into the single kv_b_proj weight that vLLM's internal model code expects.

The download itself was a saga: it failed initially ([msg 1633]), was restarted using huggingface_hub.snapshot_download for reliability, and one of the ten split files (part 4) failed with a RuntimeError: Data processing error ([msg 1634]), requiring a targeted re-download. Once all ten parts were assembled, the assistant built llama-gguf-split from llama.cpp source and merged them into a single 402 GB file ([msg 1657]).

By the time we reach message 1660, the assistant has a single merged GGUF file sitting on disk, a patched vLLM codebase, and a set of assumptions about how the tensors inside that file are structured. The time has come to test those assumptions.

The Message Itself: A Careful Dry Run

The assistant begins message 1660 with a clear intent: "Let me try loading the model with vLLM. First, let me do a quick dry run to test just the weight mapping (without actually loading tensors)." This framing is important — the assistant explicitly avoids attempting a full model load, which could consume 400+ GB of GPU memory and crash if the weight mapping is wrong. Instead, it writes a small Python script that uses gguf.GGUFReader to read only the file's metadata.

The script is written inline via a heredoc and piped directly to the remote machine over SSH. It performs three targeted inspections:

  1. General file readability: It opens the GGUF file and prints the total tensor count (1809) and the first ten tensor names and shapes. This confirms the file is valid and the merge was successful.
  2. KV bias tensor inspection: It searches for any tensor whose name contains attn_k_b, attn_v_b, or attn_kv_b, and prints its type and shape. It breaks after finding the first match in layer 0 (blk.0), assuming all layers share the same structure.
  3. Expert tensor inspection: It searches for ffn_gate_exps or exp_probs_b tensors, which are specific to the GLM-5 Mixture-of-Experts architecture. The output reveals critical information. The GGUF file is readable with 1809 tensors. The first ten tensors include expected weights like output.weight (Q6_K, shape [6144, 154880]), output_norm.weight (F32), token_embd.weight (Q4_K), and importantly blk.0.attn_k_b.weight with shape [192, 512, 64] stored as Q8_0.

The Critical Discovery: Tensor Shape Mismatch

The shape [192, 512, 64] for attn_k_b is the most important finding in this message. It reveals that the GGUF file was produced by an older version of the llama.cpp conversion scripts that used the original model's n_head_kv=64 dimension, rather than the newer MQA (Multi-Query Attention) representation where n_head_kv=1.

This distinction matters enormously for the kv_b reassembly logic. The assistant had previously written a complex reassembly routine that assumed the MQA representation, where the key and value bias tensors would need to be expanded from a single head dimension back to the full 64 heads. The actual shape [192, 512, 64] tells a different story: the first dimension (192) is the hidden dimension per head (since 192 = 64 * 3 for Q, K, V projections), the second (512) is the number of key-value heads, and the third (64) is the head dimension. This is the original, un-compressed representation.

This discovery forces a revision of the kv_b reassembly logic. Instead of the complex MQA reversal that was originally planned, the assistant can now use a simpler transpose-and-concatenate approach: the attn_k_b and attn_v_b tensors can be transposed from [dim, n_head_kv, head_dim] to [n_head_kv, dim, head_dim] and then concatenated along the first axis to form the single kv_b_proj weight that vLLM expects.

Assumptions Made and Validated

The assistant made several assumptions in this message, some of which were validated and some of which were challenged:

Validated assumptions:

Input Knowledge Required

To fully understand this message, one needs knowledge of:

  1. GGUF file format: Understanding that GGUF files contain a metadata header followed by tensor data, and that GGUFReader can read metadata without loading tensor data into GPU memory.
  2. GLM-5 architecture: The model uses a grouped-query attention mechanism where the key and value bias projections are stored separately (attn_k_b and attn_v_b) rather than as a single kv_b_proj weight. This is a quirk of the GLM family that differs from the standard Transformer convention.
  3. vLLM weight loading: vLLM expects a single kv_b_proj weight for each attention layer, which means the two separate bias tensors must be reassembled during loading. The patched gguf_loader.py must perform this reassembly transparently.
  4. MQA vs. full-head representation: Multi-Query Attention compresses the key and value heads to a single head (n_head_kv=1), while the original representation uses the full number of heads (n_head_kv=64 for GLM-5). The llama.cpp conversion scripts evolved from the latter to the former, and the GGUF file in question was produced by an older version.
  5. Quantization types: The output shows tensor types like Q6_K, Q4_K, Q8_0, and F32, which are different quantization schemes used by llama.cpp's GGUF format. Understanding these is necessary to interpret the tensor metadata.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

  1. File integrity confirmed: The merged 402 GB GGUF file is valid and contains 1809 tensors. The merge operation was successful and no data corruption occurred.
  2. Tensor layout documented: The first ten tensors provide a snapshot of the model's weight structure, including the output projection, normalization layers, attention projections, and the kv bias tensors.
  3. Converter version identified: The shape [192, 512, 64] for attn_k_b definitively identifies the GGUF as being produced by an older version of the llama.cpp conversion scripts, before the MQA optimization was applied.
  4. Reassembly strategy determined: The shape information dictates the correct approach for reassembling kv_b_proj: a transpose-and-concatenate operation rather than the more complex MQA reversal that was originally planned.
  5. Expert tensor layout confirmed: The presence of ffn_gate_exps and exp_probs_b tensors confirms that the expert routing weights are stored as expected, which is necessary for the MoE architecture to function correctly.

The Thinking Process Visible in the Message

The assistant's reasoning is evident in the structure of the test script. It proceeds from the general to the specific: first confirming the file is readable, then inspecting the critical kv_b tensors, then checking the expert tensors. This is a textbook debugging approach — verify the foundation before examining the details.

The decision to break after finding the first kv_b tensor in blk.0 is a deliberate optimization. The assistant assumes that all 60+ layers of the model have identical tensor structures, so inspecting one layer is sufficient. If this assumption were wrong, the test would still catch the discrepancy because the output would show an unexpected shape for blk.0.

The assistant also demonstrates awareness of the risk involved. By explicitly stating "without actually loading tensors," it signals that this is a cautious first step. A full model load would require allocating 400+ GB of GPU memory across 8 GPUs, and a crash mid-load could leave the GPUs in an inconsistent state requiring a reset.

The choice of gguf.GGUFReader over vLLM's own loader is also telling. The assistant could have tried to use vLLM's GGUFModelLoader class, but that would trigger the full weight mapping and loading pipeline. Instead, it uses the lower-level gguf-py library to read raw metadata, bypassing all the patched code. This isolates the test to the file itself, separating file integrity concerns from code correctness concerns.

Conclusion

Message 1660 is a quiet but pivotal moment in the GLM-5 deployment effort. It is the first time the assistant touches the merged GGUF file, and the information it extracts — particularly the tensor shape [192, 512, 64] for attn_k_b — directly shapes the next phase of work. The discovery that the GGUF uses an older tensor representation forces a simplification of the kv_b reassembly logic, turning what could have been a complex bug hunt into a straightforward adjustment.

This message exemplifies the careful, incremental approach that characterizes successful infrastructure work: test early, test cheaply, and let the data guide the next steps. The dry run costs nothing but a few seconds of execution time, yet it saves hours of debugging that would follow a failed model load. It is a small message with outsized consequences.