Chunk 15.0

In this chunk, the assistant continued debugging the GLM-5 GGUF deployment on 8× Blackwell GPUs. The immediate task was to resolve the `KeyError: 'model.layers.0.self_attn.indexer.weights_proj.qweight_type'` from the previous run. The root cause was that the model’s `Indexer` creates `weights_proj` with `quant_config=None`, but the GGUF file stores it as Q4_K, so the weight iterator incorrectly yielded a `qweight_type` tensor that had no corresponding parameter. The assistant fixed this by force‑dequantizing tensors whose model parameters are created with `quant_config=None` (e.g., `weights_proj` and the MoE routing `gate`), and adding a skip in `load_weights` for unknown parameter names. After deploying these patches, the model loaded fully and the server started serving requests. However, the generated output was incoherent (garbage tokens with flat log‑prob distributions). The assistant systematically investigated possible causes: the GGUF dequantization kernel works correctly on SM120, the weight name mapping is correct, and vLLM’s bundled FlashAttention is available. The focus then turned to the `kv_b_proj` weight loading: the weight is reassembled from `k_b` and `v_b` as a full `[28672, 512]` tensor, but the `ColumnParallelLinear` expects a TP‑sharded `[3584, 512]` parameter. No assertion error occurred, suggesting the parameter might be materialized as `UninitializedParameter` despite being in the unquantized list, or the weight loader handles the mismatch differently than expected. The assistant was preparing to further investigate this sharding discrepancy when the chunk ended. Themes in this chunk include persistent, methodical debugging of weight loading and model initialization, patching multiple vLLM components (gguf_loader, weight_utils, deepseek_v2 model) to work around incompatibilities, and the challenge of integrating GGUF‑quantized weights with vLLM’s tensor‑parallel and attention backends on a new GPU architecture (SM120 Blackwell). The assistant’s approach is to isolate each component (dequantization, mapping, flash attention, weight sharding) to find the root cause of the garbage output.

The Weight Loading Breakthrough and the Garbage Output Crisis: Debugging GLM-5 GGUF on 8× Blackwell GPUs 2808 words

Message Articles