The Weight Loading Breakthrough and the Garbage Output Crisis: Debugging GLM-5 GGUF on 8× Blackwell GPUs

Introduction

In the sprawling, multi-day effort to deploy the GLM-5 model — a 744-billion-parameter Mixture-of-Experts architecture — on eight NVIDIA RTX PRO 6000 Blackwell GPUs using GGUF quantization and vLLM, there comes a moment that every ML engineer recognizes: the moment when the model finally loads, the server announces "Application startup complete," and then the first inference request returns either a 500 error or a stream of incoherent garbage tokens. This chunk of the opencode session captures exactly that painful transition — from the triumph of getting a 402GB model to load, to the sobering realization that loading is not the same as working.

This article synthesizes the key debugging arc of this chunk: the resolution of the persistent KeyError during weight loading through force-dequantization, the discovery that the model loads but produces incoherent output, the investigation into the DeepGEMM set_stride incompatibility with PyTorch 2.10, the architectural pivot to disable sparse attention, and the final narrowing of the garbage output problem to a kv_b_proj tensor parallelism sharding mismatch. Each of these phases represents a distinct debugging methodology, and together they form a masterclass in systems-level ML infrastructure debugging.

Part I: The Weight Loading Breakthrough — Force-Dequantization Saves the Day

The chunk opens with the assistant still wrestling with a stubborn KeyError that had crashed every launch attempt. The error was deceptively simple: KeyError: 'model.layers.0.self_attn.indexer.weights_proj.qweight_type'. But behind that single line lay a subtle architectural mismatch between how the GGUF quantization format stored the model's weights and how vLLM's model code expected to receive them.

The Root Cause: A Quantization Mismatch

The debugging trail began with the assistant tracing through the error logs. The KeyError occurred in deepseek_v2.py at line 1540, inside the load_weights method. The model's params_dict — the dictionary of all registered PyTorch parameters — did not contain an entry for weights_proj.qweight_type. This was because the Indexer class (a component of GLM-5's Dynamic Sparse Attention mechanism) creates its weights_proj submodule as a ReplicatedLinear layer with quant_config=None. When quant_config is None, vLLM's model code does not register the metadata parameters like qweight_type that are needed for quantized weight handling [22].

However, the GGUF file stored weights_proj.weight as a Q4_K quantized tensor. When vLLM's gguf_quant_weights_iterator encountered this quantized tensor, it dutifully yielded both the weight data and a corresponding qweight_type metadata tensor, expecting the model to consume both. But the model had no parameter named weights_proj.qweight_type — it had never been registered — so the load_weights method raised a KeyError [23].

The assistant's investigation revealed a second instance of the same problem: the MoE routing gate (gate.weight) was also created with quant_config=None at line 256 of deepseek_v2.py. Both tensors were Q4_K in the GGUF file but expected to be unquantized (BF16 or F32) by the model code [24].

The Fix: Extending the Force-Dequantization Pattern

The assistant devised a two-pronged fix. First, in weight_utils.py, the gguf_quant_weights_iterator function needed to detect tensors whose HuggingFace names matched patterns for parameters created with quant_config=None, and force-dequantize them — just as it already did for the kv_b split tensors using sentinel suffixes __k_b and __v_b. The patterns were: *.mlp.gate.weight (the MoE routing gate) and *.self_attn.indexer.weights_proj.weight (the DSA indexer weights projection) [25].

But force-dequantizing the tensor in the iterator was only half the solution. The gguf_loader.py file also needed to know about these tensors. Specifically, it maintains an unquant_names list that tells the rest of the loading pipeline which parameters should not be expected to have quantized metadata (like qweight_type, scales, etc.). The assistant read the patch file to understand the exact structure of the unquant_names list, discovering that it was populated from the weight_type_map by selecting tensors that were already in float formats (F32, F16, BF16). But weights_proj and gate were Q4_K in the GGUF file, so they would never appear in unquant_names through this automatic process. The assistant needed to add them manually [28].

The fix was deployed across two files: weight_utils.py for the force-dequantization logic, and gguf_loader.py for the unquant_names update. After clearing stale .pyc bytecode caches — a critical step that had previously caused a patch to appear applied without actually taking effect [31] — the assistant relaunched the server.

The Moment of Triumph: The Model Loads

The launch succeeded. After 25 minutes of weight loading across 8 GPUs, the server reached "Application startup complete." The model was listed in the /v1/models endpoint. The assistant's exclamation — "THE SERVER IS UP AND RUNNING!" — captured the relief after hours of debugging [64].

But the celebration was premature. The very next test — a simple "What is 2+2?" inference request — returned a 500 Internal Server Error with the message "EngineCore encountered an issue" [65]. The model loaded, but it could not infer.

Part II: The DeepGEMM Wall — When --enforce-eager Isn't Enough

The stack trace from the failed inference pointed to sparse_attn_indexer.py:163, where fp8_paged_mqa_logits from DeepGEMM was called. The error was:

RuntimeError: set_stride is not allowed on a Tensor created from .data or .detach()

This error had been encountered before, during CUDAGraph warmup in an earlier launch attempt [43]. The assistant's initial diagnosis was that torch.compile's graph capture mechanism was incompatible with DeepGEMM's tensor manipulation. This was a reasonable hypothesis: torch.compile (and its underlying CUDAGraph mechanism) places strict constraints on tensor operations, and many CUDA kernels that manipulate tensor metadata break under these constraints.

The natural fix was --enforce-eager, a vLLM flag that disables CUDAGraph capture entirely, running the model in eager mode at the cost of some performance. The assistant killed the stuck processes, freed the GPU memory (which required multiple attempts due to zombie processes holding GPU resources), and relaunched the server with --enforce-eager. Then came the wait — approximately 25 minutes for the 402GB model to load across all 8 GPUs [60].

The Diagnostic Pivot

The result was devastating: the same set_stride error, even with --enforce-eager [69]. The assistant drew the correct conclusion: "This is NOT a torch.compile/CUDAGraph issue — it's a fundamental PyTorch 2.10 / DeepGEMM incompatibility."

This reframing was the critical intellectual contribution of this phase. The error message provided a crucial clue: set_stride is not allowed on a Tensor created from .data or .detach(). In PyTorch, .data was historically used to access a tensor's underlying storage without participating in autograd tracking. However, PyTorch 2.10 introduced stricter enforcement around tensor metadata manipulation. Specifically, calling set_stride() (or related methods like set_()) on a tensor that was created via .data or .detach() now raises an error.

DeepGEMM, a high-performance CUDA kernel library developed by DeepSeek for their Mixture-of-Experts models, uses .data internally to manipulate tensor strides for efficient memory access patterns. This pattern was common in earlier PyTorch versions but became problematic in 2.10. The fp8_paged_mqa_logits function — which computes attention logits from FP8-quantized key-value caches — apparently performs stride manipulation on tensors that were derived from .data or .detach() operations [69].

The assistant checked for a newer version of DeepGEMM that might have fixed this incompatibility, but the package did not expose version metadata. This was a dead end, but a productive one: it meant the fix would have to come from patching DeepGEMM itself or finding an alternative path.

Attempting a Surgical Patch

The assistant examined DeepGEMM's source code to understand the exact mechanism of the incompatibility. The fp8_paged_mqa_logits function was defined in deep_gemm.py, and the assistant traced through its implementation to find where .data was being used [73]. The solution was to wrap the problematic call in a torch.no_grad() context, which would prevent autograd from tracking the tensor operations and thus avoid the .data restriction [76].

The assistant also proactively patched a second function, fp8_mqa_logits, which had the same pattern [77]. Both patches were applied to the installed deep_gemm.py file on the remote machine.

But the patches did not work. The set_stride error persisted even with the torch.no_grad() wrapper [84]. The incompatibility was deeper than the assistant could fix with Python-level wrappers — it was embedded in the C++ CUDA kernel code itself.

Part III: The Architectural Pivot — Disabling Sparse Attention

With the DeepGEMM path blocked, the assistant faced a fundamental decision: continue trying to patch a C++ extension that was incompatible with the installed PyTorch version, or find an alternative way to compute attention for the GLM-5 model.

The assistant chose the latter. The GLM-5 model uses Dynamic Sparse Attention (DSA), which relies on the DSA indexer to select top-k tokens for sparse attention computation. The indexer calls DeepGEMM's fp8_paged_mqa_logits to compute attention logits. If the indexer could be disabled, the model would fall back to dense attention — computationally more expensive but functionally equivalent [86].

The DSA Indexer Architecture

The assistant read the model code to understand how the DSA indexer was integrated. The GlmMoeDsaForCausalLM class in deepseek_v2.py contained an Indexer module that computed top-k indices for sparse attention. The indexer was called during the forward pass of each attention layer, and its output determined which KV cache slots were accessed [88].

Disabling the indexer was not straightforward. The model's configuration included an index_topk attribute that controlled the number of top-k tokens selected. Simply setting this to zero or a large value might not work because the indexer's weights were still loaded and its forward method was still called. The assistant needed to modify the model code to bypass the indexer entirely [89].

The Config Serialization Trap

The assistant's first attempt to disable the indexer involved modifying the model's configuration to remove the index_topk attribute. This was done in the gguf_loader.py file, where the model config was constructed from GGUF metadata. However, the fix did not take effect because the config was serialized and deserialized across process boundaries — the API server process and the worker processes had separate copies of the configuration [108].

This was a classic serialization trap: a patch applied to the config in one process was invisible to the other processes because the config was serialized (pickled) and sent over a pipe. The assistant had to ensure the config modification happened before serialization, not after [109].

The root cause was a double-call bug in the GGUF loader: the _load_gguf_weights method was being called twice, and the config modification was only happening on the second call. The first call's modified config was serialized and sent to the workers, but the second call's modification was too late [110].

The fix was to restructure the weight map pipeline so that the config modification happened once, before serialization. The assistant moved the index_topk removal into a separate method that was called during model initialization, not during weight loading [111].

The Decisive Launch

With the DSA indexer disabled and the config serialization issue resolved, the assistant launched the server again. This time, the model loaded successfully, the server started, and the health check passed. The 25-minute weight loading cycle completed without errors [115].

The assistant then sent a test inference request: "What is 2+2? Answer briefly." The server responded with generated tokens — but the output was incoherent:

,,,,,,,[v elas(l(l(l [w(l(!(l  elaselaselaselas[telas[s,$elas[s[s[s[s,$[s[s[s[s,$ IRS[s IRS,$[s[t[s,$[s[s,$[s[s[selaselas IRS IRS[s[selas WAV[such

This was not merely wrong — it was complete garbage. The token distribution appeared flat and random, as if the model was picking tokens uniformly rather than conditioning on the input. For anyone who has worked with large language models, this pattern is unmistakable: the weights are loaded, but they are wrong [117].

Part IV: The Garbage Output Investigation — Narrowing to kv_b_proj

The garbage output was a new class of problem. Unlike the KeyError (which prevented loading) or the set_stride error (which crashed inference), this was a silent corruption: the model loaded and ran, but produced useless output. The debugging techniques required were fundamentally different.

The assistant systematically investigated possible causes. First, it confirmed that the GGUF dequantization kernel works correctly on SM120 — the Blackwell GPU architecture. The dequantization kernel had been verified earlier, but the assistant double-checked to ensure it wasn't producing incorrect values.

Second, the assistant verified the weight name mapping correctness. The GGUF file uses a specific naming convention, and vLLM's name map translates these to HuggingFace parameter names. If the mapping was wrong, the wrong weights would be loaded into the wrong parameters. The assistant confirmed that 1782 out of 1809 tensors were correctly mapped, with the remaining 27 being from the nextn/MTP layer that was intentionally skipped.

Third, the assistant confirmed that FlashAttention was available and functional on SM120. The attention backend was not the issue — the model could generate tokens, just wrong ones.

The investigation narrowed to the kv_b_proj weight loading. This weight is critical for the MLA (Multi-head Latent Attention) mechanism used by GLM-5. In the GGUF file, the kv_b_proj weight is split into separate k_b and v_b tensors (a convention from llama.cpp's converter). The assistant's code reassembles them into a full [28672, 512] tensor. However, the ColumnParallelLinear layer that receives this weight expects a tensor-parallel-sharded [3584, 512] parameter — one-eighth of the full tensor for each of the 8 GPUs.

The key question was: was the weight being correctly sharded? The assistant had noted that no assertion error occurred during loading, which suggested the parameter might be materialized as an UninitializedParameter despite being in the unquantized list, or the weight loader might handle the mismatch differently than expected. The chunk ends with the assistant preparing to further investigate this sharding discrepancy.

The Debugging Methodology: Patterns Across Phases

What makes this chunk particularly instructive is the consistent debugging methodology the assistant applied across all four phases of the investigation.

Phase 1 (Weight Loading KeyError): The assistant traced the error from the crash log to the source code, identified the root cause (quantization mismatch), enumerated all instances of the pattern (two quant_config=None sites), designed a fix that extended an existing mechanism (force-dequantization), and verified the fix by reading the patched code.

Phase 2 (DeepGEMM set_stride): The assistant formed a hypothesis (CUDAGraph capture causes the error), designed an experiment (--enforce-eager), waited for the result (25 minutes), evaluated the outcome (error persisted), and pivoted to a new hypothesis (PyTorch 2.10 / DeepGEMM incompatibility).

Phase 3 (Disabling DSA): The assistant researched the architecture, identified the dependency chain (indexer → DeepGEMM), considered alternatives (patch DeepGEMM vs. disable indexer), chose the architectural pivot, and debugged the config serialization trap that prevented the fix from taking effect.

Phase 4 (Garbage Output): The assistant systematically eliminated possible causes (dequantization kernel, name mapping, FlashAttention) and narrowed to the most likely culprit (kv_b_proj TP sharding).

Each phase demonstrates a different aspect of disciplined debugging: trace-based root cause analysis, hypothesis testing with controlled experiments, architectural reasoning about dependencies, and systematic elimination of variables.

The Broader Significance

This chunk illustrates the fundamental challenge of deploying cutting-edge AI models on new hardware. The GLM-5 model, with its novel DSA architecture and MLA attention, pushes against the boundaries of what vLLM's GGUF loader supports. The Blackwell GPUs (SM120 architecture) are so new that most CUDA kernels lack native support. DeepGEMM, a dependency of the DSA indexer, is incompatible with PyTorch 2.10. Each component works in isolation, but getting them all to work together is a systems integration challenge of the highest order.

The assistant's approach — systematic, pattern-seeking, building on prior solutions — is exactly the methodology required for such integration work. Rather than treating each error as an isolated incident, the assistant looks for the underlying structure, the recurring pattern that, once fixed, eliminates a whole class of failures.

The chunk also reveals a crucial truth about ML infrastructure debugging: the most valuable diagnostic step is often the one that eliminates a plausible but incorrect hypothesis. The --enforce-eager test cost 25 minutes of model loading time, but it saved potentially hours of chasing CUDAGraph-related workarounds that would never have worked. The config serialization trap cost multiple launch attempts, but understanding it prevented similar issues in the future.

Conclusion

This chunk of the opencode session captures the journey from a persistent weight loading crash to a model that loads but produces garbage — a journey that spans four distinct debugging phases, each with its own methodology and lessons. The force-dequantization fix resolved the KeyError and allowed the model to load. The DeepGEMM incompatibility forced an architectural pivot to disable sparse attention. The config serialization trap required understanding vLLM's process architecture. And the garbage output narrowed to a tensor parallelism sharding mismatch that remained unresolved at the chunk's end.

The model loads. The server serves. But the model does not reason. That final gap — between "it loads" and "it works" — is the hardest to close, and the chunk ends with the assistant still working to bridge it. The debugging continues, but the patterns established in this chunk — methodical tracing, hypothesis testing, architectural reasoning, and systematic elimination — will guide the investigation forward.