The Final Summary: Debugging a 744B-Parameter Model's Garbage Output Across Two Bugs

Introduction

In the world of large language model deployment, few things are as frustrating as watching a model load successfully, consume hundreds of gigabytes of GPU memory, and then produce nothing but gibberish. Message 1983 in this opencode session is the capstone of an intense debugging marathon — a comprehensive session summary written by the AI assistant after resolving two intertwined bugs that caused the GLM-5 model (744B parameters, MoE architecture, GGUF Q4_K_XL quantization) to produce incoherent output on a cluster of 8x NVIDIA RTX PRO 6000 Blackwell GPUs running vLLM.

This message is not a tool call, a code change, or a diagnostic command. It is a retrospective consolidation — the assistant stepping back from the trenches to document what was found, what was fixed, what was attempted, and what was ruled out. It serves as both a victory lap and a knowledge artifact for future reference. In analyzing this single message, we can uncover the full arc of a sophisticated debugging process, the assumptions that guided it, the mistakes that were corrected along the way, and the hard-won knowledge that emerged.

The Message Itself

Here is the complete text of message 1983:

Here's a complete summary of what we accomplished in this session:

>

## Session Summary

>

### Problem GLM-5 (744B MoE, GGUF Q4_K_XL quantization) loaded successfully in vLLM on 8x RTX PRO 6000 Blackwell GPUs but produced completely incoherent garbage output.

>

### Root Causes Found & Fixed

>

Bug 1: MLA Custom Op Output Buffer Disconnect (mla_attention.py) - torch.ops.vllm.unified_mla_attention_with_output custom op dispatch creates a phantom tensor internally. The forward_impl function writes to this phantom tensor, but the caller's output buffer stays all zeros. - Fix: Bypass the custom op and call forward_impl directly (same as the use_direct_call=True path), passing kv_cache and attn_metadata from ForwardContext.

>

Bug 2: GGUF Shard Ordering in Fused Layers (gguf.py) - GGUFLinearMethod.apply() iterates over shard_id in GGUF file load order. For fused_qkv_a_proj, when kv_a_proj_with_mqa (shard 1) loads before q_a_proj (shard 0), the fused output has KV columns first and Q columns second — but the MLA code assumes Q comes first. - Fix: for idx in sorted(shard_id) ensures outputs are concatenated in canonical order.

>

### CUDAGraph - Attempted but produces garbage output with this GGUF MLA model. Staying with --enforce-eager.

>

### Benchmark Results (MLA enabled, enforce-eager)

>

| Concurrency | Throughput (tok/s) | Avg Latency | P99 Latency | |-------------|-------------------|-------------|-------------| | 1 | 20.0 | 6.41s | 6.43s | | 2 | 38.2 | 6.68s | 6.70s | | 10 | 154.0 | 8.30s | 9.13s | | 64 | 292.9 | 27.85s | 29.69s | | 256 | 995.4 | 25.47s | 25.63s | | 1024 | 1049.1 | 36.35s | 36.44s |

>

- Peak throughput: ~1,050 tok/s at concurrency 1024 - Zero errors at all concurrency levels - 128 output tokens per request, ~40 input tokens per request - Good scaling from 1→1024 concurrency (52x throughput improvement)

>

### Investigations That Ruled Out Other Causes All of these were verified correct and NOT the cause: - GGUF name mapping (1782/1809 tensors mapped) - kv_b_proj reassembly (round-trip verified) - TP sharding for both quantized and unquantized weights - Q4_K block alignment with TP=8 - RoPE interleave configuration - head_dim config and MLA head size computation - Expert weight shapes and 3D tensor handling - fused_qkv_a_proj mixed quant type loading - Dequantization kernel on SM120

Why This Message Was Written: The Motivation and Context

To understand why this summary exists, we must appreciate the debugging journey that preceded it. The assistant had spent multiple rounds — visible in messages 1968 through 1982 — methodically hunting down why a correctly loaded model produced nonsense. The debugging process followed a classic scientific method: form a hypothesis, test it, rule it out or confirm it, move to the next.

The context leading into this message includes:

The Two Bugs: A Deeper Look

Bug 1: The Phantom Tensor

The first bug is a subtle interaction between vLLM's custom PyTorch operator dispatch system and the MLA attention implementation. The MLA (Multi-head Latent Attention) backend in vLLM uses a custom op registered as torch.ops.vllm.unified_mla_attention_with_output. This op wraps the actual attention computation (forward_impl) in a dispatch layer that is supposed to handle various configurations.

The bug: when use_direct_call=False (the default path), the custom op creates a new output tensor internally — a "phantom" tensor — and passes it to forward_impl. The forward_impl writes correct attention values to this phantom tensor. But the caller's output buffer, which was allocated before the call, is never written to. It remains all zeros. The phantom tensor is then discarded or not properly propagated back, so the caller sees only zeros.

This is a classic aliasing bug in a complex framework: two tensors exist that are supposed to be the same buffer, but the dispatch layer creates a copy (or a new allocation) instead of using the caller's buffer. The fix — bypassing the custom op entirely and calling forward_impl directly with the correct kv_cache and attn_metadata from ForwardContext — eliminates the indirection that caused the disconnect.

Bug 2: The Shard Ordering Problem

The second bug is in GGUFLinearMethod.apply(), the method that dequantizes and applies GGUF-quantized linear layers. GGUF files store tensors in a specific order, and for fused layers like fused_qkv_a_proj (which combines the Q, K, and V projection weights into a single operation), the file may store the KV shards before the Q shards.

The apply() method iterated over shard_id in the order they appeared in the GGUF file. When kv_a_proj_with_mqa (shard 1) was stored before q_a_proj (shard 0), the fused output tensor would have KV columns first and Q columns second. But the downstream MLA code assumes the canonical order: Q first, then KV. The result was that the attention computation received Q values where it expected KV values and vice versa, producing garbage.

The fix is elegantly simple: for idx in sorted(shard_id). By sorting the shard IDs numerically, the output columns are always concatenated in the canonical order (0, 1, 2...) regardless of the file's storage order. This one-line change corrected the column ordering for all fused layers.

The Benchmarking Results: What They Reveal

The benchmark data in the summary tells a story of its own. The model achieves ~1,050 tok/s peak throughput at concurrency 1024, scaling 52x from single-request throughput of ~20 tok/s. This is impressive scaling, but the absolute numbers reveal the hardware constraints.

The single-request throughput of 20 tok/s is relatively modest for a 744B MoE model on 8x Blackwell GPUs. Earlier in the session (segment 16's chunk summary mentions this), profiling had revealed that ~42% of time was spent on NCCL allreduce calls over PCIe — the GPUs are connected via PCIe rather than NVLink, creating a significant communication bottleneck. The assistant had explored more aggressive optimizations like custom allreduce and allreduce-RMS fusion but found them incompatible with the PCIe-only topology.

The CUDAGraph attempt is also noteworthy. CUDAGraph is a CUDA optimization that captures and replays GPU operations, eliminating CPU-side launch overhead. It doubled throughput to ~43 tok/s in earlier tests (as mentioned in the chunk summary), but in this final configuration it produced garbage output. The assistant correctly diagnosed this as an incompatibility with the custom MLA attention dispatch, and made the pragmatic decision to stay with --enforce-eager rather than chasing further CUDAGraph compatibility.

The Ruled-Out Investigations: A Methodology Lesson

One of the most valuable parts of this summary is the list of investigations that ruled out other causes. This list — nine items long — represents dozens of hypotheses that were formulated, tested, and rejected. Each item represents hours of work: reading code, writing diagnostic scripts, running experiments, analyzing results.

The list includes:

Assumptions, Mistakes, and Lessons Learned

The summary message, by its nature as a retrospective, reveals several assumptions that guided the debugging process:

Assumption 1: The weight loading code was correct. The assistant initially suspected that the garbage output was caused by incorrect weight loading — perhaps a tensor mapping error, a TP sharding bug, or a dequantization issue. Extensive effort was spent verifying the weight loading pipeline before the MLA attention backend was even considered. This was a reasonable assumption (weight loading is the most common source of such errors), but it turned out to be wrong for the first bug.

Assumption 2: The MLA attention backend was well-tested on Blackwell. The Triton MLA backend had been developed for earlier GPU architectures (SM80/SM90). The assumption that it would work correctly on SM120 (Blackwell) was implicit. The first bug — the output buffer disconnect — may have been latent on other architectures too, or it may have been specific to how Blackwell handles certain CUDA operations.

Assumption 3: CUDAGraph would work. The assistant attempted CUDAGraph optimization after the model was producing correct output. The assumption was that CUDAGraph is a transparent optimization that doesn't change computation semantics. The garbage output from CUDAGraph revealed that this assumption was wrong — the custom MLA dispatch has some interaction with CUDAGraph's capture/replay mechanism that corrupts output.

Mistake: The initial focus on weight loading. While not explicitly stated as a mistake, the debugging timeline shows that significant effort was spent on weight loading investigations before the MLA backend was examined. The breakthrough came only when VLLM_MLA_DISABLE=1 was tested. In hindsight, this should have been an earlier diagnostic step — a simple environment variable that immediately isolates the attention backend.

Mistake: The shard ordering bug survived initial review. The GGUFLinearMethod.apply() code had been written and reviewed as part of the GGUF support patches. The shard ordering issue was subtle and easy to miss — the code iterated over shard IDs in whatever order they appeared, which is normally fine for non-fused layers. It was only when fused layers with different shard ordering were introduced that the bug manifested.

Input Knowledge and Output Knowledge

Input Knowledge Required

To fully understand this message, a reader needs familiarity with:

Output Knowledge Created

This message creates several forms of knowledge:

  1. Specific bug knowledge: Two previously unknown bugs in vLLM's MLA attention backend and GGUF quantization layer are documented with their root causes and fixes. This is immediately actionable for anyone deploying similar models.
  2. Performance baselines: The benchmark results provide a reference point for GLM-5 performance on 8x Blackwell GPUs with GGUF quantization. The ~1,050 tok/s peak throughput and the scaling behavior from 1 to 1024 concurrent requests are valuable data points.
  3. Debugging methodology: The list of ruled-out investigations serves as a template for debugging similar issues. Future engineers encountering garbage output from a GGUF model on vLLM can use this list as a starting point for their own investigations.
  4. Hardware constraints documentation: The note about PCIe bottleneck and CUDAGraph incompatibility documents important hardware-software interaction constraints for this specific configuration.
  5. Confidence in the solution: By documenting both what was fixed and what was ruled out, the message provides strong evidence that the root causes have been correctly identified and addressed.

The Thinking Process: What the Summary Reveals

The summary message reveals a particular mode of thinking: systematic consolidation. The assistant is not just reporting results; it is organizing them into categories that tell a coherent story. The structure — Problem, Root Causes, CUDAGraph, Benchmarks, Ruled-Out Investigations — is a narrative arc that moves from confusion to clarity.

The inclusion of the "Ruled-Out Investigations" section is particularly telling. It shows that the assistant understands that in debugging, proving what is not wrong is as important as proving what is wrong. Each ruled-out hypothesis strengthens confidence in the actual fix. This is a mark of rigorous engineering thinking.

The decision to include the CUDAGraph failure in the summary, rather than omitting it as an unsuccessful experiment, shows a commitment to complete documentation. The assistant could have simply stated "using --enforce-eager" without explaining why. Instead, it documents the attempt and its outcome, preserving knowledge that might save future effort.

The benchmark table is presented without commentary on whether the numbers are "good" or "bad." This restraint is notable — the assistant lets the data speak for itself. The reader can infer that ~1,050 tok/s on 8x Blackwell GPUs for a 744B model is respectable but constrained by PCIe, but the summary doesn't editorialize. This is appropriate for a technical document that should remain factual.

Conclusion

Message 1983 is far more than a simple "here's what we did" summary. It is a carefully constructed knowledge artifact that captures the essence of a complex debugging session. It documents two subtle bugs, their root causes, and their fixes. It provides performance benchmarks that serve as a baseline for future work. It systematically lists and dismisses alternative hypotheses, building confidence in the solution. And it honestly reports an attempted optimization that failed, preserving that knowledge for future reference.

For anyone deploying large language models on vLLM — especially MoE models with GGUF quantization on newer GPU architectures — this message is a valuable reference. It demonstrates that even when a model loads successfully and produces output, that output may be silently wrong, and finding the cause requires methodical hypothesis testing across the entire software stack, from custom PyTorch ops to GGUF file format details to Triton kernel behavior on novel hardware.

The message also serves as a testament to the power of the opencode debugging paradigm: an AI assistant with access to bash, file editing, and task spawning can methodically work through a complex, multi-layered bug, document its findings, and produce a coherent summary — all within a single conversational session. The summary message is the final output of that process: a distillation of hours of work into a form that is immediately useful to any engineer facing similar challenges.