The Checkpoint Message: Verifying a Radical Pivot in the GLM-5 GGUF Deployment

[bash] sleep 30 && ssh root@10.1.230.174 'grep -i "Disabling DSA\|attention backend\|TRITON_MLA\|Starting to load" /tmp/vllm_serve3.log | head -10' 2>&1
(Worker_TP0 pid=48505) INFO 02-20 01:31:46 [gpu_model_runner.py:4129] Starting to load model /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf...

At first glance, message 1878 in this opencode session appears almost trivial — a single bash command that sleeps for 30 seconds, greps a log file for a handful of patterns, and returns a single line of output confirming that the model has begun loading. But this message sits at a critical inflection point in a long and arduous debugging session. It is the moment of verification after a radical architectural decision: the deliberate disabling of the GLM-5 model's core sparse attention mechanism — the Directed Sparse Attention (DSA) indexer — in order to work around a fundamental incompatibility between the DeepGEMM C++ extension and PyTorch 2.10.

The Crisis That Led to This Moment

To understand why this message was written, one must appreciate the debugging nightmare that preceded it. The assistant had been working for hours to deploy the GLM-5 model — a massive 402 GB GGUF-quantized Mixture-of-Experts language model — across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using vLLM. The deployment had already survived numerous crises: CUDA toolkit version mismatches, flash-attn compilation failures, tensor parallelism sharding bugs, and a custom Triton sparse attention backend written specifically for the Blackwell SM120 architecture.

But the most stubborn problem emerged when the server finally started accepting requests, only to crash with an inscrutable error: set_stride is not allowed on a tensor created outside of PyTorch's automatic differentiation. This error originated deep inside DeepGEMM's fp8_paged_mqa_logits C++ extension — compiled code that the assistant could not easily modify. The root cause was a PyTorch 2.10 safety feature that restricted stride manipulation on tensors created outside of PyTorch's autograd system, even within torch.no_grad() blocks. The C++ extension was fundamentally incompatible with the installed PyTorch version.

The assistant explored several avenues: upgrading or recompiling DeepGEMM, searching for environment variables to relax the restriction, wrapping the call in torch.no_grad() (which had already been tried and failed). None worked. The incompatibility was baked into the compiled .so extension file.

The Radical Decision: Disabling Sparse Attention

Faced with an immovable obstacle, the assistant made a bold decision: disable the DSA indexer entirely. The GLM-5 model uses a sophisticated sparse attention mechanism where, for each token, an "indexer" module selects the top-K most relevant tokens from the KV cache, and attention is computed only over those tokens. This is a core architectural feature of the DeepSeek V3.2 / GLM-5 family. But the indexer's forward pass calls fp8_paged_mqa_logits from DeepGEMM — the very function that crashes.

The insight was that the model could fall back to dense (full) attention, where every token attends to every other token in the context window. This is computationally more expensive for long sequences, but it would bypass the broken C++ kernel entirely. The assistant identified that vLLM's codebase checks hasattr(config, "index_topk") to determine whether the model is a "v3.2" sparse model. By removing the index_topk attribute from the HuggingFace configuration before model initialization, the entire sparse attention infrastructure — the indexer buffers, the indexer weights, the SparseAttnIndexer module, and the custom TRITON_MLA_SPARSE backend — would never be created.

This was not a configuration tweak; it was a fundamental alteration of how the model processes information. The assistant was essentially telling an 8-GPU, 402-billion-parameter model to ignore one of its defining architectural innovations.

The Patches Required

Disabling the indexer was not a one-line change. The assistant had to modify three separate components of vLLM:

  1. gguf_loader.py: A line was added to delete index_topk from the HuggingFace config object (del hf_config.index_topk) right before model initialization. A warning message was also added to log the decision.
  2. deepseek_v2.py: The load_weights method contained a line param = params_dict[name] that would raise a KeyError if the GGUF file contained tensors (like the indexer's weights_proj.qweight_type) that had no corresponding model parameter. The assistant added a if name not in params_dict: continue guard to silently skip these orphaned weights.
  3. The attention backend selection: With is_v32=False, vLLM would automatically select the standard TRITON_MLA backend instead of the custom TRITON_MLA_SPARSE backend. This was actually beneficial — the standard backend was already confirmed to work on Blackwell SM120 GPUs. These patches were deployed in messages 1874-1876, and then the server was relaunched in message 1877.

What Message 1878 Actually Checks

The bash command in message 1878 serves as a verification probe. After relaunching the server, the assistant waits 30 seconds (enough time for the Python process to start and begin initialization) and then greps the log file for four specific patterns:

Assumptions and Risks

The assistant made several assumptions in this approach:

  1. That dense attention would produce coherent output. This was far from guaranteed. The GLM-5 model was trained with sparse attention, and the indexer's top-K selection is deeply integrated into the attention computation. Removing it could produce garbage output, but it was the only path forward given the DeepGEMM incompatibility.
  2. That the standard TRITON_MLA backend would handle the full attention computation correctly. The assistant had previously confirmed this backend works on SM120, but it had only been tested with the sparse backend's weight layout. The dense backend might expect different tensor shapes or weight arrangements.
  3. That skipping unknown weights during loading would not cause downstream errors. The indexer weights (about 2-3 MB per layer across 60+ layers) would simply be ignored. If any other part of the code expected these weights to be present, it would fail.
  4. That PyTorch 2.10's set_stride restriction was truly the only issue. There could be other incompatibilities between DeepGEMM and PyTorch 2.10 that would surface later.

The Outcome

The next message (msg 1879) would reveal that the "Disabling DSA" warning did appear on all 8 workers, confirming the patch was active. The model would load to completion. However, the story does not end there — the generated output would later be discovered to be incoherent (garbage tokens with flat log-prob distributions), leading to yet another round of debugging focused on the kv_b_proj tensor parallelism sharding mismatch. The dense attention workaround solved the crash but introduced a new problem: the model's weights, originally trained for sparse attention, did not produce meaningful output when run in dense mode without the indexer.

Input and Output Knowledge

To understand this message, the reader needs knowledge of: the GLM-5 model architecture (Mixture-of-Experts with DSA sparse attention), the vLLM inference engine's model loading pipeline, the DeepGEMM library's role in FP8 attention computations, PyTorch 2.10's tensor safety restrictions, the GGUF quantization format, and the concept of tensor parallelism across multiple GPUs. The message itself creates new knowledge: it confirms that the patches were syntactically correct and that the server initialization proceeds past the critical config-loading stage. It does not yet confirm that inference works — that verification would come later.

The Thinking Process

The reasoning visible in this message is diagnostic and incremental. The assistant does not check for success immediately; it waits 30 seconds to give the server time to initialize. It does not check a single pattern but four distinct ones, each corresponding to a different stage of the initialization pipeline. The choice of patterns reveals the assistant's mental model of the failure modes: the patch might not have been deployed correctly ("Disabling DSA"), the attention backend might have selected incorrectly ("attention backend" / "TRITON_MLA"), or the entire process might have crashed before loading ("Starting to load"). The single returned line — "Starting to load model" — is the minimum viable signal that the experiment is proceeding. It is not a victory, but it is permission to continue waiting.