The Battle Plan: How a 744B-Parameter Model Was Wrangled onto 8 Blackwell GPUs

Introduction

In the long and winding history of deploying large language models onto exotic hardware, few moments are as revealing as the status report. After dozens of tool calls, multiple kernel panics, a kernel upgrade, three separate attention backends, and at least two critical bugs discovered in upstream code, the assistant in this opencode session produced message 1775 — a sprawling, meticulously structured document that is equal parts after-action report, technical specification, battle plan, and inventory manifest. It is not a message that does anything. It is a message that knows everything the assistant knows at this moment, laid bare for the user (and for itself) to see.

This article examines message 1775 in depth: why it was written, what it reveals about the assistant's reasoning process, the assumptions it encodes, the knowledge it synthesizes, and the strategic picture it paints of a project at the knife's edge between success and yet another obscure error. The message is a snapshot of a mind (artificial though it may be) taking stock before the next push — and in that snapshot, we can see the entire shape of the deployment challenge.

The Context: A Project at War with Itself

To understand message 1775, one must first understand the project that produced it. The goal was brutally simple on its face: deploy GLM-5, a 744-billion-parameter mixture-of-experts (MoE) language model, on a server with 8× NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs, using GGUF UD-Q4_K_XL quantization served through vLLM. But beneath that simple goal lay a labyrinth of incompatibilities, missing features, and outright bugs.

The hardware was bleeding-edge: Blackwell GPUs with SM120 compute capability (compute capability 12.0), an architecture so new that most of the CUDA ecosystem had not yet caught up. The model was unusual: GLM-5 uses a "glm_moe_dsa" architecture with Multi-head Latent Attention (MLA) and a Dynamic Sparse Attention (DSA) indexer — a combination that existed in no pre-built inference engine. The quantization format was GGUF, which vLLM supported only partially, and only for a handful of architectures. The attention mechanism required a sparse MLA backend, and no existing backend supported SM120 + sparse attention + the specific head dimensions used by GLM-5.

The assistant had already spent many rounds fighting through these problems. It had patched vLLM's GGUF loader to support the GLM-5 architecture. It had discovered and fixed a critical bug where the kv_b_proj weight was silently not loaded from GGUF files — a bug that would have caused silent corruption of every single attention computation. It had found and fixed another bug where vLLM's weight renaming logic used a naive global string replacement (name.replace("weight", "qweight")) that corrupted parameter names containing "weight" as a substring (turning weights_proj into qweights_proj). It had written an entirely new Triton-based sparse MLA attention backend from scratch because no existing backend worked on SM120. It had tuned the system at every level: kernel boot parameters, sysctl settings, PCIe MaxReadReq, NVIDIA UVM configuration.

And still, the model would not load.

The most recent launch attempt had crashed during weight loading. The backend had been successfully selected — TRITON_MLA_SPARSE was now in the rotation — but something in the weight loading pipeline was still failing. Message 1775 was written at precisely this moment: the moment when the assistant had done everything it could think of, had fixed every bug it had found, and needed to take stock before the next push into the unknown.

Why This Message Was Written: The Strategic Pause

Message 1775 is not a typical assistant response. It does not issue tool calls. It does not analyze output. It does not propose a single next action. Instead, it is a comprehensive state dump — a document that the assistant wrote to consolidate its understanding of the entire project.

There are several reasons why such a message was necessary at this point in the conversation.

First, the conversation had become extraordinarily complex. By message 1775, the session had spanned dozens of rounds across multiple segments. The assistant had modified six different files in vLLM's source tree, created one new file, deployed patches via SCP, rebuilt CUDA kernels, upgraded the Linux kernel, and debugged crashes across multiple processes (API server, worker processes, GPU kernels). The sheer volume of context — what had been tried, what had failed, what had succeeded, what was still pending — had grown beyond what could be held in working memory. A written consolidation was essential.

Second, the assistant was at a natural decision point. The most recent launch had failed, but the error details were not fully captured. Before diving into yet another debugging cycle, the assistant needed to ask itself: what do we know? What have we fixed? What could still be wrong? The message serves as a reasoning scaffold — a way to systematically enumerate all possible failure modes before picking the next target.

Third, the message functions as a shared artifact between the assistant and the user. The user had been deeply engaged throughout the session, making strategic decisions (e.g., "patch vLLM rather than switch to llama.cpp," "patch Triton MLA rather than bypass DSA"). The status document allows the user to verify that their intent has been correctly understood and executed, and to correct course if the assistant has gone down a wrong path. It is a communication tool as much as a reasoning tool.

Fourth, the message is a form of metacognitive self-correction. By writing down everything — the bugs found, the fixes applied, the assumptions made — the assistant can check for inconsistencies. For example, the message notes that weights_proj is created with quant_config=None in the model code but is stored as Q4_K in the GGUF file. This tension between what the model expects and what the file provides is explicitly flagged as a "potential next blocker." Writing it down makes it visible and actionable.

The Structure of Knowledge: What the Message Contains

Message 1775 is organized into clearly labeled sections, each serving a distinct epistemic purpose. Let us walk through them.

Goal and Instructions

The opening section restates the project goal and, crucially, the constraints that the user has imposed. These are not technical constraints — they are strategic choices that the user has made and that the assistant must respect:

• The user explicitly rejected llama.cpp ("it's not an inference engine") in favor of a proper serving engine with continuous batching.
• The user chose to patch Triton MLA to support sparse attention rather than bypass DSA, build from source, or switch to SGLang.
• The user encouraged deep code modifications ("think big and don't be afraid to fork/modify code").
• The user wants legitimate throughput improvements, not benchmark gaming. These instructions are not merely decorative. They form the decision boundary for every technical choice the assistant makes. When the assistant later considers "may need to disable indexer or provide fallback" as a potential next blocker, it is operating within the constraint that the user wants the DSA indexer to work, not to be bypassed. The instructions section anchors the entire document to the user's strategic intent.

Discoveries: The Hardware and Software Landscape

The "Discoveries" section is a remarkable piece of systems archaeology. It catalogs everything the assistant has learned about the deployment environment:

• Hardware: 8× Blackwell GPUs (SM120, ~96GB VRAM each, 768GB total, 600W TDP), AMD EPYC 9335 (2 sockets, 32 cores each + SMT = 128 threads), ~516GB RAM across 2 NUMA nodes, no NVLink (all inter-GPU communication is PCIe Gen5). The absence of NVLink is a critical performance constraint — it means that tensor-parallel communication must go over PCIe, which will be a bottleneck for any operation that requires frequent all-reduce across GPUs.
• Software versions: vLLM nightly (0.16.0rc2.dev313), transformers from git HEAD (5.3.0.dev0), gguf-py 0.17.1 from llama.cpp git HEAD, PyTorch 2.10.0, Triton 3.6.0, CUDA 12.8, NVIDIA Driver 590.48.01, Kernel 6.14.11-5-bpo12-pve. Each version number tells a story of deliberate choice — the assistant is running bleeding-edge software across the board because stable releases do not support the required hardware/software combination.
• Critical bugs found and fixed: The message documents two bugs in vLLM's GGUF loading pipeline that the assistant discovered and patched. The kv_b_proj bug is particularly insidious: llama.cpp's converter splits the kv_b_proj weight into separate attn_k_b and attn_v_b tensors, but vLLM's GGUF loader expected a single attn_kv_b tensor. The weight was simply never loaded — silently absent, causing every attention computation to use uninitialized (or zero-initialized) data. The second bug, the global weight → qweight replacement, is a classic string-processing error: name.replace("weight", "qweight") replaces all occurrences, so weights_proj.weight becomes qweights_proj.qweight, which does not match any parameter name in the model.
• SM120 MLA attention backend problem: This section enumerates every MLA attention backend in vLLM and explains why each one fails on SM120 with GLM-5's specific configuration. It is a complete impossibility proof: no existing backend can do the job. The assistant's solution — a new TritonMLASparseBackend — is presented as the only viable path forward.
• Sparse MLA architecture understanding: The assistant documents its understanding of how sparse MLA differs from dense MLA. In dense MLA, prefill tokens go through forward_mha() and decode tokens go through forward_mqa(). In sparse MLA, all tokens go through forward_mqa() with top-k indices from the DSA indexer. The indexer produces logical token positions, which are then converted to global physical cache slots via triton_convert_req_index_to_global_index(). The physical indices are used like a virtual block table with page_size=1. This section is not just documentation — it is the assistant's theory of the system. It encodes what the assistant believes to be true about how the software and hardware interact. Any of these beliefs could be wrong, and the message's value lies partly in making them explicit so they can be tested.

Accomplished and Not Yet Done

The "Accomplished" section is a victory lap — but a cautious one. Each completed item is marked with ✅, and the list is substantial: full hardware/software audit, kernel upgrade, GGUF download and merge, five separate code patches deployed, a new attention backend written and registered, backend selection verified working.

But the "In Progress" section is where the tension lives. The latest launch crashed. The backend was selected (TRITON_MLA_SPARSE), which is progress, but the weight loading failed. The error details are in /tmp/vllm_serve2.log, which the assistant has not yet fully read. The message is honest about this uncertainty: "the error details weren't fully retrieved before the conversation ended."

The "Not Yet Done" list is a roadmap: debug the remaining error, test full model loading, handle potential issues (DSA indexer on SM120, tensor shape mismatches, Triton kernel compilation), benchmark, update documentation.

Potential Next Blockers: The Anticipation of Failure

One of the most impressive aspects of message 1775 is the "Potential Next Blockers to Watch For" section. Here, the assistant explicitly enumerates what it expects to go wrong next:

1. DSA Indexer kernel: The SparseAttnIndexer calls fp8_paged_mqa_logits from DeepGEMM, which is a CUDA kernel that may not support SM120. This is a hard wall — if the kernel binary does not contain SM120 code paths, it will fail at runtime with a "no kernel available" error.
2. GGUF quantized weights for indexer: The indexer's wq_b and wk weights are Q4_K quantized but may need special handling with ReplicatedLinear + GGUF quant.
3. Triton MLA Sparse kernel correctness: The new backend reuses the existing Triton decode kernel by treating physical indices as a virtual block table with page_size=1. This is conceptually correct but may have edge cases.
4. weights_proj with quant_config=None: The indexer creates weights_proj as unquantized but the GGUF tensor is Q4_K. This mismatch may cause issues. This section reveals the assistant's mental model of the system as a series of layers, each with its own failure modes. The weight loading layer has already been debugged extensively. The attention backend layer has been addressed with the new Triton backend. The next layer is the DSA indexer kernel, which depends on DeepGEMM — a dependency the assistant cannot patch. The layer after that is the correctness of the attention computation itself. By anticipating these blockers, the assistant is doing something very human: it is preparing itself for the emotional and cognitive load of the next failure. When the DSA indexer kernel fails on SM120 (as it almost certainly will), the assistant will not be surprised — it will already have considered that possibility and can move immediately to mitigation strategies.

Assumptions Embedded in the Message

Every status document is built on assumptions, and message 1775 is no exception. Some of these assumptions are explicit; others are implicit in the structure of the document.

Explicit assumptions:

• The remaining error is in weight loading (not in attention backend selection, which succeeded).
• The weight → qweight suffix fix has been deployed and will resolve the qweight_types_proj corruption.
• The DSA indexer kernel will likely fail on SM120 and will need a fallback.
• The Triton MLA Sparse backend is conceptually correct (physical indices as virtual block table with page_size=1). Implicit assumptions:
• The hardware configuration is stable and will not change (no NVLink, specific NUMA topology, specific kernel parameters).
• The user's strategic preferences remain constant (no switching to llama.cpp, no bypassing DSA).
• The vLLM nightly build is the correct base to patch (rather than building from source or using a different fork).
• The GGUF file is correct and complete (the 27 unmapped tensors from the nextn/MTP layer are genuinely skippable).
• The model architecture (glm_moe_dsa) is stable and matches what the GGUF file encodes. Potentially incorrect assumptions:
• The assumption that the remaining error is "likely in weight loading" may be wrong. The crash could be in model initialization, in the attention backend registration, or in some other component that only fails after weights are loaded.
• The assumption that the DSA indexer kernel will fail on SM120 is well-founded (DeepGEMM is not known to support SM120), but the mitigation strategy is unclear. "Disable indexer or provide fallback" is vague — what would the fallback look like? Dense attention on a model designed for sparse attention would likely produce incorrect results.
• The assumption that 1782/1809 mapped tensors is sufficient may be optimistic. The 27 unmapped tensors are from the nextn/MTP layer, which the assistant has "intentionally skipped." If the model code actually references these tensors during inference, the skip will cause a runtime error.

The Knowledge That Was Required to Write This Message

Message 1775 could only have been written by someone (or something) with deep knowledge across multiple domains:

vLLM internals: The message references the GGUF loading pipeline (gguf_loader.py, weight_utils.py), the attention backend system (registry.py, cuda.py, backend.py), the model definition (deepseek_v2.py with GlmMoeDsaForCausalLM and Indexer classes), the tensor parallelism system (ColumnParallelLinear, ReplicatedLinear), and the quantization system (gguf.py). Understanding how these components interact — and where the bugs were — requires a detailed mental model of vLLM's architecture.

GGUF format: The message demonstrates knowledge of GGUF tensor naming conventions, quantization types (Q4_K, Q8_0, F32), the gguf-py name mapping system, and the relationship between HF model weights and GGUF tensors. The discovery that kv_b_proj is split into attn_k_b and attn_v_b by llama.cpp's converter is a specific piece of GGUF arcana that would be invisible to anyone who had not inspected the actual file contents.

MLA attention: The message distinguishes between dense MLA (with separate prefill and decode paths) and sparse MLA (with a unified MQA path and top-k indices from the DSA indexer). It understands the role of qk_nope_head_dim, qk_rope_head_dim, v_head_dim, and kv_lora_rank — the hyperparameters that define the MLA computation. It knows that different attention backends support different combinations of these parameters and different compute capabilities.

CUDA and GPU architecture: The message understands SM120 (compute capability 12.0) as a distinct target that existing MLA backends do not support. It knows that FlashAttention, FlashMLA, and FlashInfer have specific SM version requirements. It knows that DeepGEMM kernels may not include SM120 code paths.

System administration: The message references kernel boot parameters (iommu=pt, amd_pstate=active, processor.max_cstate=1, nmi_watchdog=0), sysctl settings, PCIe MaxReadReq tuning, and NVIDIA UVM configuration (uvm_disable_hmm=1). These are not standard knowledge — they are the result of deep system-level debugging.

The Knowledge That This Message Creates

Message 1775 is not just a record of what was known — it is itself a knowledge artifact that creates new value.

First, it creates a shared mental model. Before this message, the assistant's understanding was distributed across dozens of tool calls, file edits, and log inspections. The user, who may have been following along, had to reconstruct the state of the project from scattered evidence. Message 1775 consolidates everything into a single document that can be referenced, questioned, and built upon.

Second, it creates a prioritized action plan. The "Immediate Next Steps" section transforms the amorphous problem of "the model doesn't work" into a concrete, ordered checklist: check the log, diagnose the error, fix the weight loading, handle the DSA indexer, benchmark. Each step is a discrete unit of work with a clear success criterion.

Third, it creates an inventory of the deployment. The "Relevant Files / Directories" section is a complete map of every patched file, every log, every configuration. If the assistant were to be interrupted and resumed in a new session, this inventory would allow it to reconstruct the state of the system without re-exploring.

Fourth, it creates a record of bugs found and fixed. The two critical bugs — kv_b_proj not loaded and global weight → qweight replacement — are documented with their root causes and fixes. This is valuable not just for this project but for the broader vLLM community. The assistant has effectively done upstream bug reporting, even if the fixes are currently only deployed locally.

Fifth, it creates a theory of the remaining failure. By enumerating potential next blockers, the message generates hypotheses that can be tested. When the next error occurs, the assistant can check: is it one of the predicted blockers? If so, the mitigation strategy is already partially thought through. If not, the theory needs revision.

The Thinking Process Visible in the Message

Although message 1775 is presented as a static document, the thinking process that produced it is visible in its structure and content.

Systematic enumeration: The assistant thinks in terms of complete sets. When listing MLA backends, it enumerates all of them (FLASH_ATTN_MLA, FLASHMLA, FLASHINFER_MLA, TRITON_MLA, FLASHMLA_SPARSE, FLASHINFER_MLA_SPARSE) and evaluates each one against the same criteria (SM120 support, sparse support, qk_nope_head_dim support). This is exhaustive reasoning — the assistant is not guessing; it is checking every option.

Causal chain reasoning: The assistant traces failure causes through multiple layers. The qweight_types_proj error is traced back to name.replace("weight", "qweight") doing global replacement, which is traced back to the parameter name weights_proj containing "weight" as a substring, which is traced back to the GGUF tensor indexer.proj.weight being quantized. Each link in the chain is explicit.

Hypothesis generation: The "Potential Next Blockers" section is pure hypothesis generation. The assistant is saying: "Given what I know about the system, here is what I expect to fail next, and here is why." This is a form of predictive reasoning that anticipates problems before they occur.

Prioritization under uncertainty: The assistant acknowledges uncertainty ("the error details weren't fully retrieved") but does not let it paralyze action. Instead, it prioritizes: check the log first, then fix weight loading, then handle the DSA indexer. The order reflects both likelihood (weight loading is the most likely current failure) and dependency (the model must load before the attention backend can be tested).

Metacognitive awareness: The assistant is aware of its own knowledge boundaries. It says "may need to disable indexer or provide fallback" — not "will need to" but "may need to." It flags the weights_proj mismatch as something that "may cause issues." This hedging is not uncertainty for its own sake; it is an honest assessment of what the assistant does and does not know.

Mistakes and Incorrect Assumptions

While message 1775 is remarkably thorough, it is not infallible. Several potential issues deserve scrutiny.

The assumption that the weight loading error is the only current problem. The message treats the latest crash as a weight loading issue because the previous crash was a weight loading issue (KeyError: 'model.layers.0.self_attn.indexer.qweight_types_proj.qweight_type'). But the latest crash could be different. The backend selection succeeded, which is new — and new code paths can fail in new ways. The assistant's assumption that "the remaining issues are likely in weight loading" may be correct, but it is an extrapolation from limited data.

The assumption that the Triton MLA Sparse backend is correct. The message states that the backend "reuses the existing Triton decode kernel by treating physical indices as a virtual block table with page_size=1" and that this is "conceptually correct but may have edge cases." This is a significant architectural claim. The Triton decode kernel was designed for dense attention with block tables; using it with physical indices from the DSA indexer is a novel application that may have subtle bugs. The "edge cases" could include incorrect handling of padding tokens, out-of-bounds index accesses, or incorrect cache slot addressing.

The assumption that 27 unmapped tensors are safely skippable. The nextn/MTP (Multi-Token Prediction) layer is an auxiliary component that predicts multiple future tokens simultaneously. If the model code attempts to load weights for this layer and finds them missing, it may crash or produce incorrect output. The assistant has "intentionally skipped" these tensors, but the justification for this skip is not fully explained.

The assumption that the GGUF file is structurally sound. A 402GB file merged from 10 splits could have corruption at the split boundaries, incorrect metadata, or tensor shape mismatches. The assistant verified the name mapping (1782/1809 tensors mapped), but this does not guarantee that every tensor's data is correct or that the tensor shapes match what the model expects.

Conclusion: The Message as a Mirror

Message 1775 is, in the end, a mirror held up to the assistant's own mind. It reflects everything the assistant knows, everything it suspects, everything it fears might go wrong, and everything it plans to do next. It is a document of extraordinary thoroughness — the kind of document that a human engineer would write after weeks of debugging a complex system, capturing every lesson learned and every open question.

But it is also a document of vulnerability. The assistant does not know why the latest launch failed. It does not know if the DSA indexer will work on SM120. It does not know if the Triton MLA Sparse backend is correct. It does not know if the 27 skipped tensors will cause problems later. The message is honest about all of this uncertainty, and that honesty is its greatest strength.

In the broader narrative of the opencode session, message 1775 is the calm before the storm. It is the moment when the assistant takes a breath, surveys the battlefield, and prepares for the next engagement. The bugs that have been fixed are documented. The bugs that remain are anticipated. The path forward is laid out in numbered steps.

Whether that path leads to a successfully serving model or to yet another unexpected error is a question that the message itself cannot answer. But it has done something almost as valuable: it has made the question answerable. By consolidating knowledge, enumerating assumptions, and anticipating failures, message 1775 transforms an overwhelming tangle of problems into a structured, tractable project. That is the mark of a mature engineering mindset — whether human or artificial.