The Full-Stack Odyssey: Deploying, Debugging, and Optimizing Frontier LLMs Across 75 Segments on 8× Blackwell GPUs

Introduction

In the history of machine learning infrastructure engineering, few sessions capture the full breadth of the discipline as comprehensively as this one. Spanning 75 segments, thousands of messages, and dozens of distinct sub-investigations, this opencode session traces the complete lifecycle of deploying frontier large language models on cutting-edge hardware. What began as a simple request — "Install latest nvidia drivers, cuda 13, python/uv/venv/etc ML tooling" on a remote Ubuntu 24.04 machine — unfolded into a multi-week odyssey touching nearly every layer of modern ML deployment: hardware driver installation, CUDA toolkit management, Python environment configuration, compiled CUDA extension debugging, dependency resolution warfare, hypervisor topology analysis, kernel-level performance optimization, profiler-driven bottleneck discovery, custom attention backend development, speculative decoding pipeline construction, and production-grade service deployment.

The hardware platform was formidable: eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs (SM120 architecture, 96 GB each, 600W TDP), connected via PCIe Gen5 with no NVLink, hosted inside a Proxmox VE virtualized environment with LXC containers. The models deployed ranged from GLM-5-NVFP4 (744B parameters) through Kimi-K2.5 (1T parameters), MiniMax-M2.5 (230B active), Qwen3.5-397B, DeepSeek-V4-Flash, and many others. The inference engines evolved from SGLang to vLLM and back again, with custom patches applied to both.

This article synthesizes the entire journey, tracing the narrative arc from bare-metal setup through infrastructure stabilization, model deployment, performance optimization, speculative decoding, and production hardening. It examines the key decisions, the false leads, the moments of breakthrough, and the lessons learned across each phase.

Part I: The Foundation — From Bare Metal to Working Environment

The First Principles: Drivers, CUDA, and Python

The session opened with the user providing an SSH target (10.1.230.175) and a high-level specification of what needed to be built. The assistant's first move was to assess the machine — discovering it ran Ubuntu 24.04.4 LTS, had 128 CPU cores, and was named llm-one — a clear signal of its intended purpose as a large language model server. The machine initially had no NVIDIA drivers installed, and the assistant had to build the entire stack from scratch.

The driver installation proceeded through several phases. The assistant added the NVIDIA CUDA repository, installed the NVIDIA driver package (version 590.48.01), and then discovered something surprising: CUDA Toolkit 13.1 was actually available, contradicting its earlier assumption that "CUDA 13 doesn't exist yet." This moment of discovery — when the assistant's static knowledge was corrected by live system interrogation — became a recurring theme throughout the session.

The Python environment was set up using uv, a modern Rust-based Python package manager that the user specifically requested. The assistant created a virtual environment at ~/ml-env and installed PyTorch. At this point, the foundation appeared solid: NVIDIA drivers, CUDA 13.1, Python 3.12 with uv, and PyTorch with GPU support. But the foundation had a hidden crack: PyTorch's official wheels were compiled against CUDA 12.8, not the system's CUDA 13.1. This discrepancy would soon become the source of cascading failures.

The Flash-Attention Crucible

The installation of flash-attn — a CUDA-accelerated attention kernel library — transformed the session from a routine setup into a debugging epic. Flash-attn is a critical dependency for many modern LLM serving frameworks (including vLLM and SGLang), providing the fused attention kernels that make inference efficient on modern GPUs. Building it from source on a new system exposed the full complexity of the CUDA compilation toolchain.

The first attempt at a bulk package install failed because flash-attn's build system performed a strict CUDA version check. The system's CUDA 13.1 did not match PyTorch's CUDA 12.8 base, and the build refused to proceed. The assistant installed a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation, setting CUDA_HOME to point at the older version during extension builds. This dual-CUDA workaround resolved the version mismatch but introduced a split personality in the environment — the system default remained CUDA 13.1, but extension builds used CUDA 12.8.

With the CUDA version issue resolved, the assistant launched the flash-attn build — and immediately hit a second wall: memory exhaustion. The machine had 128 CPU cores, and flash-attn's build system spawned up to 128 parallel nvcc and ptxas processes. Each CUDA compiler process consumed gigabytes of RAM, and the system quickly ran out of memory, becoming unresponsive.

What followed was a remarkable collaborative debugging session between the user and the assistant. The user provided terse, precise guidance — "oom, go 32," "oom, go 20" — while the assistant executed the builds, cleaned up after failures, and reported results. The MAX_JOBS parameter was systematically reduced: 128 → 64 → 48 → 32 → 20. Each attempt required killing lingering compiler processes, dropping kernel caches, and verifying that memory had been fully reclaimed before the next try. The machine was rebooted with expanded RAM — from 288GB to 432GB — yet even then, builds at MAX_JOBS=32 consumed 84GB before being killed. Only at MAX_JOBS=20 did the build finally succeed, taking nearly 10 minutes of compilation.

The Silent Catastrophe: ABI Incompatibility

And then everything broke. The verification script revealed a silent catastrophe: flash-attn's CUDA extension failed to load with an ImportError on the compiled .so file. The cause was insidious. vLLM 0.15.1 had a dependency constraint that required PyTorch ≤ 2.9.x, and the package resolver silently downgraded PyTorch from 2.10.0 to 2.9.1. Flash-attn's compiled binary was built against PyTorch 2.10's C++ ABI, and the downgrade rendered it incompatible. The Python-level imports all succeeded — PyTorch worked, CUDA was detected — but the C++ extension could not load because the symbols it expected from PyTorch's runtime were different.

This is a particularly dangerous failure mode because it is silent. The package manager (uv/pip) treats packages as independent units and has no awareness that a compiled CUDA extension is ABI-locked to a specific PyTorch build. The downgrade happens without any warning or error. The verification script — a comprehensive import test of every package in the stack — was the only reason the problem was detected before runtime.

The assistant entered a new phase of debugging: rebuilding flash-attn against the correct PyTorch version. But the dependency resolver kept fighting back. When the assistant pinned PyTorch at 2.10.0 and rebuilt flash-attn, the torchvision package forced a downgrade back to 2.9.1. Each rebuild took ~10 minutes. The final successful build was almost anticlimactic — it completed in 4.39 seconds because uv reused cached build artifacts from a previous compilation. The verification command python3 -c "import flash_attn; print(flash_attn.__version__)" returned 2.8.3 — a clean, unambiguous success.

The final verified environment included PyTorch 2.9.1+cu128, flash-attn 2.8.3 (built for Blackwell sm_100), vLLM 0.15.1, transformers 4.57.6, and a dozen other packages — all working together. The infrastructure phase was complete.

Part II: The First Deployment — GLM-5-NVFP4 and the NaN Crisis

The SM120 Correction

With the environment stabilized, the user pivoted the session sharply: "Added 8 GPUs; Deploy glm-5 nvfp4." The machine had been upgraded from 2 to 8 GPUs — a fourfold increase in compute capacity. The focus shifted from building infrastructure to deploying a specific model: GLM-5-NVFP4, a large language model in NVIDIA's 4-bit floating-point quantization format, optimized for Blackwell architecture. The serving framework was SGLang.

The first verification step confirmed that all eight RTX PRO 6000 Blackwell GPUs (96 GB each, compute capability SM120) were visible and operational. But the first launch attempt failed immediately — not with a GPU error, but with a KeyError: 'glm_moe_dsa'. The installed Transformers library (version 4.57.1) did not recognize the model architecture used by GLM-5. This was the first critical discovery: the GLM-5-NVFP4 model uses a custom architecture (glm_moe_dsa) that was only added to Transformers in version 5.2.0.

Before the second launch could proceed, the user interjected with a crucial observation: the installed SGLang version lacked support for the SM120 compute capability of Blackwell GPUs. A specific pull request, #14311, had been merged into SGLang's main branch to fix shared memory block sizes for the RTX PRO 6000's architecture. Without this fix, attention kernels would use Hopper-sized block sizes that exceeded the Blackwell GPU's smaller shared memory capacity (100 KB versus 160+ KB on Hopper), causing crashes or silent correctness issues. The solution required cloning the SGLang repository and building from the main branch.

The NaN Debugging Spiral

The second launch succeeded in loading the model — a process that consumed significant time as the assistant monitored the download of approximately 250 GB of checkpoint shards across all eight GPUs. But the launch log contained two warnings that would prove prophetic. The first warned: "DeepGemm is enabled but the scale_fmt of checkpoint is not ue8m0. This might cause accuracy degradation on Blackwell." The second warned: "Transformers version 5.2.0 is used for model type glm_moe_dsa. If you experience issues related to RoPE parameters, they may be due to incompatibilities."

The model eventually loaded successfully across all eight GPUs, consuming approximately 63 GB per GPU for weights. CUDA graphs were captured for batch sizes up to 64. The server reported itself as running. But the first test query — "Hello! What model are you?" — returned nothing. The server had crashed with a device-side assert triggered error.

The crash log revealed a CUDA runtime assertion failure: "probability tensor contains either inf, nan or element < 0." The model was producing garbage values during decode. This was not a kernel compatibility issue — it was a numerical stability failure.

The debugging that followed spanned dozens of messages. The assistant pursued multiple hypotheses in parallel:

Hypothesis 1: Attention backend incompatibility. The assistant tried switching from flashinfer to triton attention backends, but the triton backend crashed with a different assertion about NSA KV cache FP8 incompatibility. It then tried flashmla_sparse as the NSA decode backend, which appeared to work during warmup but crashed on the first real query.

Hypothesis 2: DeepGemm scale format mismatch. The warning about DeepGemm being enabled with a non-ue8m0 scale format pointed to potential numerical corruption in the GEMM operations. The assistant forced --fp8-gemm-backend cutlass to bypass DeepGemm. The server loaded successfully and warmup passed, but the NaN crash persisted during decode.

Hypothesis 3: CUDA graph capture instability. After consulting a local research repository, the assistant learned that a previous successful NVFP4 deployment on the same hardware had used --disable-cuda-graph. The assistant launched a new attempt with this flag, along with --kv-cache-dtype auto to avoid forcing FP8 KV cache. The server loaded, but the DeepGemm and Transformers warnings persisted.

The breakthrough came when the assistant launched the server with --nsa-decode-backend trtllm --nsa-prefill-backend trtllm and received the first coherent output. The trtllm backend, backed by NVIDIA's TensorRT-LLM library, handled the SM120 architecture correctly where the flashmla-based implementations failed. The NaN crash had been defeated.

The Virtualization Revelation

With a stable server running, the assistant turned to performance benchmarking. The initial baseline showed approximately 144 output tok/s and 304 total tok/s with 32 concurrent requests. Further tuning pushed throughput to approximately 225 output tok/s and 516 total tok/s. But single-stream latency was a starkly different picture: only 11.10 output tokens per second.

The investigation into this discrepancy revealed a startling discovery: the machine was a Proxmox KVM virtual machine, and P2P (Peer-to-Peer) access was disabled across all GPU pairs. All cross-GPU NCCL transfers were going through host memory with a 13.7µs+ latency floor per small transfer. With 78 layers of all-reduce operations, this latency compounded into a significant throughput penalty.

Further investigation of the Proxmox host configuration revealed several issues: missing iommu=pt in the kernel cmdline, missing pcie=1 in the VM configuration's GPU passthrough entries, and the use of the i440fx machine type instead of q35. The assistant documented a remediation plan, but the fundamental constraint was clear: each GPU sat on its own PCIe root complex of the AMD EPYC Turin processor, and no software configuration could bridge that physical separation.

Part III: The Infrastructure War — From KVM to LXC to Bare Metal

The LXC Pivot

The user asked a question that opened a new door: "Would I get better results in proxmox LXC containers?" Unlike KVM virtual machines, LXC containers share the host kernel and access devices through bind mounts, not VFIO passthrough. There is no IOMMU translation layer between the GPUs. P2P DMA should theoretically work.

The assistant installed the NVIDIA driver directly on the Proxmox VE host — a non-trivial step as Proxmox VE is built on a custom kernel (6.8.12-9-pve). The LXC container (ID 129, named llm-two) was converted from unprivileged to privileged mode for full GPU device access. Inside the container, the NVIDIA userspace driver was installed with the --no-kernel-module flag — the critical insight being that LXC containers share the host kernel and cannot install their own kernel modules.

Then came the moment of truth. The assistant ran nvidia-smi topo -m inside the container, and the output was exactly what they had been chasing for hours: NODE connections within sockets (GPUs 0-3 on NUMA 0, GPUs 4-7 on NUMA 1) and SYS across sockets — precisely the topology needed for P2P DMA to function. The bare-metal topology was confirmed.

But CUDA refused to initialize. The cuInit() call returned error code 3 (CUDA_ERROR_NOT_INITIALIZED). After extensive debugging — checking device file permissions, kernel module types, GSP firmware, and kernel versions — the breakthrough came from a GitHub issue in the NVIDIA/open-gpu-kernel-modules repository. The root cause was the Heterogeneous Memory Management (HMM) feature of the nvidia_uvm module, which was incompatible with certain kernel configurations — including the Proxmox VE kernel. The fix was elegantly simple: set uvm_disable_hmm=1 as a module parameter for nvidia_uvm.

With CUDA working, the NCCL bandwidth test confirmed the topology improvement: same-NUMA P2P at 53.76 GB/s, cross-NUMA P2P at 40.24 GB/s — a massive improvement over the KVM VM's ~13µs latency floor.

The Kernel Upgrade and Post-Reboot Crisis

Despite the P2P breakthrough, throughput remained stuck. The assistant computed the theoretical maximum single-stream throughput — 309 tok/s — and compared it to the actual measured performance of 10.36 tok/s. This 3.4% efficiency was a shocking wake-up call. Something fundamental was wrong.

The assistant launched ten parallel exploration agents to audit every layer of the system. The findings were organized into three categories: Confirmed Good (16 items verified as optimal), P0 Actionable Fixes (high-impact runtime changes), and P1 Actionable Fixes (medium-impact changes requiring reboot). The P0 findings included a litany of misconfigurations: PCIe MaxReadReq stuck at 512 bytes instead of 4096, NUMA balancing enabled, CPU C-states allowing C2 sleep with 100μs wake latency, and the container's RLIMIT_MEMLOCK capped at ~63GB.

The runtime fixes were applied systematically, but the benchmark showed no improvement — single-stream throughput remained at 10.30 tok/s. This negative result was itself a form of knowledge: the bottleneck was not in the platform layer.

The assistant then executed a kernel upgrade from 6.8.12 to 6.14.11 — one of the most meticulously orchestrated operations in the entire session. The kernel installation revealed a critical dependency chain: DKMS auto-build for the NVIDIA driver was skipped because kernel headers for the new kernel were not installed. The assistant immediately installed the headers package and manually triggered the DKMS build. The kernel cmdline was updated to include amd_pstate=active, processor.max_cstate=1, and nmi_watchdog=0. Runtime sysctl parameters were written to /etc/sysctl.d/99-gpu-compute.conf for persistence.

The reboot brought a new crisis: CUDA initialization failed inside the LXC container. The kernel upgrade had changed the device major numbers for NVIDIA drivers — nvidia-uvm had moved from major 504 to 509, and nvidia-caps from 507 to 237. The container's cgroup rules still referenced the old numbers, silently blocking access. The fix was to stop the container, edit the configuration to replace the stale cgroup rules, and restart.

Part IV: The 69% Bottleneck — Profiler-Driven Discovery

The 86ms Gap

The optimization campaign had been running for days. Despite extensive tuning — kernel upgrades, CPU governor optimization, PCIe configuration, NCCL tuning, FlashInfer MoE autotune, and experiments with expert parallelism, CUDA graphs, and MSCCLPP allreduce — single-stream decode throughput remained stubbornly stuck at approximately 10.5 tokens per second, corresponding to a time-per-output-token (TPOT) of roughly 95 milliseconds.

Theoretical analysis had established a stark baseline: a simulated decode step using BF16 GEMMs and NCCL AllReduce completed in just 8.9 milliseconds. The difference — 86 milliseconds — was a vast, unexplained gap representing 3.4% efficiency against the theoretical maximum of 309 tok/s.

The assistant had already built and deployed a custom gap analysis script to decompose this gap into measurable components. The results were illuminating but incomplete. The script measured MoE routing overhead (2.4ms total), token permutation (1.6ms), RMSNorm (3.4ms), CPU dispatch overhead (5.3ms), and FP4 GEMMs (2.3ms). Combined with AllReduce (6.6ms), these accounted for only ~22 milliseconds of the 95ms TPOT. That left ~73 milliseconds completely unexplained — nearly 77% of the total decode time was a black box.

The Failed Nsight Systems Campaign

The assistant's first instinct was correct: use NVIDIA Nsight Systems (nsys), the industry-standard GPU profiling tool, to capture a full trace of a single inference request. The plan was methodical: launch the server under nsys with --capture-range=none for delayed capture, then trigger profiling during a single inference request.

But something went wrong. The detokenizer heartbeat began failing. The health endpoint returned HTTP 503 errors. The scheduler processes became defunct zombies. The assistant's diagnosis was perceptive: "The nsys overhead + --enable-layerwise-nvtx-marker seems to have made the scheduler/detokenizer hang." Nsight Systems, when configured to trace every CUDA API call and NVTX event across eight GPU worker processes, introduced enough instrumentation overhead to break the server's timing-sensitive scheduling loop.

The Torch Profiler Breakthrough

After the nsys failure, the assistant made a critical strategic decision. Instead of wrapping the entire server in an external profiler, it would inject torch.profiler instrumentation directly into SGLang's model runner — modifying the source code of the inference engine itself to capture a targeted profile of the forward pass.

The patch was a model of surgical precision. The assistant located the forward_decode method and replaced it with a version that included profiler state management. The profiler was configured to activate only on rank 0, perform 20 warmup decode steps, capture 30 active steps of detailed CPU and CUDA activity, and export the trace to a Chrome trace format file.

The results were stunning. The profiler summary revealed that a single operation consumed the majority of decode time:

| Component | Per-step Time | % of TPOT | |---|---|---| | aten::copy_ (dtype cast) | 64.6 ms | 69.3% | | NCCL AllReduce | 7.7 ms | 8.2% | | aten::mm (GEMM) | 6.1 ms | 6.6% | | CUTLASS GemmUniversal | 3.0 ms | 3.3% | | cuBLAS gemvx | 2.8 ms | 3.0% |

The aten::copy_ operation, backed by the unrolled_elementwise_kernel, consumed 64.6 milliseconds per decode step — more than eight times the next largest contributor. The pattern was unmistakable: 2,340 calls across 30 steps = 78 calls per step. GLM-5 has 78 layers. Something was performing a dtype conversion once per layer, every decode step.

The real culprit was the FlashInfer MLA attention backend — specifically, the KV cache being cast from FP8 to BF16 on every layer for the entire 495,552-token pool. The code at flashinfer_mla_backend.py was executing k_buffer = forward_batch.token_to_kv_pool.get_key_buffer(layer.layer_id).to(q.dtype), casting the entire KV cache pool — 285.6 million elements per layer, approximately 857 MB of data — from FP8 to BF16 before the attention computation.

The Gather-Then-Cast Patch

The assistant designed and implemented a targeted fix. The core insight was simple: the existing code was casting the entire KV cache pool from FP8 to BF16 on every layer, every decode step. But the attention computation only needed the KV entries for the active tokens in the current batch — a tiny fraction of the total pool.

The gather-then-cast patch changed the logic to:

1. Gather only the KV entries corresponding to active token positions from the FP8 pool
2. Cast only those gathered entries from FP8 to BF16
3. Pass the smaller, cast buffer to the attention kernel The benchmark results showed a 29% improvement: 13.5–13.6 tok/s with TPOT of approximately 74ms, compared to the baseline of ~10.5 tok/s (95.6ms TPOT). But the theoretical savings from eliminating the 64.6ms cast was not fully realized. The actual improvement was only about 21ms, roughly one-third of the theoretical maximum. Despite the improvement, the fundamental architectural limitation remained. The user made the decisive call: "Maybe let's just consider abandoning this quant." The NVFP4 path was abandoned in favor of unsloth's GGUF quantization and vLLM deployment.

Part V: The GGUF Frontier — Engineering vLLM Support from First Principles

The Strategic Pivot

The pivot to GGUF was not a simple switch. It was the culmination of weeks of profiling and optimization work that had revealed fundamental architectural limitations in the NVFP4 path. The assistant freed disk space by deleting the NVFP4 model files, installed vLLM nightly, and began downloading the 431 GB GGUF split files from Hugging Face.

But before the download could complete, the assistant discovered the critical blocker: vLLM's GGUF support depends on transformers to parse the GGUF file's metadata header into a HuggingFace model configuration, and transformers simply did not know about the glm-dsa architecture. The GGUF_CONFIG_MAPPING dictionary in transformers.integrations.ggml.py contained entries for architectures like bloom, falcon, gemma, llama, mistral, phi3, qwen2, and qwen2_moe, but nothing for any DeepSeek-derived architecture.

The user's response was unambiguous and transformative: "E. add this gguf support to vllm." With those five words, the effort shifted from a deployment exercise into a software development project — one that would require writing a comprehensive patch for vLLM's gguf_loader.py, installing bleeding-edge versions of transformers and gguf-py from source, and enabling a deployment configuration that had never been successfully achieved before.

The Parallel Reconnaissance

The assistant launched a three-pronged parallel research effort that illuminated the entire GGUF loading pipeline. Three subagent tasks were dispatched simultaneously, each exploring a different codebase:

Task 1: Study transformers GGUF mapping. This task investigated how the HuggingFace transformers library handles GGUF files, specifically the GGUF_CONFIG_MAPPING dictionary that maps GGUF metadata keys to HuggingFace config parameters. The task discovered that the mapping pipeline has three stages: config mapping, tensor name mapping, and weight loading. The config mapping stage was the blocker — without an entry for deepseek2 or glm-dsa in GGUF_CONFIG_MAPPING, the entire pipeline fails before reaching the tensor name mapping stage.

Task 2: Study vLLM GGUF loader code. This task examined vLLM's gguf_loader.py, specifically the GGUFModelLoader class and its _get_gguf_weights_map() method. The critical discovery was that vLLM's GGUF loader does NOT use transformers' ggml.py — it has its own _get_gguf_weights_map() method that builds a dictionary mapping GGUF tensor names to HuggingFace tensor names. When using --hf-config-path (external config) and --tokenizer (external tokenizer), the transformers GGUF config mapping is bypassed entirely.

Task 3: Study GLM-5 GGUF tensor names. This task explored the actual tensor names inside the GLM-5 GGUF file using the installed gguf-py Python package. The task discovered that the glm-dsa architecture in gguf-py defines 38 tensor types, including specialized entries like INDEXER_ATTN_K, INDEXER_ATTN_Q_B, NEXTN_EH_PROJ, and NEXTN_EMBED_TOKENS that are unique to the DSA (Dynamic Sparse Attention) mechanism.

The kv_b_proj Split and the Latent DeepSeek Bug

The most technically subtle discovery was the kv_b_proj split. In the HuggingFace reference model, the key-value projection exists as a single combined weight matrix. But during GGUF conversion via llama.cpp's convert_hf_to_gguf.py, this tensor is split into two separate tensors: attn_k_b (the key portion) and attn_v_b (the value portion). The vLLM loader expects the original combined tensor, so the patch must reverse this split.

The assistant traced the exact transformation in the llama.cpp conversion script. The conversion performs a complex sequence: reshape, split into k_b and v_b, transpose k_b, permute v_b. The reversal of these operations is non-trivial and depends on the exact sequence of reshape, split, transpose, and permute operations applied during conversion.

During this investigation, the assistant made a discovery that extended far beyond GLM-5. By tracing through vLLM's _get_gguf_weights_map() function, the assistant found that the existing DeepSeek V2/V3 GGUF support was also broken due to the same kv_b_proj mapping issue. The gguf-py library's auto-mapping function maps kv_b_proj to attn_kv_b (the combined form), but the actual GGUF file has attn_k_b and attn_v_b (the split form). The auto-mapping creates entries for attn_kv_b that don't match any tensor in the file, while the split tensors are left unmapped. This means the kv_b_proj weight is silently skipped during loading — the model loads without errors, but the attention weights are uninitialized, producing garbage outputs.

This latent bug had been present in vLLM's codebase for months, affecting anyone trying to load DeepSeek V2 or V3 models in GGUF format. The assistant's patch for GLM-5 would fix this bug for all three architectures simultaneously.

The Three Launch Failures

With the patches deployed and the mappings validated, the assistant launched vllm serve. The first attempt died immediately with no output. Running it directly revealed the first of three blockers.

Blocker 1: The Speculators Crash. The error was a ValueError from transformers: the glm-dsa architecture was not in GGUF_SUPPORTED_ARCHITECTURES. The fix was surgical: wrap the get_config_dict call in a try/except that catches the ValueError for unsupported architectures.

Blocker 2: The Bfloat16 Barrier. The next launch attempt hit a new error: torch.bfloat16 is not supported for quantization method gguf. The GLM-5 Hugging Face config defaults to torch_dtype: bfloat16, but vLLM's GGUF quantization backend only supports float16 and float32. The fix was trivial — adding --dtype float16 to the launch command.

Blocker 3: The Attention Backend Wall. The third blocker was the most consequential. The server failed with a multidimensional compatibility error: no existing attention backend could simultaneously satisfy SM120 compute capability, sparse MLA attention, qk_nope_head_dim=192, and float16 dtype.

The assistant systematically evaluated every MLA-capable backend and documented why each one failed. The Triton MLA backend was the closest to working — it supported SM120 (since Triton compiles JIT for the target architecture) and the correct head dimensions, but it had no sparse attention support.

Designing the Triton MLA Sparse Backend

The user and assistant agreed on a strategy: patch the Triton MLA backend to support sparse attention. The Triton backend was pure Python/Triton code, not a compiled CUDA kernel, making it the most feasible target for modification.

The assistant's breakthrough came from reading the Triton decode kernel's source code. The kernel's _fwd_kernel_stage1 function iterates over KV positions using a Req_to_tokens block table. The critical insight was that the Req_to_tokens parameter is just a lookup table — it maps a logical position to a physical cache slot. The kernel doesn't care whether the logical positions are contiguous (0, 1, 2, ...) or arbitrary sparse indices. If you replace the block table with the pre-resolved sparse indices, the kernel naturally performs sparse attention.

The design that emerged was elegantly simple: obtain sparse indices from the DSA indexer, reshape them as a block table with page_size=1, set b_seq_len = topk, and call the existing kernel. This approach meant that no Triton kernel modifications were needed. The entire implementation was a data transformation layer that reshapes the sparse indices into the format the existing kernel expects.

The String Replacement Bug

With the backend registered, the assistant launched the server again. The log showed the milestone: Using TRITON_MLA_SPARSE attention backend. But the server still crashed during weight loading.

The error was a KeyError with a malformed parameter name: model.layers.0.self_attn.indexer.qweight_types_proj.qweight_type. The name qweight_types_proj was clearly corrupted.

The assistant traced the error through vLLM's distributed worker architecture to the load_weights method, then to the GGUF tensor metadata, and finally to the root cause in weight_utils.py. At lines 977 and 1004, the code used Python's str.replace to rename quantized tensors:

weight_type_name = name.replace("weight", "qweight_type")
name = name.replace("weight", "qweight")

The bug was devastating in its simplicity: str.replace() replaces all occurrences of the substring, not just the suffix. For the parameter weights_proj.weight, the first replacement turned it into qweight_types_proj.weight (replacing "weight" with "qweight_type" in "weights" → "qweight_types"), and the second replacement turned it into qweight_types_proj.qweight. The model then tried to look up qweight_types_proj.qweight_type, which didn't exist.

This was a latent bug in vLLM's codebase that had never manifested because DeepSeek V2/V3 — the primary targets of the MLA GGUF support — don't have parameters with "weight" as a substring in their names. The GLM-5 architecture's weights_proj parameter (part of the DSA indexer) was the first to trigger it.

The Garbage Output Crisis

With all fixes deployed, the model finally loaded. But the first inference request returned a stream of incomprehensible tokens: ,,,,,,,[v elas(l(l(l [w(l(!(l elaselaselaselas.... The model loaded, but it could not reason.

The debugging that followed was a masterclass in systematic hypothesis elimination. The assistant ruled out the chat template, the GGUF dequantization kernel (verified on SM120 with a maximum difference of 0.00012), the weight name mapping, and the kv_b_proj reassembly logic. A diagnostic test with VLLM_MLA_DISABLE=1 — forcing standard FlashAttention instead of the optimized Triton MLA backend — produced correct output. This was the first clear signal that the bug was in the attention computation path.

The root cause was a shard ordering bug in GGUFLinearMethod.apply(). For fused projections like fused_qkv_a_proj, the method iterated over shard IDs in GGUF file load order rather than canonical order. When KV shards happened to be stored before Q shards in the file, the fused output had KV columns first and Q columns second — but the downstream MLA code assumed Q came first. The fix was a one-line change: for idx in sorted(shard_id).

With both bugs fixed, the assistant ran a comprehensive benchmark: 300/300 requests succeeded with zero errors at 1,024-way concurrency, delivering 1,049 tok/s peak throughput. The single-request decode throughput was ~20 tok/s — correct but slow.

Part VI: The Optimization Sprint — From 20 to 57 tok/s

Profiling the Bottleneck

The user's response was terse and unambiguous: "20t/s is still really slow, gat you profile and try to improve to at least 100/s." The assistant dispatched a subagent to profile single-request decode performance. The results were precise and actionable: 50.5ms per decode step, 19.8 tok/s, 91% GPU utilization. The subagent's comprehensive analysis revealed that NCCL allreduce consumed 42% of decode time — 157 separate allreduce calls per step, each taking ~135μs, totaling ~21ms. The GPUs were connected via PCIe Gen5 without NVLink, making inter-GPU communication inherently expensive.

CUDAGraph and NCCL Tuning

The assistant's first optimization target was CUDAGraph, PyTorch's mechanism for capturing GPU operations into a replayable graph. There was a complication: earlier in the debugging phase, CUDAGraph had produced garbage output. The assistant had attributed this to the direct-call patch that bypassed the custom MLA op. But in a critical re-examination, the assistant discovered that the shard ordering fix was the real root cause all along — the direct-call patch was never needed. With the shard ordering fix applied, the upstream custom op path worked correctly, and CUDAGraph was now viable.

The result was a 2.15× improvement: single-request throughput jumped from 20 tok/s to 43 tok/s. CUDAGraph delivered 2× improvement at low concurrency where NCCL overhead dominated, but at high concurrency the benefit shrank to near-zero because batching already amortized kernel launch overhead.

The assistant then turned to NCCL tuning. A subagent investigated NCCL protocol options and found that NCCL_PROTO=LL (Low Latency protocol) yielded a 34% improvement, pushing throughput from 43 tok/s to 57 tok/s. The LL protocol is optimized for small message sizes, and each of the 158 allreduce calls per step carried only ~12KB of data — exactly the regime where LL excels.

The Hardware Ceiling

The assistant explored more aggressive optimizations: custom allreduce via shared memory (broken on PCIe with more than 2 GPUs), allreduce-RMSNorm fusion using flashinfer (requires NVSwitch multicast hardware, unavailable on PCIe), and pipeline parallelism combined with tensor parallelism. Each was systematically investigated and found either incompatible with the PCIe-only topology or yielding zero additional benefit.

The assistant's honest assessment captured the moment of reckoning: "The flashinfer allreduce fusion requires NVSwitch/multicast hardware — not available on PCIe GPUs. And all other NCCL tuning beyond NCCL_PROTO=LL shows zero additional benefit. We're at 57.6 tok/s and the theoretical ceiling with zero-cost allreduce would be 100-140 tok/s. The remaining ~11ms of allreduce can't be eliminated on this hardware."

This was the hardware ceiling — a physical constraint imposed by the PCIe Gen5 interconnect, not a software bug to be fixed or a knob to be tuned. The assistant had established the practical performance envelope for this configuration.

Part VII: The Model Gauntlet — Three 1T-Parameter Models in One Session

The Pivot to Kimi-K2.5

The user delivered a concise but consequential verdict: the GLM-5 GGUF experiment was being abandoned. The new target was nvidia/Kimi-K2.5-NVFP4, a 1-trillion-parameter Mixture-of-Experts model based on the DeepSeek V3 architecture. The assistant's reconnaissance revealed the full scope of the cleanup required: all 8 GPUs were still occupied by the old GLM-5 model, each consuming 93,783 MiB out of 97,887 MiB total.

The download of 119 safetensor shards totaling 540GB was an operation in itself. At one point, a race condition nearly derailed the download — the assistant had started two parallel download processes that could have corrupted the files. The assistant caught this and killed the redundant process.

The first launch attempt failed with an error that became the central blocker: No valid MLA attention backend supports FP8 KV cache on SM120 (Blackwell RTX PRO 6000). The model's configuration specified FP8 KV cache, and every MLA backend on SM120 either didn't support the compute capability or didn't support the data type. The TRITON_MLA backend, the only viable option on Blackwell, had explicit NotImplementedError guards for FP8.

The fix was surgical: remove the FP8 KV cache configuration from the model files, forcing vLLM to fall back to fp16 KV cache. A Python script edited two JSON files, removing kv_cache_quant_algo from hf_quant_config.json and kv_cache_scheme from config.json. This was the minimal intervention that unblocked the entire deployment.

The benchmark results showed approximately 61 tok/s single-stream and ~1,200 tok/s at high concurrency. The root cause was identified with surgical precision: PCIe allreduce across 8 GPUs for the 61-layer MLA architecture. Approximately 65–70% of decode time was spent in NCCL allreduce.

MiniMax-M2.5: The GQA Revolution

The turning point came with a single message from the user: "Try https://huggingface.co/MiniMaxAI/MiniMax-M2.5, which is native fp8, smaller activation so should be faster." This was not a casual suggestion — it was a hypothesis grounded in architectural awareness. The user understood that the bottleneck was MLA's allreduce overhead, and they proposed a model with a fundamentally different attention mechanism: Grouped Query Attention (GQA).

The assistant immediately recognized the strategic significance. Through detailed architectural analysis, it identified six reasons MiniMax-M2.5 should outperform Kimi-K2.5 on this specific hardware: GQA instead of MLA (standard FlashAttention works), 10B active parameters vs ~37B (3.7× less compute per token), FP8 native weights (half the memory bandwidth of BF16), 3 MTP heads (built-in speculative decoding), 230GB vs 540GB (more comfortable fit), and no cross-GPU attention (each GPU gets exactly 1 KV head).

The results were immediate and dramatic. The model loaded in just 75 seconds — compared to Kimi-K2.5's 13-minute cold start. Single-stream throughput reached 84 tok/s, a 38% improvement. At high concurrency, the system delivered 2,586 tok/s at C=256, more than double the previous peak.

The EP Breakthrough: 4,000 tok/s

The user asked a question that would unlock the session's highest throughput: "can we do tp/ep or tp6?" The assistant embarked on a systematic exploration of vLLM's source code to understand how expert parallelism interacts with tensor parallelism. Through targeted grep commands across the codebase, the assistant discovered that vLLM supports --enable-expert-parallel and that when EP is enabled, the MoE layer uses EP for expert distribution while TP is used only for non-expert layers. Critically, with EP, the expert weights are not tensor-sharded — each EP rank holds a subset of experts in full.

The breakthrough came when the assistant launched MiniMax-M2.5 with TP=8 and EP enabled. The model started successfully in about 90 seconds, and the benchmark results were transformative: the system achieved a flat ~4,000 tok/s throughput from C=256 through C=1024, with zero errors and no degradation. This was the highest throughput recorded in the entire session — 3.2× what the NVFP4 Kimi-K2.5 model achieved at its peak.

The Native INT4 Kimi-K2.5

The session made one final pivot — back to the Kimi-K2.5 model family, but this time using the native INT4 quantization variant (moonshotai/Kimi-K2.5). This model, despite its 547GB size and 36-minute load time, delivered an impressive 82 tok/s single-stream — well exceeding the 40–50 tok/s target — and scaled to 2,276 tok/s at high concurrency.

The INT4 variant significantly outperformed the NVFP4 variant (82 vs 61 tok/s single-stream), demonstrating that quantization format choice matters as much as model architecture. Extensive NCCL tuning experiments — testing Ring vs LL algorithms, varying channel counts, and adjusting thread configurations — confirmed that the bottleneck was fundamental hardware bandwidth, not algorithm choice. The session concluded by deploying the INT4 Kimi-K2.5 as a persistent systemd service.

Part VIII: The Speculative Decoding Campaign — EAGLE-3 and DFlash

Building the EAGLE-3 Pipeline

With the base model deployed, the assistant turned to speculative decoding — a technique that uses a small "draft" model to predict multiple tokens ahead, which are then verified by the large "target" model in parallel. The goal was to improve single-stream throughput beyond the PCIe allreduce bottleneck.

The assistant built a complete EAGLE-3 training pipeline for Kimi-K2.5, spanning data generation, hidden state extraction, model training, and deployment. The pipeline was tested end-to-end on 1000 samples, then scaled to 100K samples. The training achieved 74.7% validation accuracy.

But the deployment revealed a fundamental problem: vLLM's EAGLE-3 integration with MLA attention yielded only ~15% acceptance rate, translating to 0.66× throughput — worse than the baseline. The assistant pivoted to SGLang, which loaded the model in 22 seconds but deadlocked on SM120.

The debugging that followed spanned dozens of messages. The assistant discovered that the hidden state input format differed between training and inference — the training pipeline used single-layer 7168-dim hidden states, but SGLang expected multi-layer 21504-dim states. The fix required modifying the draft model config to include the embedding layer and correcting the speculative algorithm flag from EAGLE to EAGLE3.

After all fixes, the EAGLE-3 deployment achieved 94 tok/s — a 5.9% improvement over the 90 tok/s baseline. The user diagnosed that the verify step lacked CUDA graphs, limiting the benefit. The assistant analyzed the viability math and concluded that EAGLE-3 speculation was performing worse than baseline due to the verify step overhead.

The CUDA 13 Breakthrough

The assistant upgraded the CUDA stack to version 13, patched SGLang for SM120 support, and enabled FlashInfer allreduce fusion and Torch symmetric memory. This transformed EAGLE-3 speculative decoding from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s.

But parallel throughput benchmarks showed that baseline strictly outperformed EAGLE-3 at all concurrency levels. The assistant attempted to implement dynamic speculation disable but hit fundamental state coupling issues in the standard EAGLE worker. The spec_v2 overlap path was explored but ultimately abandoned.

The DFlash Training Pipeline

The assistant pivoted to DFlash (Draft-and-Flash), a different speculative decoding architecture that promised better acceptance rates. The training pipeline was built from scratch, spanning data generation, tokenization, model definition, and distributed training across 8 Blackwell GPUs.

The training pipeline encountered and overcame an extraordinary number of obstacles. The assistant fixed six training bugs, resolved FLA Triton autotuner crashes through sequential warmup and lazy compilation, upgraded Triton to 3.7.0, and debugged a CachedAutotuner race condition. The pipeline was transformed from a synchronous lock-step loop to a fully asynchronous CSP-style architecture, achieving 16 Ktok/s with 100% GPU utilization.

The debugging continued: homogeneous batching issues, wrong gamma parameters, noise warmup bugs, AdamW betas, and a fundamental loss function mismatch between soft KL and hard cross-entropy. Each bug was diagnosed and fixed systematically.

The assistant built an evaluation harness to compare the DFlash drafter against a reference model from z-lab. The comparison revealed a 4× performance gap, traced to three critical bugs: noise corrupting target logits, the fc shortcut including the target layer, and the loss function mismatch. After all fixes, the v5 training run was launched.

The DDTree Deployment

The assistant pivoted from training to deploying the z-lab DFlash DDTree drafter on Pro6000 hardware. A standalone OpenAI-compatible DDTree service was deployed on CT200, achieving a 24% throughput improvement over linear DFlash through budget tuning.

The DDTree was deployed for Kimi K2.6, achieving up to 2.15× speedup over the autoregressive baseline across PCIe Blackwell and NVLink B300 platforms. The assistant authored a comprehensive findings report documenting the DDTree optimization campaign.

A custom sm_120 verify attention kernel was built with CUDA graph support, achieving 3-6× decode speedup over Triton. KV defragmentation was implemented, and MoE imbalance was identified as the remaining bottleneck.

Part IX: DeepSeek-V4-Flash and the Full-Stack Optimization

The PD Disaggregation Deployment

The assistant deployed DeepSeek-V4-Flash on Blackwell with prefill-decode disaggregation — splitting the prefill (prompt processing) and decode (token generation) phases across different GPU sets. This allowed each phase to be optimized independently.

The optimization journey was comprehensive. The assistant designed custom MMA attention kernels, flipped FP32 indexer operations to bf16 tensor-core operations, and fixed the indexer O(max_context) bottleneck for ~17× throughput gain. PD disaggregation was deployed with systemd services, Prometheus/Grafana monitoring was set up from scratch, and tool-calling quality issues were resolved.

The final throughput reached 531 tok/s — a dramatic improvement from the initial 11 tok/s.

The bf16 Index-K Bug

The deployment encountered a multi-turn context-loss failure that required deep debugging. The assistant diagnosed DSA sparse attention recall failure on long contexts, fixed by increasing index_topk to 1024 and implementing bf16 index keys in the fused CUDA kernel.

But the bf16 index-K patch introduced a high-concurrency tool-call corruption bug. The assistant root-caused the issue to a multi-stream-overlap race condition under CUDA graph capture. The fix was to disable multi-stream overlap, which resolved the corruption but limited throughput scaling.

The production deployment encountered a PD deadlock, resolved by disabling overlap schedule. A mass-abort wedge in NIXL bootstrap_thread was fixed. Pool sizing issues were corrected. The assistant conducted A/B tests to isolate the corruption to the bf16 index-K path, and ultimately identified that the decode-side bf16 index-K handling was the trigger.

The Production Engineer's Playbook

The production stabilization phase was a masterclass in incident response. The assistant fixed a PD deadlock by disabling overlap schedule, root-caused high-concurrency tool-call corruption, fixed a mass-abort wedge in NIXL bootstrap_thread, and corrected pool sizing. The bf16 index-K corruption was ultimately fixed by disabling multi-stream-overlap.

The assistant investigated decode throughput scaling to C90, evaluated Two-Batch Overlap feasibility, and A/B tested overlap scheduler on decode. A TP-collective desync hazard was identified and fixed. The production PD bootstrap incident was resolved with a co-restart procedure.

The session concluded with the restoration of the SGLANG_SM120_MMA_TARGET_CTAS=512 knob after the user identified a faulty client-side proxy as the true cause of a multi-round harness hang. A full PD co-restart was performed, and the documentation was corrected to reflect the actual root cause.

Themes and Lessons

The Full-Stack Nature of ML Infrastructure

This session demonstrates that deploying frontier LLMs is not a single-discipline activity. It requires expertise spanning hardware driver installation, CUDA toolkit management, Python environment configuration, compiled CUDA extension debugging, dependency resolution, hypervisor topology analysis, kernel-level performance optimization, profiler-driven bottleneck discovery, custom attention backend development, speculative decoding pipeline construction, and production-grade service deployment.

The assistant navigated all of these domains, often within a single segment. The ability to move from kernel module parameters to HuggingFace transformers versions to JIT compilation build tools — and to understand how each layer interacts with the others — is the defining characteristic of effective ML infrastructure engineering.

The Primacy of Methodology

Throughout the session, the assistant maintained a consistent methodology: form a hypothesis, design an experiment, run it, interpret the results honestly, and iterate. This discipline is what enabled progress even when individual experiments failed. The OEA null result was not a setback — it was knowledge gained. The 3.4% efficiency revelation was not a failure — it was a forcing function that redirected the investigation toward the kernel level.

The most important tool in the assistant's arsenal was not a debugger or a profiler but the ability to read source code — on the remote machine, on GitHub, and in the local filesystem. Understanding the Triton decode kernel's iteration pattern, the SparseMLAAttentionImpl class hierarchy, and the maybe_override_with_speculators function's call site were all achieved through careful code reading.

The Hardware Ceiling

Perhaps the most important lesson from this session is the recognition of hardware-imposed limits. The assistant explored every plausible optimization path: CUDAGraph, NCCL protocol tuning, custom allreduce, allreduce-RMS fusion, pipeline parallelism, expert parallelism, speculative decoding. Each was evaluated against profiling data, and each was either adopted or rejected based on empirical results.

The final ceiling of ~57 tok/s for GLM-5, ~61 tok/s for NVFP4 Kimi-K2.5, and ~4,000 tok/s for MiniMax-M2.5 were not failures of software optimization but physical constraints of the PCIe-only GPU topology. The assistant's honest assessment — documenting what was and wasn't possible — demonstrated engineering integrity over wishful thinking.

The Value of Negative Results

The most striking feature of this session is the assistant's willingness to accept negative results. Piecewise CUDA Graphs was blocked by a framework incompatibility. MSCCLPP delivered ~2%. SBO delivered ~2%. EP8 regressed low-concurrency performance and crashed under load. The OEA experiment produced a clean null result. EAGLE-3 speculation was net-negative until CUDA 13.

Each of these results could have been rationalized away — "maybe if I tweak the parameter," "maybe with a different backend," "maybe with more memory." Instead, the assistant accepted each result, documented it, and moved to the next hypothesis. This discipline is rare and valuable. In optimization work, the sunk cost fallacy is a constant temptation, and the assistant resisted it consistently.

The Power of Parallel Exploration

The assistant's use of parallel subagent tasks was a recurring pattern that dramatically accelerated the work. When faced with a complex problem — understanding three codebases simultaneously, auditing ten system layers, researching six optimization approaches — the assistant dispatched independent subagents that ran concurrently. The wall-clock time was roughly the duration of the longest-running agent, compressing what could be hours of sequential investigation into a single round.

This pattern was enabled by opencode's task tool, which spawns independent subagent sessions that run to completion concurrently. The parent session blocks until all subagents return, but the parallelism within each round is substantial.

The User-Assistant Partnership

Throughout the session, a productive partnership emerged between the user and the assistant. The user provided high-level direction ("improve to at least 100/s," "productionalise into systemd service," "add this gguf support to vllm") while the assistant executed with autonomy and thoroughness. The user's single-sentence messages at critical junctures demonstrated trust and effective delegation.

This partnership worked because both parties understood their roles. The assistant drove the technical investigation, formulated hypotheses, ran experiments, and reported results. The user set priorities, cut off diminishing-returns exploration, and provided domain knowledge. The result was a collaboration that achieved more than either could have alone.

Conclusion

The opencode session spanning 75 segments is a landmark in ML infrastructure engineering. It traces the complete lifecycle of deploying frontier large language models on cutting-edge hardware, from bare-metal driver installation through production-grade service deployment. The models deployed ranged from GLM-5-NVFP4 (744B) through Kimi-K2.5 (1T), MiniMax-M2.5 (230B active), Qwen3.5-397B, DeepSeek-V4-Flash, and many others. The inference engines evolved from SGLang to vLLM and back again, with custom patches applied to both.

The session's achievements are substantial: a 29% improvement from the gather-then-cast patch, a 2.85× improvement from CUDAGraph and NCCL tuning, a 4,000 tok/s peak throughput with MiniMax-M2.5, a custom Triton MLA sparse attention backend for Blackwell GPUs, a complete EAGLE-3 training and deployment pipeline, a DFlash DDTree deployment with 2.15× speedup, a custom sm_120 verify attention kernel, and a full PD disaggregation deployment with monitoring.

But the true legacy of this session is not the specific configurations chosen or the throughput numbers achieved. It is the methodology demonstrated: systematic hypothesis testing, profiling-first optimization, parallel exploration, honest acceptance of negative results, and the relentless pursuit of understanding at every layer of the stack. These are the skills that separate effective ML engineering from trial-and-error configuration.

The session stands as a testament to the challenges of deploying at the frontier, where every component — model architecture, quantization format, serving framework, GPU hardware, and even the virtualization layer — is evolving simultaneously. The only way forward is systematic, iterative debugging informed by local empirical knowledge, with the humility to recognize that the problem may not be where you're looking.

References

[1] Building an ML Server from Scratch: Drivers, CUDA Hell, and the Flash-Attention Gauntlet [2] Through the Blackwell Storm: Deploying GLM-5-NVFP4 from NaN Crashes to Production Benchmarking [3] Deploying GLM-5-NVFP4 on 8× RTX PRO 6000 Blackwell GPUs: From NaN Crashes to Virtualization Bottlenecks [4] The Architecture of Impossibility: An 8-GPU Proxmox VM's Battle Against the Physics of PCIe Topology [5] The LXC Pivot: When Bare-Metal GPU Topology Met the Blackwell GSP Firmware Wall [6] The HMM That Almost Stopped Blackwell: From CUDA Blocker to 806 tok/s on Eight RTX PRO 6000 GPUs [7] The Blackwell Divide: From 880 to 3,740 tok/s and the Allreduce Fusion Wall [8] From 235 Watts to a Ranked Roadmap: The GLM-5-NVFP4 Optimization Campaign on Blackwell GPUs [9] Testing the Optimization Frontier: Systematic Hypothesis Evaluation on Blackwell GPUs [10] The Methodical Campaign: From SGLang Update to Theoretical Ceiling in the GLM-5 Inference Project [11] The 30x Gap: From Theoretical Maximum to Kernel-Level Bottleneck in Blackwell FP4 Inference [12] The 69% Bottleneck: How a Torch Profiler Trace Reshaped the GLM-5 Deployment Strategy [13] The GGUF Frontier: Engineering vLLM Support for GLM-5 from First Principles [14] The Architecture of a Patch: Reverse-Engineering GLM-5's GGUF Attention Weights for vLLM [15] The Final Gauntlet: Deploying GLM-5 GGUF on Blackwell GPUs Through a Cascade of Breakthroughs [16] From Crash to Coherence: Debugging the GLM-5 GGUF Deployment on 8× Blackwell GPUs [17] From Garbage to Production: Debugging, Optimizing, and Deploying GLM-5 on 8× Blackwell GPUs [18] The Kimi-K2.5-NVFP4 Deployment: Pivoting from GLM-5 to a 1T-Parameter Model on Blackwell GPUs [19] The Three-Model Gauntlet: Systematic Evaluation of 1T-Parameter LLMs on 8× Blackwell GPUs## Part VIII: The Speculative Decoding Campaign — EAGLE-3 and DFlash (Extended)

The EAGLE-3 Training Pipeline

With the base model deployed, the assistant turned to speculative decoding — a technique that uses a small "draft" model to predict multiple tokens ahead, which are then verified by the large "target" model in parallel. The goal was to improve single-stream throughput beyond the PCIe allreduce bottleneck.

The EAGLE-3 training pipeline was built from scratch. The assistant wrote scripts for hidden state extraction, data generation, model training, and deployment. The pipeline was tested end-to-end on 1000 samples, then scaled to 100K samples. The training achieved 74.7% validation accuracy.

But the deployment revealed a fundamental problem: vLLM's EAGLE-3 integration with MLA attention yielded only ~15% acceptance rate, translating to 0.66× throughput — worse than the baseline. The assistant pivoted to SGLang, which loaded the model in 22 seconds but deadlocked on SM120.

The debugging that followed spanned dozens of messages. The assistant discovered that the hidden state input format differed between training and inference — the training pipeline used single-layer 7168-dim hidden states, but SGLang expected multi-layer 21504-dim states. The fix required modifying the draft model config to include the embedding layer and correcting the speculative algorithm flag from EAGLE to EAGLE3.

After all fixes, the EAGLE-3 deployment achieved 94 tok/s — a 5.9% improvement over the 90 tok/s baseline. The user diagnosed that the verify step lacked CUDA graphs, limiting the benefit. The assistant analyzed the viability math and concluded that EAGLE-3 speculation was performing worse than baseline due to the verify step overhead.

The CUDA 13 Breakthrough

The assistant upgraded the CUDA stack to version 13, patched SGLang for SM120 support, and enabled FlashInfer allreduce fusion and Torch symmetric memory. This transformed EAGLE-3 speculative decoding from a net-negative 54.1 tok/s to a net-positive 96.1 tok/s.

But parallel throughput benchmarks showed that baseline strictly outperformed EAGLE-3 at all concurrency levels. The assistant attempted to implement dynamic speculation disable but hit fundamental state coupling issues in the standard EAGLE worker. The spec_v2 overlap path was explored but ultimately abandoned.

The DFlash Training Pipeline

The assistant pivoted to DFlash (Draft-and-Flash), a different speculative decoding architecture that promised better acceptance rates. The training pipeline was built from scratch, spanning data generation, tokenization, model definition, and distributed training across 8 Blackwell GPUs.

The training pipeline encountered and overcame an extraordinary number of obstacles. The assistant fixed six training bugs, resolved FLA Triton autotuner crashes through sequential warmup and lazy compilation, upgraded Triton to 3.7.0, and debugged a CachedAutotuner race condition. The pipeline was transformed from a synchronous lock-step loop to a fully asynchronous CSP-style architecture, achieving 16 Ktok/s with 100% GPU utilization.

The debugging continued: homogeneous batching issues, wrong gamma parameters, noise warmup bugs, AdamW betas, and a fundamental loss function mismatch between soft KL and hard cross-entropy. Each bug was diagnosed and fixed systematically.

The assistant built an evaluation harness to compare the DFlash drafter against a reference model from z-lab. The comparison revealed a 4× performance gap, traced to three critical bugs: noise corrupting target logits, the fc shortcut including the target layer, and the loss function mismatch. After all fixes, the v5 training run was launched.

The DDTree Deployment

The assistant pivoted from training to deploying the z-lab DFlash DDTree drafter on Pro6000 hardware. A standalone OpenAI-compatible DDTree service was deployed on CT200, achieving a 24% throughput improvement over linear DFlash through budget tuning.

The DDTree was deployed for Kimi K2.6, achieving up to 2.15× speedup over the autoregressive baseline across PCIe Blackwell and NVLink B300 platforms. The assistant authored a comprehensive findings report documenting the DDTree optimization campaign.

A custom sm_120 verify attention kernel was built with CUDA graph support, achieving 3-6× decode speedup over Triton. KV defragmentation was implemented, and MoE imbalance was identified as the remaining bottleneck.

The Data Quality Revolution

A critical insight emerged during the EAGLE-3 campaign: the quality of training data mattered more than the model architecture. The assistant initially used random prompts for data generation, but the OEA experiment showed that uniform expert routing left no clustering to exploit. The pivot to OpenRouter API for data generation — completing all B-datasets (B3-B8) in 33 minutes at $86 cost — transformed the training data quality.

The assistant built an OpenRouter inference script with high concurrency, reconstructed Kimi-K2.5 token IDs from text responses, and completed all B-datasets via the API. The merge-and-shuffle script was written, and the old 10K hidden states were identified for deletion. This data quality revolution was essential for the EAGLE-3 training to achieve meaningful acceptance rates.

Part IX: DeepSeek-V4-Flash and the Full-Stack Optimization (Extended)

The PD Disaggregation Deployment

The assistant deployed DeepSeek-V4-Flash on Blackwell with prefill-decode disaggregation — splitting the prefill (prompt processing) and decode (token generation) phases across different GPU sets. This allowed each phase to be optimized independently.

The optimization journey was comprehensive. The assistant designed custom MMA attention kernels, flipped FP32 indexer operations to bf16 tensor-core operations, and fixed the indexer O(max_context) bottleneck for ~17× throughput gain. PD disaggregation was deployed with systemd services, Prometheus/Grafana monitoring was set up from scratch, and tool-calling quality issues were resolved.

The final throughput reached 531 tok/s — a dramatic improvement from the initial 11 tok/s.

The Custom MMA Sparse-MLA Decode Kernel

The centerpiece of the DeepSeek-V4 optimization was a custom MMA (Matrix Multiply-Accumulate) sparse-MLA decode kernel. The assistant designed the kernel with split-K parallelization, where the attention computation is split across multiple thread blocks that each compute a partial sum, then combined via a final reduction. This approach maximized occupancy on SM120's constrained shared memory budget.

The kernel was capture-safe for CUDA graphs, meaning it could be compiled into a static graph that eliminates CPU-side launch overhead. The assistant verified that the kernel produced correct results by comparing against a reference Triton implementation, then deployed it into the live SGLang service.

The Indexer O(max_context) Fix

The most impactful single fix in the DeepSeek-V4 deployment was the indexer O(max_context) bottleneck. The DSA indexer's attention computation scaled linearly with the maximum context length, regardless of how many tokens were actually active. For a 512K context, this meant the indexer was computing attention over 512K positions even when only 1 token was being decoded.

The assistant fixed this by capping the context length to 8192 in the indexer path, reducing the indexer's compute from O(512K) to O(8K) — a 64× reduction. The fix required modifying both the Triton kernel and the Python-level indexer logic to support an early-exit per page mechanism. The result was a ~17× end-to-end throughput improvement.

The Production Monitoring Stack

The assistant built a complete Prometheus and Grafana monitoring stack from scratch. This included:

• GPU metrics exporter: Custom exporter that scraped nvidia-smi output and exposed it as Prometheus metrics
• vLLM metrics integration: Configured vLLM's built-in Prometheus endpoint for request-level metrics (throughput, latency, error rates)
• Grafana dashboard: A 17-panel KV-cache dashboard showing cache utilization, hit rates, eviction patterns, and memory pressure
• Node health monitoring: CPU, memory, disk, and network metrics for the host system
• HiCache monitoring: Hierarchical cache metrics for the two-tier KV cache system The monitoring stack was essential for diagnosing production incidents, including the PD deadlock, the NIXL bootstrap_thread wedge, and the bf16 index-K corruption.

The bf16 Index-K Bug

The deployment encountered a multi-turn context-loss failure that required deep debugging. The assistant diagnosed DSA sparse attention recall failure on long contexts, fixed by increasing index_topk to 1024 and implementing bf16 index keys in the fused CUDA kernel.

But the bf16 index-K patch introduced a high-concurrency tool-call corruption bug. The assistant root-caused the issue to a multi-stream-overlap race condition under CUDA graph capture. The fix was to disable multi-stream overlap, which resolved the corruption but limited throughput scaling.

The production deployment encountered a PD deadlock, resolved by disabling overlap schedule. A mass-abort wedge in NIXL bootstrap_thread was fixed. Pool sizing issues were corrected. The assistant conducted A/B tests to isolate the corruption to the bf16 index-K path, and ultimately identified that the decode-side bf16 index-K handling was the trigger.

The Production Engineer's Playbook

The production stabilization phase was a masterclass in incident response. The assistant fixed a PD deadlock by disabling overlap schedule, root-caused high-concurrency tool-call corruption, fixed a mass-abort wedge in NIXL bootstrap_thread, and corrected pool sizing. The bf16 index-K corruption was ultimately fixed by disabling multi-stream-overlap.

The assistant investigated decode throughput scaling to C90, evaluated Two-Batch Overlap feasibility, and A/B tested overlap scheduler on decode. A TP-collective desync hazard was identified and fixed. The production PD bootstrap incident was resolved with a co-restart procedure.

The session concluded with the restoration of the SGLANG_SM120_MMA_TARGET_CTAS=512 knob after the user identified a faulty client-side proxy as the true cause of a multi-round harness hang. A full PD co-restart was performed, and the documentation was corrected to reflect the actual root cause.

The Proxy That Wasn't There

One of the most remarkable debugging episodes in the entire session was the "proxy that wasn't there." The assistant had reverted the SGLANG_SM120_MMA_TARGET_CTAS=512 knob after suspecting it caused a multi-round harness hang. But the user, through careful investigation, identified that a faulty client-side proxy was the true cause. The assistant restored the knob, performed a full PD co-restart, and corrected the documentation.

This episode exemplifies the importance of not jumping to conclusions about root causes. The assistant's initial diagnosis — blaming the kernel parameter — was reasonable given the symptoms, but it was wrong. The user's deeper investigation revealed the true cause in an entirely different layer of the system. The willingness to correct the record and update the documentation is a hallmark of engineering integrity.

The Enduring Lessons

The Full-Stack Nature of ML Infrastructure

This session demonstrates that deploying frontier LLMs is not a single-discipline activity. It requires expertise spanning hardware driver installation, CUDA toolkit management, Python environment configuration, compiled CUDA extension debugging, dependency resolution, hypervisor topology analysis, kernel-level performance optimization, profiler-driven bottleneck discovery, custom attention backend development, speculative decoding pipeline construction, and production-grade service deployment.

The assistant navigated all of these domains, often within a single segment. The ability to move from kernel module parameters to HuggingFace transformers versions to JIT compilation build tools — and to understand how each layer interacts with the others — is the defining characteristic of effective ML infrastructure engineering.

The Primacy of Methodology

Throughout the session, the assistant maintained a consistent methodology: form a hypothesis, design an experiment, run it, interpret the results honestly, and iterate. This discipline is what enabled progress even when individual experiments failed. The OEA null result was not a setback — it was knowledge gained. The 3.4% efficiency revelation was not a failure — it was a forcing function that redirected the investigation toward the kernel level.

The most important tool in the assistant's arsenal was not a debugger or a profiler but the ability to read source code — on the remote machine, on GitHub, and in the local filesystem. Understanding the Triton decode kernel's iteration pattern, the SparseMLAAttentionImpl class hierarchy, and the maybe_override_with_speculators function's call site were all achieved through careful code reading.

The Hardware Ceiling

Perhaps the most important lesson from this session is the recognition of hardware-imposed limits. The assistant explored every plausible optimization path: CUDAGraph, NCCL protocol tuning, custom allreduce, allreduce-RMS fusion, pipeline parallelism, expert parallelism, speculative decoding. Each was evaluated against profiling data, and each was either adopted or rejected based on empirical results.

The final ceiling of ~57 tok/s for GLM-5, ~61 tok/s for NVFP4 Kimi-K2.5, and ~4,000 tok/s for MiniMax-M2.5 were not failures of software optimization but physical constraints of the PCIe-only GPU topology. The assistant's honest assessment — documenting what was and wasn't possible — demonstrated engineering integrity over wishful thinking.

The Value of Negative Results

The most striking feature of this session is the assistant's willingness to accept negative results. Piecewise CUDA Graphs was blocked by a framework incompatibility. MSCCLPP delivered ~2%. SBO delivered ~2%. EP8 regressed low-concurrency performance and crashed under load. The OEA experiment produced a clean null result. EAGLE-3 speculation was net-negative until CUDA 13.

Each of these results could have been rationalized away — "maybe if I tweak the parameter," "maybe with a different backend," "maybe with more memory." Instead, the assistant accepted each result, documented it, and moved to the next hypothesis. This discipline is rare and valuable. In optimization work, the sunk cost fallacy is a constant temptation, and the assistant resisted it consistently.

The Power of Parallel Exploration

The assistant's use of parallel subagent tasks was a recurring pattern that dramatically accelerated the work. When faced with a complex problem — understanding three codebases simultaneously, auditing ten system layers, researching six optimization approaches — the assistant dispatched independent subagents that ran concurrently. The wall-clock time was roughly the duration of the longest-running agent, compressing what could be hours of sequential investigation into a single round.

This pattern was enabled by opencode's task tool, which spawns independent subagent sessions that run to completion concurrently. The parent session blocks until all subagents return, but the parallelism within each round is substantial.

The User-Assistant Partnership

Throughout the session, a productive partnership emerged between the user and the assistant. The user provided high-level direction ("improve to at least 100/s," "productionalise into systemd service," "add this gguf support to vllm") while the assistant executed with autonomy and thoroughness. The user's single-sentence messages at critical junctures demonstrated trust and effective delegation.

This partnership worked because both parties understood their roles. The assistant drove the technical investigation, formulated hypotheses, ran experiments, and reported results. The user set priorities, cut off diminishing-returns exploration, and provided domain knowledge. The result was a collaboration that achieved more than either could have alone.

Conclusion

The opencode session spanning 75 segments is a landmark in ML infrastructure engineering. It traces the complete lifecycle of deploying frontier large language models on cutting-edge hardware, from bare-metal driver installation through production-grade service deployment. The models deployed ranged from GLM-5-NVFP4 (744B) through Kimi-K2.5 (1T), MiniMax-M2.5 (230B active), Qwen3.5-397B, DeepSeek-V4-Flash, and many others. The inference engines evolved from SGLang to vLLM and back again, with custom patches applied to both.

The session's achievements are substantial: a 29% improvement from the gather-then-cast patch, a 2.85× improvement from CUDAGraph and NCCL tuning, a 4,000 tok/s peak throughput with MiniMax-M2.5, a custom Triton MLA sparse attention backend for Blackwell GPUs, a complete EAGLE-3 training and deployment pipeline, a DFlash DDTree deployment with 2.15× speedup, a custom sm_120 verify attention kernel, and a full PD disaggregation deployment with monitoring.

But the true legacy of this session is not the specific configurations chosen or the throughput numbers achieved. It is the methodology demonstrated: systematic hypothesis testing, profiling-first optimization, parallel exploration, honest acceptance of negative results, and the relentless pursuit of understanding at every layer of the stack. These are the skills that separate effective ML engineering from trial-and-error configuration.

The session stands as a testament to the challenges of deploying at the frontier, where every component — model architecture, quantization format, serving framework, GPU hardware, and even the virtualization layer — is evolving simultaneously. The only way forward is systematic, iterative debugging informed by local empirical knowledge, with the humility to recognize that the problem may not be where you're looking.