From Garbage to Production: Debugging, Optimizing, and Deploying GLM-5 on 8× Blackwell GPUs

Introduction

In the span of roughly eighty messages — from message 1952 to message 2035 — a single opencode coding session traversed the entire lifecycle of a production ML deployment: from debugging incoherent model output, through performance optimization, to final productionalization as a systemd service. The subject was GLM-5, a 744-billion-parameter Mixture-of-Experts model quantized to GGUF Q4_K_XL format, running on eight NVIDIA RTX PRO 6000 Blackwell GPUs connected only via PCIe Gen5. The journey was anything but linear. It involved two correctness bugs, a misdiagnosis, a 2.85× throughput improvement, a hardware ceiling, and a production deployment that initially failed due to a stale process and a Triton kernel import error.

This article synthesizes the entire arc of this chunk, tracing the narrative from the first comprehensive status report through the final systemd service deployment. It examines the key decisions, the assumptions that guided them, the mistakes that were corrected, and the knowledge that emerged. The story is one of systematic engineering: methodical debugging, profiling-first optimization, and disciplined productionalization — all within a single AI-assisted coding session.

Part I: The Debugging Crisis — Two Bugs That Silenced a 402-Billion-Parameter Model

The Great Consolidation

The chunk opens with message 1952, a massive 4,000-word status report that the assistant produced at a critical inflection point in the debugging process. The GLM-5 model had been loaded successfully onto eight Blackwell GPUs after weeks of effort — environment setup, driver installation, flash-attn compilation, GGUF file merging, and extensive patches to vLLM's gguf_loader.py, weight_utils.py, and deepseek_v2.py. Yet every request returned incoherent text: tokens like "IRS", "BW", "Promo", "Version" repeated in patterns that looked nothing like coherent language. Logprobs analysis showed known continuation tokens with scores around -20 to -24 when they should have been near zero.

The status report catalogued everything: hardware configuration (8× RTX PRO 6000 Blackwell, SM120, 96GB VRAM each, NO NVLink), software versions (vLLM 0.16.0rc2.dev313, PyTorch 2.10.0, Triton 3.6.0, CUDA 12.8), every bug already fixed (the kv_b_proj weight not being loaded from GGUF files, a global string replacement bug that corrupted parameter names, force-dequantization issues), and the central hypothesis: a tensor parallelism (TP) sharding mismatch for the kv_b_proj weight.

The user's response at message 1953 was a single sentence: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed." This quiet authorization — a masterclass in effective delegation — unlocked the debugging odyssey that followed.

The Hypothesis That Was Wrong

The assistant committed to the TP sharding hypothesis in message 1954. The reasoning was sound: the kv_b_proj parameter had shape [28672, 512] (full, unsharded), but with TP=8, each GPU should receive a shard of shape [3584, 512]. The GGUF loading path for force-dequantized weights was not applying TP sharding. The assistant launched a subagent task in message 1955 to trace through vLLM's source code and understand how ColumnParallelLinear.weight_loader handles GGUF weights.

The subagent's analysis returned a crucial finding: for quantized GGUF weights, each rank does load the full-size tensor, and the dequantization kernel handles sharding implicitly during computation. TP sharding for GGUF weights is computational, not storage-based. This meant the TP sharding hypothesis was unlikely to be the root cause.

Over the next fifteen messages (1956–1970), the assistant systematically investigated and eliminated hypothesis after hypothesis: weight name mappings, kv_b_proj reassembly shape, RoPE interleave configuration, expert weight shapes, fused projection loading, dequantization kernel correctness on SM120, and GGUF block alignment with TP=8. Each investigation was methodical, each ruled out a potential cause, and each narrowed the search space.

The Breakthrough

The breakthrough came in message 1970 with a simple diagnostic test: setting VLLM_MLA_DISABLE=1 — which forces vLLM to use standard FlashAttention instead of the optimized Triton MLA backend — produced correct output. This was the first clear signal that the bug was not in the weight loading code at all, but in the attention computation path.

Messages 1971–1974 narrowed the problem to an output buffer disconnect in the Triton MLA attention backend. The forward_mha function was computing correct attention values, but the wrapper code was reading all zeros. The culprit was a custom PyTorch operator (torch.ops.vllm.unified_mla_attention_with_output) that created a "phantom tensor" internally during dispatch. The forward_impl function wrote correct values to this phantom tensor, but the caller's output buffer — allocated before the call — was never written to. The fix was to bypass the custom op entirely and call forward_impl directly.

But fixing this bug revealed a second one. The model still produced gibberish. In message 1976, the assistant dispatched a subagent to debug the remaining garbage, and the root cause was found: a shard ordering bug in GGUFLinearMethod.apply(). For fused projections like fused_qkv_a_proj, the method iterated over shard IDs in GGUF file load order rather than canonical order. When KV shards happened to be stored before Q shards in the file, the fused output had KV columns first and Q columns second — but the downstream MLA code assumed Q came first. The fix was a one-line change: for idx in sorted(shard_id).

The Validation

With both bugs fixed, the assistant ran a comprehensive benchmark in message 1981. The results were clean: 300/300 requests succeeded with zero errors at 1,024-way concurrency, delivering 1,049 tok/s peak throughput. The single-request decode throughput was ~20 tok/s — correct but slow. Message 1983 provided a complete summary of the debugging session, documenting both bugs, the ruled-out investigations, and the benchmark results.

Part II: The Optimization Sprint — From 20 to 57 Tok/s

The Demand

The user's response at message 1984 was terse and unambiguous: "20t/s is still really slow, gat you profile and try to improve to at least 100/s." This single sentence pivoted the entire session from debugging to optimization. The assistant immediately computed the per-token latency (~50ms per decode step), recognized that this was anomalously slow for 8 GPUs, and committed to a profiling-first approach.

Profiling the Bottleneck

Message 1986 dispatched a subagent to profile single-request decode performance. The results were precise and actionable: 50.5ms per decode step, 19.8 tok/s, 91% GPU utilization. The subagent's comprehensive analysis revealed that NCCL allreduce consumed 42% of decode time — 157 separate allreduce calls per step, each taking ~135μs, totaling ~21ms. The GPUs were connected via PCIe Gen5 without NVLink, making inter-GPU communication inherently expensive.

CUDAGraph: The First Major Win

The assistant's first optimization target was CUDAGraph, PyTorch's mechanism for capturing GPU operations into a replayable graph. But there was a complication: earlier in the debugging phase, CUDAGraph had produced garbage output. The assistant had attributed this to the direct-call patch that bypassed the custom MLA op.

In a critical re-examination (messages 1991–1993), the assistant discovered that the shard ordering fix was the real root cause all along — the direct-call patch was never needed. With the shard ordering fix applied, the upstream custom op path worked correctly, and CUDAGraph was now viable. The result was a 2.15× improvement: single-request throughput jumped from 20 tok/s to 43 tok/s.

The benchmark comparison (message 1997) told a nuanced story. CUDAGraph delivered 2× improvement at low concurrency where NCCL overhead dominated, but at high concurrency the benefit shrank to near-zero because batching already amortized kernel launch overhead. The remaining ~23ms per decode step was characterized as "pure GPU compute — memory-bandwidth-bound GGUF matmuls through 78 layers."

NCCL Protocol Tuning: The Second Win

The assistant then turned to NCCL tuning. Message 2006 dispatched a subagent to investigate NCCL protocol options, and message 2007 reported the finding: NCCL_PROTO=LL (Low Latency protocol) yielded a 34% improvement, pushing throughput from 43 tok/s to 57 tok/s. The LL protocol is optimized for small message sizes, and each of the 158 allreduce calls per step carried only ~12KB of data — exactly the regime where LL excels.

The Hardware Ceiling

The assistant explored more aggressive optimizations: custom allreduce via shared memory (broken on PCIe with more than 2 GPUs), allreduce-RMSNorm fusion using flashinfer (requires NVSwitch multicast hardware, unavailable on PCIe), and pipeline parallelism combined with tensor parallelism (PP=2+TP=4 to keep allreduces NUMA-local). Each was systematically investigated and found either incompatible with the PCIe-only topology or yielding zero additional benefit.

Message 2011 captured the moment of reckoning: "The flashinfer allreduce fusion requires NVSwitch/multicast hardware — not available on PCIe GPUs. And all other NCCL tuning beyond NCCL_PROTO=LL shows zero additional benefit. We're at 57.6 tok/s and the theoretical ceiling with zero-cost allreduce would be 100-140 tok/s. The remaining ~11ms of allreduce can't be eliminated on this hardware."

This was the hardware ceiling — a physical constraint imposed by the PCIe Gen5 interconnect, not a software bug to be fixed or a knob to be tuned. The assistant had established the practical performance envelope for this configuration.

Part III: The Productionalization — From Optimization to Systemd Service

The Directive

The user's message at 2013 cut the optimization exploration short: "No keep this config for now, productionalise into vllm-glm5 systemd service." The assistant accepted the directive without argument and immediately pivoted to productionalization.

Message 2014 laid out a four-step plan: verify the current server configuration, document all patched files, create the systemd service unit, and enable and test the service. The assistant then executed a series of verification commands (messages 2015–2020): checking running processes, confirming systemd availability, auditing all patched files, verifying the 402GB GGUF model file, and confirming the Python environment.

The Service Creation

Message 2021 marked the transition to service creation. The assistant wrote a systemd unit file (vllm-glm5.service) that encapsulated the exact launch command with all flags (--tensor-parallel-size 8, --dtype float16, --max-model-len 8192, --gpu-memory-utilization 0.90, --trust-remote-code, --port 8000, --disable-log-requests), the critical environment variable (NCCL_PROTO=LL), and the working directory. The unit was copied to the server, enabled, and started.

The Startup Failure

The first start attempt failed. Message 2025 showed Active: inactive (dead) with no journal entries — a silent failure. The user diagnosed the problem in message 2029: "Seems previous vllm is still live preventing sysd one from starting." A stale vLLM process from a previous session was still occupying GPU memory and port bindings.

The assistant killed the stale process and tried again. Message 2032 showed a new failure: WorkerProc initialization failed. Looking further up in the journal (message 2033), the assistant discovered the real error: a Triton kernel import error — cannot import name 'SparseMatrix' from 'triton_kernels.tensor'. This was a version incompatibility between the installed triton_kernels package and what vLLM's MoE kernel module expected.

The assistant resolved the import error (likely by installing a compatible version or patching the import), and the service finally started successfully. The final messages in the chunk show the assistant verifying the service status and confirming that the model was loading correctly.

Themes and Lessons

Systematic Debugging Methodology

The debugging phase of this chunk is a textbook example of hypothesis-driven investigation. The assistant formulated explicit hypotheses, designed experiments to test them, and documented the results. When the TP sharding hypothesis was ruled out, the assistant did not persist with it — it moved on to the next candidate. When the MLA output buffer bug was fixed but the model still produced garbage, the assistant recognized that there must be a second independent bug rather than assuming the first fix was wrong.

The list of ruled-out investigations in message 1983 — nine items covering weight mapping, TP sharding, RoPE configuration, dequantization kernels, and more — represents dozens of hours of work. Each item was a hypothesis that could have been the root cause, and each was systematically eliminated. This is the essence of scientific debugging: proving what is not wrong is as important as proving what is wrong.

The Two-Bug Problem

A key insight from this chunk is that complex systems often fail due to multiple interacting bugs. The assistant initially found two separate bugs (MLA output buffer disconnect and GGUF shard ordering) and applied two separate fixes. Only later did it realize that one of the "bugs" — the phantom tensor in the custom op dispatch — was actually a misdiagnosis. The shard ordering bug was the true root cause; the MLA output buffer appeared to be all zeros because the attention computation received misaligned Q/KV columns.

This is a classic debugging pitfall: when two bugs co-occur, each can mask or mimic the other. The assistant's willingness to re-examine its earlier diagnosis — to admit that the direct-call patch was never needed — was essential to reaching the correct understanding.

Profiling-First Optimization

The optimization phase demonstrated the power of profiling-first methodology. Rather than guessing at optimizations, the assistant invested in measurement — dispatching a subagent to profile the decode step at the kernel level. The resulting insight that NCCL allreduce consumed 42% of decode time shaped the entire optimization strategy. CUDAGraph was chosen because it batches NCCL calls. NCCL_PROTO=LL was chosen because it optimizes for small message sizes. Each optimization was targeted at a specific, measured bottleneck.

This approach contrasts with the common pattern of applying random optimizations and hoping for improvement. The assistant's methodical approach — measure, identify, fix, measure again — is the gold standard for performance engineering.

The Hardware Ceiling

Perhaps the most important lesson from this chunk is the recognition of hardware-imposed limits. The assistant explored every plausible optimization path: CUDAGraph, NCCL protocol tuning, custom allreduce, allreduce-RMS fusion, pipeline parallelism. Each was evaluated against the profiling data, and each was either adopted or rejected based on empirical results. The final ceiling of ~57 tok/s was not a failure of software optimization but a physical constraint of the PCIe-only GPU topology.

The assistant's honest assessment in message 2011 — "The remaining ~11ms of allreduce can't be eliminated on this hardware" — is a model of engineering integrity. It would have been easy to continue chasing diminishing returns, to promise that one more optimization would crack the 100 tok/s barrier. Instead, the assistant documented the ceiling, quantified the theoretical maximum, and presented the user with a clear picture of what was and wasn't possible.

Production Thinking

The productionalization phase revealed a different kind of engineering discipline. The assistant's first step was not to write the systemd unit file, but to verify the current state: check running processes, audit patched files, confirm the model file exists, verify the Python environment. The "Document all patched files" step showed production thinking — recognizing that the ad-hoc patches to vLLM's installed packages were fragile and needed documentation for maintainability.

The startup failures that followed — stale process, Triton kernel import error — are the kind of edge cases that only emerge when you transition from development-mode experimentation to production-grade service management. The assistant's systematic approach to diagnosing these failures (checking journalctl, filtering for error patterns, tracing the "WorkerProc initialization failed" error to its root cause) demonstrated the same methodological rigor that characterized the debugging phase.

Conclusion

This chunk of the opencode session is a microcosm of the entire ML deployment lifecycle. It begins with a model that loads successfully but produces incoherent garbage — a deeply frustrating situation that could easily lead to random changes and wasted effort. Instead, the assistant applies systematic debugging methodology, ruling out hypothesis after hypothesis until two root causes are identified and fixed.

With correctness achieved, the focus shifts to performance. Profiling reveals the NCCL allreduce bottleneck, and targeted optimizations — CUDAGraph, NCCL_PROTO=LL — deliver a 2.85× improvement from 20 to 57 tok/s. The hardware ceiling is recognized and documented.

Finally, the configuration is productionalized into a systemd service. The initial deployment fails due to a stale process and a Triton kernel import error, but each failure is diagnosed and resolved systematically. The service starts, the model loads, and the deployment is complete.

What makes this chunk remarkable is not just the technical achievements — fixing two bugs, optimizing throughput by 2.85×, deploying a production service — but the methodology that underpins them. The assistant's approach is consistent across all three phases: form hypotheses, test them with targeted experiments, interpret the results honestly, and iterate. This is the essence of engineering discipline, and it is on full display in every message of this chunk.