The Final Commit: Productionalizing a 57 tok/s GLM-5 GGUF Deployment on 8× Blackwell GPUs

Message Overview

The subject message ([msg 2022]) is deceptively simple — a single tool call confirming that a systemd service file has been written to disk:

[assistant] [write] /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-glm5.service Wrote file successfully.

Beneath this laconic confirmation, the message is littered with LSP diagnostic noise — spurious import errors from unrelated Python files in the workspace. But the true weight of this message lies not in what it says, but in what it represents: the terminus of a grueling optimization odyssey, the acceptance of hard hardware-imposed ceilings, and the pivot from exploration to production.

Context: The Road to 57 tok/s

To understand why this message was written, one must trace the arc of the preceding hours. The assistant had been tasked with deploying the GLM-5 model — a massive Mixture-of-Experts architecture with Multi-Head Latent Attention (MLA) — on a machine equipped with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The deployment path had already been tortuous: the NVFP4 quantization path was abandoned in favor of GGUF, vLLM had to be patched in multiple places to support the novel glm_moe_dsa architecture, a Triton MLA attention backend had to be implemented from scratch for Blackwell's SM120 compute capability, and weight loading bugs had been systematically hunted down and fixed.

Once the model produced coherent output, the optimization phase began. The initial single-request decode throughput was a dismal ~20 tok/s with --enforce-eager. Enabling CUDAGraph doubled this to ~43 tok/s. Then came the breakthrough: profiling revealed that ~87% of decode time was spent on NCCL allreduce calls — 158 calls per step, each transferring tiny ~12 KB payloads over PCIe Gen5. The NCCL_PROTO=LL (low-latency protocol) setting boosted throughput to ~57 tok/s, a 34% improvement.

But 57 tok/s was still far from the 100+ tok/s target. The assistant embarked on a systematic exploration of remaining optimization avenues: custom allreduce (broken on PCIe with >2 GPUs), allreduce-RMSNorm fusion (impossible without NVSwitch multicast hardware), PP=2+TP=4 configurations (to keep allreduces NUMA-local), and various NCCL tuning parameters. Each avenue was explored and found to be a dead end on this hardware topology.

The User's Decision

At [msg 2013], the user made a decisive intervention:

No keep this config for now, productionalise into vllm-glm5 systemd service

This instruction carries multiple layers of meaning. First, it signals acceptance of the current performance ceiling — the user is satisfied enough with 57 tok/s to put it into production. Second, it represents a prioritization decision: further optimization effort would yield diminishing returns, and the marginal gains of chasing 100 tok/s are not worth the engineering time. Third, it reflects a pragmatic understanding that the hardware topology (PCIe-only, no NVLink/NVSwitch) imposes a fundamental bound on allreduce performance that no amount of software tuning can fully eliminate.

The Assistant's Response: Methodical Productionalization

Message 2022 is the assistant's response to this directive. But the message itself is just the capstone of a multi-step verification process that preceded it. In the messages leading up to 2022 ([msg 2014] through [msg 2021]), the assistant systematically:

1. Verified the server state: Checked that no vLLM process was currently running (it had been stopped).
2. Confirmed systemd availability: Verified that the container uses systemd as its init system and that systemctl is available.
3. Audited all patches: Grepped through every patched file in the vLLM installation to confirm that critical fixes were still in place — the GGUF shard ordering fix (gguf.py), the GLM-DSA architecture support (gguf_loader.py), the force-dequantization logic (weight_utils.py), and the Triton MLA attention backend (mla_attention.py).
4. Verified the model artifact: Confirmed the 402 GB GGUF file exists at /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf and that the Hugging Face cache for the tokenizer is intact.
5. Checked the software stack: Confirmed Python 3.12.3, CUDA 12.8, and vLLM 0.16.0rc2.dev313+g662205d34. Only after this thorough inventory did the assistant write the service file. The LSP errors that accompany the success message are harmless — they come from decode_latency_breakdown.py and decode_gap_analysis.py, profiling scripts in the local workspace that can't resolve torch imports because the LSP is running outside the container's Python environment. The assistant correctly ignores them.

What the Message Does Not Say

The message does not reveal the contents of the service file itself. Based on the context, it would include:

• The vLLM API server command with all optimized flags: --model /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf, --tensor-parallel-size 8, --dtype float16, --max-model-len 8192, --gpu-memory-utilization 0.90, and the critical NCCL_PROTO=LL environment variable.
• The Python virtual environment path (/root/ml-env/bin/python3).
• Standard systemd service boilerplate: [Unit], [Service], [Install] sections, Restart=on-failure, User=root, etc. The subsequent messages ([msg 2023] and [msg 2024]) show the assistant deploying the service via scp, then enabling and starting it — but not without first killing any stale vLLM process that might conflict with port binding.

Assumptions and Knowledge Boundaries

This message assumes substantial context that a reader must possess to understand its significance:

• The hardware topology: 8 GPUs connected only via PCIe Gen5, with no NVLink or NVSwitch. This is the single most important constraint shaping the performance ceiling.
• The optimization history: The journey from 20 → 43 → 57 tok/s through CUDAGraph and NCCL_PROTO=LL, and the dead ends encountered beyond.
• The patching history: vLLM does not natively support the GLM-5 architecture in GGUF format; the assistant had to patch gguf_loader.py, weight_utils.py, gguf.py, and mla_attention.py to make it work.
• The user's authority: The user's instruction to stop optimizing and productionalize is accepted without question — the assistant does not argue for further optimization.

The Thinking Process

The assistant's reasoning is visible primarily in the surrounding messages rather than in message 2022 itself. The pattern is one of methodical due diligence: before writing a single line of production configuration, verify every dependency. The todo list in [msg 2021] shows the assistant's mental model:

1. Check current running server config and verify it's correct ✅
2. Document all patched files on the container ✅
3. Create systemd service unit for vllm-glm5 🔄 (in progress)
4. Enable and test the service ⏳ (pending) This structured approach — verify, document, create, test — is characteristic of production engineering. The assistant is not just writing a file; it is ensuring that when the service starts, it will work correctly with all patches intact, the right model file, and the optimized environment variables.

Conclusion

Message 2022 is a milestone. It marks the transition from a development project — patching vLLM, debugging attention kernels, profiling NCCL allreduce — to a production service. The 57 tok/s throughput, while below the aspirational 100 tok/s target, represents the practical optimum for this hardware configuration. The systemd service file written in this message will become the entry point for serving GLM-5 to users, running as a managed daemon that survives reboots and crashes.

The message is also a testament to the iterative nature of ML infrastructure engineering. What began as a quest to deploy a cutting-edge model on novel hardware ended not with a bang of breakthrough performance, but with the quiet satisfaction of a service file written to disk — and the pragmatic acceptance that sometimes, good enough is exactly right.