The Final Deployment: Productionalizing a 402GB GGUF Model on 8× Blackwell GPUs

A Single Command That Carried the Weight of a Marathon

In the sprawling, multi-session journey to deploy the GLM-5 744B model on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment that appears deceptively simple on the surface. Message [msg 2023] reads:

[assistant] Now deploy and test: [bash] scp /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-glm5.service root@10.1.230.174:/etc/systemd/system/vllm-glm5.service

One scp command. A single file copy. But this message represents the culmination of an extraordinary debugging and optimization odyssey — the moment when a research prototype is transformed into a production service. Understanding why this particular message was written, and what it meant in context, requires unpacking the entire arc of the session that led to it.

The Context: From Incoherent Output to 57 tok/s

The path to this message began with the model producing complete garbage. After successfully loading a 402GB GGUF quantized model onto eight Blackwell GPUs, the assistant discovered that the output was incoherent — not a configuration issue, not a weight loading problem, but two deeply subtle bugs in vLLM's custom inference stack. The first was a Triton MLA attention backend bug where a custom PyTorch op created a phantom tensor that disconnected the output buffer. The second was a shard ordering bug in the GGUF dequantization layer for fused projections, where the q, k, and v shards needed to be explicitly sorted to match the expected layout.

Fixing these bugs restored correct output, but the performance was abysmal: only ~20 tokens per second for single-request decoding. Profiling revealed that a staggering 87% of decode time was spent on NCCL allreduce operations — the communication overhead of synchronizing gradients and activations across 8 GPUs connected only via PCIe Gen5, with no NVLink or NVSwitch to accelerate peer-to-peer transfers.

The assistant then embarked on a systematic optimization campaign. Enabling CUDAGraph (vLLM's CUDA graph capture mechanism) doubled throughput to ~43 tok/s by batching kernel launches and eliminating Python dispatch overhead. But the NCCL allreduce remained the bottleneck. Discovering that the allreduce payloads were tiny (only ~12KB per operation), the assistant tuned NCCL_PROTO=LL — the low-latency protocol — which boosted performance to ~57 tok/s, a 34% improvement.

Further optimizations were explored but hit hard hardware walls. Custom allreduce (which uses shared memory instead of NCCL) was broken on PCIe systems with more than 2 GPUs due to a C++ kernel bug. Allreduce-RMS fusion (which would fuse the allreduce with the subsequent RMSNorm operation into a single flashinfer kernel) required NVSwitch multicast hardware that simply didn't exist on these PCIe-connected GPUs. The assistant calculated the theoretical ceiling: with zero-cost allreduce, the model could reach 100–140 tok/s, but the remaining ~11ms of allreduce latency was fundamentally inescapable on this hardware topology.

The User's Decision: Stop Optimizing, Start Serving

At this point, the assistant was preparing to explore pipeline parallelism (PP=2 combined with TP=4) to keep allreduces within a single NUMA node — a promising but complex reconfiguration. But the user intervened in [msg 2013]:

[user] No keep this config for now, productionalise into vllm-glm5 systemd service

This was a pivotal decision point. The user was explicitly choosing to freeze the configuration at 57 tok/s — accepting the PCIe bottleneck as a hard constraint — and prioritize reliability and production readiness over further optimization. This decision reflects a mature understanding of the deployment lifecycle: at some point, chasing the last 10–20% of performance must yield to the operational concerns of uptime, monitoring, and maintainability.

The Preparation: Verifying Every Piece Before Deployment

Before the scp command in [msg 2023] could be issued, the assistant performed a meticulous verification of the entire deployment environment across multiple messages ([msg 2015] through [msg 2022]). This preparation reveals the assistant's assumptions and reasoning about what constitutes a production-ready state.

First, the assistant checked whether the vLLM server was currently running ([msg 2015]). It was stopped — a clean slate for deployment. Then it verified every patched file in the vLLM installation:

The Service File: Encoding an Entire Optimization Journey

The systemd service file created in [msg 2022] and deployed in [msg 2023] is not just a simple launcher. It encodes the entire optimization journey in its configuration. From the later status checks ([msg 2027], [msg 2031]), we can reconstruct the service definition:

The Assumptions Embedded in the scp Command

The scp command in [msg 2023] carries several assumptions, some of which would prove incorrect:

  1. That the remote system has systemd properly configured: Verified earlier ([msg 2017]) — which systemctl returned /usr/bin/systemctl and ps -p 1 -o comm= returned systemd.
  2. That the service file path is correct: /etc/systemd/system/vllm-glm5.service is the standard location for systemd unit files.
  3. That no stale vLLM process would interfere: This assumption proved wrong. In [msg 2029], the user reported "Seems previous vllm is still live preventing sysd one from starting," and subsequent investigation ([msg 2031]) showed the service entering an auto-restart loop with WorkerProc initialization failed errors — likely caused by GPU memory or shared memory state left by a prior process.
  4. That the service would start cleanly on first attempt: The initial systemctl start in [msg 2024] resulted in an inactive state ([msg 2025]), requiring a second attempt with pkill to clear stale processes ([msg 2027]).
  5. That the NCCL_PROTO=LL environment variable would be inherited: The service file needed to explicitly set NCCL_PROTO=LL in its Environment directive or the systemd service would default to the standard NCCL protocol, losing the 34% throughput gain.

The Thinking Process: What the Assistant Was Reasoning

The assistant's reasoning in this message is best understood by examining the sequence of actions and the todowrite state. In [msg 2021], the assistant had marked "Create systemd service unit for vllm-glm5" as in_progress and "Enable and test the service" as pending. The service file was written locally in [msg 2022]. Message [msg 2023] executes the deployment and test phase.

The assistant's thinking appears to follow this logic:

  1. The service file exists locally — written to /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-glm5.service.
  2. It needs to be on the remote server — the model, GPUs, and vLLM installation are all on root@10.1.230.174.
  3. The correct location is /etc/systemd/system/ — standard systemd convention.
  4. After copying, enable and start — the next message ([msg 2024]) runs systemctl daemon-reload && systemctl enable && systemctl start.
  5. Then verify health — the subsequent messages check systemctl status and wait for the health endpoint. The assistant did not anticipate the stale process conflict, which is a common operational pitfall when deploying services on machines that have been used for interactive debugging sessions. The previous manual vLLM invocations left behind GPU state that the new systemd-managed process couldn't cleanly acquire.

The Broader Significance: From Debugging to Operations

Message [msg 2023] marks a transition in the lifecycle of this deployment. The preceding 70+ messages in this segment were dominated by debugging (fixing incoherent output), profiling (identifying the NCCL bottleneck), and optimization (CUDAGraph, NCCL_PROTO=LL). With this scp command, the assistant shifts from a development mindset to an operations mindset. The questions change from "How do we make it faster?" to "How do we make it stay running?"

This transition is non-trivial. It requires the assistant to encode all the ad-hoc optimizations discovered during development into a repeatable, automated configuration. The NCCL protocol tuning, the CUDAGraph mode, the tensor parallelism size, the model path — every detail that was previously set via interactive shell commands must now be baked into the service definition.

The fact that the first deployment attempt failed ([msg 2025]) is not a failure of the message itself but a realistic depiction of the deployment process. Systemd services interact with the operating system's process lifecycle, resource management, and logging infrastructure in ways that interactive shell sessions do not. The stale process conflict is precisely the kind of edge case that only emerges when you transition from "it works on my terminal" to "it works as a service."

Conclusion

Message [msg 2023] — a single scp command — is the keystone of an entire segment's work. It represents the moment when a fragile, interactively-tuned research deployment is committed to a production configuration. The assistant's careful pre-flight verification, the encoding of optimization parameters into the service file, and the subsequent debugging of deployment issues all flow from this one action. It is a reminder that in complex systems engineering, the most important commands are often the simplest ones — the ones that say "now we make this permanent."