The Final Wait: Productionalizing GLM-5 on Blackwell GPUs

"Service is running. Now let me wait for it to finish loading the model (~7 min for weights + CUDAGraph compilation)."

At first glance, message [msg 2028] appears to be one of the most mundane moments in any software deployment: a health-check polling loop, waiting for a server to come online. The assistant runs a bash command that curls http://localhost:8000/health every five seconds for up to fifteen minutes, printing progress updates every minute. There is no debugging, no code change, no clever optimization — just patience. Yet this message is anything but mundane. It is the quiet culmination of a journey that spanned dozens of hours, multiple failed approaches, deep kernel debugging, and the bending of open-source software to support a bleeding-edge model on bleeding-edge hardware.

To understand why this simple polling loop matters, we must understand everything that led to it.

The Road to This Moment

The conversation leading to [msg 2028] is a case study in the brutal reality of deploying large language models on novel hardware. The GLM-5 model — a 744-billion-parameter mixture-of-experts architecture from Zhipu AI — was to be deployed on eight NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs connected only by PCIe Gen5, with no NVLink or NVSwitch. This hardware topology would prove to be the single most consequential constraint on the entire deployment.

The journey began with the NVFP4 quantization path using SGLang, which was abandoned after the user decided to pivot to a GGUF quantized model using unsloth's UD-Q4_K_XL format and vLLM as the inference engine. This pivot triggered an extraordinary engineering effort: writing comprehensive patches for vLLM's gguf_loader.py to support the novel glm_moe_dsa architecture, fixing a latent DeepSeek V2/V3 bug in the kv_b_proj mapping, building llama-gguf-split from source to merge ten split GGUF files into a single 402 GB file, and implementing a new Triton MLA sparse attention backend for Blackwell's SM120 compute capability.

When the model finally loaded, it produced incoherent garbage. The debugging that followed was methodical and relentless: ruling out weight loading, tensor parallelism sharding, and configuration issues before isolating the root cause to two bugs in vLLM's Triton MLA attention backend. One was an output buffer disconnect caused by a custom PyTorch op creating a phantom tensor; the other was a shard ordering bug in the GGUF dequantization layer for fused projections. Fixing these restored correct output.

The Performance Ceiling

With correct output achieved, the assistant turned to performance optimization. Single-request decode was initially ~20 tok/s in eager mode. Profiling revealed that ~42% of the time was spent on NCCL allreduce calls — the penalty of tensor parallelism across eight GPUs connected only by PCIe. Enabling CUDAGraph doubled throughput to ~43 tok/s by batching kernel launches and eliminating Python dispatch overhead. Tuning NCCL_PROTO=LL — the low-latency protocol for small message sizes — further boosted throughput to ~57 tok/s, a 34% improvement.

But 57 tok/s was still far from the user's aspirations. The assistant explored every remaining avenue: custom allreduce (broken on PCIe with more than two GPUs due to a C++ kernel bug), allreduce-RMSNorm fusion (impossible without NVSwitch multicast hardware), pipeline parallelism to keep allreduces NUMA-local (theoretically promising but requiring a full server restart and model reload). Each optimization was systematically evaluated and either implemented or ruled out with clear evidence.

The user made a pragmatic decision in [msg 2013]: "No keep this config for now, productionalise into vllm-glm5 systemd service." This was the moment the optimization phase ended and the productionalization phase began. The assistant was being told to stop chasing the theoretical ceiling and ship what worked.

Creating the Systemd Service

The assistant dutifully pivoted. It verified the state of all patches on the container ([msg 2018]), confirmed the model file existed at /shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf (402 GB), checked GPU count and CUDA version, and wrote a systemd service unit file ([msg 2022]).

The service file itself encoded all the accumulated knowledge from the optimization phase. It set NCCL_PROTO=LL for the low-latency protocol. It configured VLLM_HOST=127.0.0.1 to bind to localhost. It specified tensor parallelism of 8, the GGUF load format, the ngram speculative decoding with 5 tokens, and the full CUDAGraph mode. It set CUDA_DEVICE_MAX_CONNECTIONS=1 to prevent NCCL connection timeouts. Every environment variable and command-line flag was the product of empirical testing.

The first deployment attempt failed. The service showed as inactive (dead) with no journal entries ([msg 2025]-[msg 2026]). The assistant diagnosed this as a stale vLLM process from a previous session interfering with the new one. After killing the old process, re-enabling, and restarting, the service showed as active (running) ([msg 2027]).

The Meaning of the Wait

This brings us to [msg 2028]. The service is running, but the model is not yet ready. Loading a 402 GB GGUF file across eight GPUs and compiling CUDAGraphs takes approximately seven minutes. The assistant's response is to wait — not passively, but with a structured polling loop that will check every five seconds and report progress every minute.

The message is notable for what it does not contain. There is no debugging command, no patch, no configuration change. The assistant has exhausted its repertoire of interventions. All the bugs have been fixed, all the optimizations applied, all the configuration validated. The only remaining task is to let the software do its work.

This represents a fundamental shift in the assistant's role. Throughout the conversation, the assistant was an active agent of change — writing code, running experiments, diagnosing failures. Now it becomes an observer, monitoring a process it can no longer accelerate. The seven-minute load time is a hard constraint: the GGUF file must be read from disk, dequantized, distributed across eight GPUs, and the CUDAGraph must capture the entire forward pass. No amount of cleverness can make this faster.

Assumptions and Knowledge

The message makes several assumptions that are worth examining. First, it assumes the health endpoint will eventually return success. This is not guaranteed — the model could fail to load due to an out-of-memory error, a tensor shape mismatch, or any number of runtime issues that only manifest during actual loading. The polling loop will catch a failure only by timeout after fifteen minutes, at which point the assistant would need to inspect the service logs to diagnose the problem.

Second, the assistant assumes that curl -s http://localhost:8000/health is the correct health check for vLLM. This is a reasonable assumption — vLLM exposes a /health endpoint that returns a 200 status when the server is ready — but it depends on the server having started its HTTP listener before model loading completes. If vLLM's architecture defers listener startup until after model loading, the health check would return connection refused rather than a non-200 response, and the grep -q "ok\|200" pattern would correctly detect this as "not ready."

Third, the assistant assumes a seven-minute load time based on previous experience with this model and hardware. This estimate encompasses both weight loading (reading and dequantizing 402 GB from disk) and CUDAGraph compilation (capturing the full model graph across eight GPUs). The fifteen-minute timeout provides a generous margin.

The input knowledge required to understand this message is substantial. One must know that vLLM serves models via an HTTP API with a /health endpoint. One must understand that GGUF models must be loaded and dequantized before inference can begin, and that CUDAGraph compilation is a separate, expensive step that happens after weight loading. One must know the hardware configuration (eight Blackwell GPUs, PCIe-only interconnect) and the model size (402 GB GGUF, 744B parameters) to appreciate why loading takes seven minutes. And one must understand the entire history of patches, bug fixes, and optimizations that preceded this moment to recognize why the assistant's only remaining action is to wait.

Output Knowledge Created

The message itself creates no new knowledge in the traditional sense — it does not produce benchmark results, log output, or code changes. But it does create a different kind of knowledge: the knowledge that the deployment is proceeding as expected. The polling loop is a verification mechanism. Each successful curl response that does not return "ok" tells the assistant that the server is still loading. The eventual successful response will confirm that all the patches, all the optimizations, and all the debugging have produced a working production service.

In a broader sense, the message creates operational knowledge about the deployment lifecycle of large models on this specific hardware configuration. The seven-minute load time, the health check mechanism, the polling interval — these become part of the operational playbook for the GLM-5 service.

The Thinking Process

The assistant's reasoning in this message is revealed through its structure. The opening line — "Service is running. Now let me wait for it to finish loading the model (~7 min for weights + CUDAGraph compilation)" — shows that the assistant has a mental model of the loading process and its duration. It knows that "running" does not mean "ready," and it knows approximately how long the gap will be.

The choice of a polling loop rather than a single sleep-and-check reflects an awareness that load times are variable. The assistant builds in a generous timeout (180 iterations × 5 seconds = 15 minutes) and provides periodic progress updates. The if [ $((i % 12)) -eq 0 ] condition prints a status message every minute, giving visibility into progress without flooding the output.

The use of curl -s with stderr redirected to /dev/null shows attention to clean output — only the HTTP response body will be visible. The grep -q "ok\|200" pattern is deliberately permissive, accepting either the literal string "ok" or an HTTP 200 status code in the response, covering variations in how different vLLM versions format their health check responses.

Conclusion

Message [msg 2028] is the final step in a journey that transformed a collection of hardware, software, and model artifacts into a production inference service. The polling loop is the last action the assistant takes before the service enters its steady state. Everything that could be optimized has been optimized. Every bug that could be fixed has been fixed. The only remaining variable is time.

In the broader narrative of the conversation, this message marks the transition from development to operations. The assistant's role shifts from builder to operator, from debugger to monitor. The health check loop is the bridge between these two phases — a moment of suspended judgment while the system proves itself.

The message is also a testament to the complexity of modern AI infrastructure. A single curl command, five lines of bash, represents the successful resolution of dozens of distinct challenges: driver installation, CUDA toolkit configuration, flash-attn compilation, GGUF patching, Triton kernel debugging, NCCL tuning, and systemd service configuration. Each of these challenges could have derailed the entire deployment. The fact that the assistant is waiting for a health check, rather than debugging yet another failure, is itself a significant achievement.