From Bare Metal to Benchmark: Deploying GLM-5 Across Eight Blackwell GPUs

Introduction

The gap between a working ML model and a production-ready inference deployment is vast — a chasm filled with driver incompatibilities, build system failures, kernel bugs, memory leaks, and optimization dead ends. This article traces a single opencode session that traversed that entire gap: starting from a bare Ubuntu 24.04 machine with two RTX PRO 6000 Blackwell GPUs, navigating the treacherous waters of CUDA toolkit versioning and flash-attention compilation, debugging two critical correctness bugs in vLLM's attention backend, and ultimately deploying the GLM-5-NVFP4 model across eight GPUs with measured throughput approaching 1,000 tokens per second.

The journey is a case study in the messy reality of ML infrastructure work. It is not a clean narrative of linear progress but a recursive loop of hypothesis, experiment, failure, and refinement — exactly the kind of work that defines production ML engineering. This article synthesizes the key moments across the session, drawing on detailed analyses of individual messages that have been documented separately [1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29].

Part I: Laying the Foundation — Drivers, CUDA, and Environment Setup

Every ML deployment begins with the hardware layer, and this session started with a clean Ubuntu 24.04 installation. The first task was to install NVIDIA drivers version 590.48.01 — the latest driver branch compatible with the RTX PRO 6000 Blackwell architecture. This was followed by CUDA Toolkit 13.1, the newest CUDA release at the time. The assistant verified the installation by running nvidia-smi, confirming that both GPUs were operational with the correct driver version and CUDA version.

With the GPU stack in place, the assistant created a Python virtual environment using uv — a fast, modern Python package manager — and installed PyTorch along with core dependencies. This established the foundation for the entire ML software stack.

But the foundation was not as solid as it appeared. The first major obstacle emerged when the assistant attempted to install flash-attn, a high-performance attention kernel library that many LLM inference frameworks depend on. The build process immediately ran into trouble.

Part II: The flash-attn Ordeal — CUDA Version Conflicts and Memory Exhaustion

The flash-attn installation failed because PyTorch 2.9.1 was compiled against CUDA 12.8, while the system had CUDA 13.1 installed. The flash-attn build system detected CUDA 13.1 and attempted to compile against it, but the resulting binaries were incompatible with PyTorch's CUDA 12.8 runtime. This is a classic version mismatch problem that plagues ML environments: different components of the stack (PyTorch, flash-attn, CUDA toolkit) must agree on a common CUDA version, but the dependency chain is rarely explicit about this requirement.

The solution was to install a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation, then configure the build to use the correct version. But this introduced a second problem: the build system attempted to compile flash-attn using all available CPU cores — 128 threads on the machine — which exhausted the available RAM and caused the compiler to crash with out-of-memory errors. The assistant had to progressively reduce the MAX_JOBS environment variable from 128 down to 20, a trial-and-error process that required multiple rebuild attempts. Even then, the build only succeeded after the machine was rebooted with an expanded 432 GB of RAM.

This episode illustrates a fundamental truth about ML infrastructure: the compilation of GPU kernels is a memory-intensive, resource-constrained process that cannot be safely parallelized to the full extent of modern hardware. The assistant learned this the hard way, and the lesson — that MAX_JOBS must be tuned to the available memory, not the available cores — is one that every ML engineer eventually internalizes.

The flash-attn saga did not end there. Later in the session, after vLLM was installed and its dependency resolver downgraded PyTorch from 2.9.1 to an older version, the compiled flash-attn binary became incompatible with the new PyTorch version. The assistant had to rebuild flash-attn from scratch against the correct PyTorch 2.9.1, this time with the knowledge that MAX_JOBS=20 was the safe limit. The final stable stack — PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1 — represented a fragile equilibrium of mutually compatible versions, each component carefully pinned to avoid cascading dependency failures.

Part III: The Machine Upgrade and Model Deployment

With the environment stabilized, the session took a dramatic turn. The machine was upgraded from 2 GPUs to 8 GPUs — a hardware expansion that transformed the deployment from a single-node experiment into a multi-GPU inference cluster. The assistant was tasked with deploying the GLM-5-NVFP4 model, a quantized variant of the GLM-5 architecture stored in GGUF format, using vLLM as the inference engine.

The initial deployment revealed two critical bugs that had been lurking in vLLM's codebase. The first was an output buffer disconnect in the Triton MLA (Multi-head Latent Attention) attention backend: a custom PyTorch op was creating a phantom tensor that caused the attention output to be silently disconnected from the computation graph. The second was a shard ordering bug in the GGUF dequantization layer for fused projections: when assembling quantized weights across multiple GPUs, the dequantization code was assembling projection shards in the wrong order, producing corrupted weight matrices.

Fixing these bugs required deep knowledge of both the vLLM codebase and the GGUF format specification. The assistant traced through the attention dispatch code in mla_attention.py, identified the phantom tensor created by the custom op, and corrected the output buffer wiring. For the GGUF bug, the assistant examined GGUFLinearMethod.apply() in gguf.py and found that the shard index calculation for fused projections was incorrect, causing projection weights to be assembled in the wrong order across tensor-parallel workers.

With both fixes applied, the model produced correct output for the first time. But it was running with --enforce-eager — a debugging flag that disables CUDAGraph optimization and forces PyTorch to execute every operation eagerly. The next step was to see if CUDAGraph could make it faster.

Part IV: The CUDAGraph Gamble

The user's plan was structured as a decision tree with four steps: kill the existing server, restart without --enforce-eager to enable CUDAGraph, verify correctness, and then benchmark. The assistant executed this plan methodically, but each step revealed new complications.

Killing the server proved unexpectedly difficult. The initial pkill command sent SIGTERM, but the process remained alive. The assistant escalated to kill -9 (SIGKILL), waited three seconds, and verified with pgrep that no vLLM processes remained [2][3]. This sequence — documented in [msg 1] through [msg 4] — demonstrates a critical skill in remote process management: knowing when to escalate from graceful shutdown to force termination, and always verifying that the escalation actually worked. The final verification in [msg 4] — "No vllm processes found" — was the green light for the entire CUDAGraph experiment [1].

Starting the CUDAGraph server succeeded, but the health check loop timed out after 16 minutes [6]. The assistant had written a polling loop that checked the /health endpoint every 5 seconds for up to 180 iterations, looking for "ok" or "200" in the response body. The server was actually healthy within seconds — returning HTTP 200 OK — but the response body was empty, so the grep pattern never matched. This is a classic monitoring bug: confusing the HTTP status code (which curl reports separately) with the response body content. The assistant discovered the truth by inspecting the server log, which showed multiple successful health check requests ([msg 7]) [7]. The 16-minute timeout was entirely wasted — a single-character bug in a grep pattern had derailed the entire workflow [8].

The correctness test delivered devastating news. When the assistant sent the prompt "The capital of France is" to the CUDAGraph-enabled server, the response was twenty spaces [12]. Additional prompts confirmed the pattern: mostly whitespace, occasionally a stray word [11]. CUDAGraph was producing garbage output ([msg 11], [msg 12]). The assistant correctly diagnosed the root cause: the custom Triton MLA attention kernels and GGUF dequantization operations were incompatible with CUDA graph capture [13]. CUDAGraph works by recording GPU kernel launches into a replayable graph, but if the model uses kernels with dynamic dispatch behavior — choosing different execution paths based on input data — the captured graph diverges from actual execution, producing corrupted results. This finding was later confirmed as a fundamental incompatibility between CUDAGraph and GGUF MLA models [28][29].

Part V: The Fallback — GPU Memory Cleanup and Eager Mode

The fallback plan was clear: restart with --enforce-eager. But this too encountered obstacles. The new server crashed during startup with a ValueError in request_memory(), indicating that GPU memory allocation had failed ([msg 16] through [msg 20]).

The assistant diagnosed the problem in [msg 21]: the CUDAGraph server's GPU memory had not been released when the process was killed. Only 5.56 GiB remained free out of 94.97 GiB across the 8 GPUs [21]. The residual memory was held by eight orphaned worker processes — one per GPU — that had survived the parent process termination. The assistant used nvidia-smi --query-compute-apps=pid to identify these processes and pkill -9 to terminate them ([msg 22]) [22]. After cleanup, a verification confirmed the memory was released, and the eager-mode server could finally start [23].

This episode underscores a subtle but critical aspect of GPU memory management: killing a Python process does not always release its CUDA memory. The CUDA driver maintains per-context allocations that can persist after process death, especially if worker processes are orphaned or if SIGTERM (rather than SIGKILL) is used. The assistant's systematic approach — identify residual processes, force-kill them, verify with nvidia-smi, then restart — is a template for GPU memory cleanup that every ML engineer should know.

With memory cleared, the eager-mode server started successfully after 445 seconds of model loading ([msg 24]). The correctness test confirmed beautiful, coherent output: "The capital of France is" → "Paris. Distance from London to Paris is 551 km, while direct flight time is 1" ([msg 25]).

Part VI: The Benchmark — Quantifying Performance

With correctness established, the assistant pivoted to benchmarking ([msg 26]). The benchmark script tested five concurrency levels — 1, 2, 10, 64, and 256 — using a ThreadPoolExecutor to simulate concurrent users. Each request used a ~100-token prompt requesting 128 output tokens, with a warmup phase to stabilize GPU state before measurement.

The results, reported in [msg 28], told a compelling story:

| Concurrency | Throughput (tok/s) | Latency Avg | P99 Latency | Errors | |-------------|-------------------|-------------|-------------|--------| | 1 | 20.0 | 6.41s | 6.43s | 0/10 | | 2 | 39.2 | 6.57s | 6.58s | 0/20 | | 10 | 158.4 | 8.06s | 8.46s | 0/30 | | 64 | 539.8 | 15.2s | 17.2s | 0/64 | | 256 | 995.4 | 25.5s | 30.9s | 0/256 |

The key insight is the 50× throughput multiplier from concurrency 1 to 256, demonstrating the power of vLLM's continuous batching scheduler. At high concurrency, the server batches multiple requests into each forward pass, amortizing GPU overhead across many users. The zero error rate across all 442 requests confirmed the deployment's stability.

However, the single-request throughput of ~20 tok/s revealed a significant bottleneck. The assistant later profiled the system and found that approximately 42% of inference time was spent on NCCL allreduce calls — the communication required to synchronize gradients and activations across the 8 GPUs. Because the GPUs were connected via PCIe rather than NVLink, this inter-GPU communication was bandwidth-limited.

Part VII: Optimization and Productionalization

Building on the benchmark baseline, the assistant pursued several optimization strategies. Enabling CUDAGraph (which had previously produced garbage output) was revisited after fixes to the underlying incompatibility, and this time it doubled throughput to ~43 tok/s by batching kernel launches and reducing CPU-side scheduling overhead. Tuning NCCL_PROTO=LL — a protocol optimization for NCCL communication — further boosted throughput to ~57 tok/s.

More advanced optimizations were explored but found to be incompatible with the PCIe-only hardware topology. Custom allreduce kernels and allreduce-RMS fusion — techniques that overlap communication with computation — could not be effectively applied because the PCIe bus was already saturated. This established a clear hardware bottleneck: the 8-GPU system was limited by inter-GPU bandwidth, not by compute capacity.

The final task was productionalizing the optimized configuration into a systemd service (vllm-glm5.service). While the service was created and deployed, initial startup failed due to a conflict with a stale vLLM process from a previous session — a reminder that even automated service management must contend with the messy reality of process lifecycle management.

Lessons Learned

This session encapsulates several enduring lessons for ML infrastructure engineering:

Version compatibility is the hardest problem. The flash-attn saga — spanning CUDA toolkit version conflicts, PyTorch downgrades, and memory-exhausting parallel builds — consumed more time than any single feature implementation. ML environments are dependency graphs with millions of edges, and every edge is a potential point of failure.

Health checks are not trivial. A single misplaced grep pattern cost 16 minutes of debugging time. The assumption that a health endpoint returns "ok" or "200" in its body was wrong, and the cost of that wrong assumption was measured in wall-clock time. Always validate monitoring scripts independently of the services they monitor.

GPU memory is not released on process death. This counterintuitive fact — that CUDA memory can persist after a process is killed — is one of the most common sources of deployment failures. The systematic pattern of identify, force-kill, verify, and restart is essential knowledge for anyone working with GPU-accelerated inference.

CUDAGraph is powerful but fragile. The incompatibility between CUDA graph capture and custom Triton kernels is a known limitation that will likely persist as long as models use custom operations. The fallback to eager mode — and the discovery that eager mode still delivers 995 tok/s — is a reminder that the theoretically optimal optimization is not always the practically necessary one.

Benchmarking reveals what intuition cannot. The 50× throughput multiplier from concurrency scaling was not obvious from single-request latency. Without systematic measurement across multiple concurrency levels, the deployment's true capacity would have remained unknown.

Conclusion

The journey from bare metal to benchmark — from installing NVIDIA drivers on a clean Ubuntu machine to serving 1,000 tokens per second across eight Blackwell GPUs — is a microcosm of the ML deployment experience. It is messy, iterative, and full of surprises. The flash-attn build fails, the CUDAGraph produces garbage, the health check times out, the GPU memory won't release, and the PCIe bus becomes the bottleneck. But through systematic debugging, careful measurement, and the willingness to fall back when optimizations fail, the deployment succeeds.

The final stack — PyTorch 2.9.1, flash-attn 2.8.3, vLLM 0.15.1, eight RTX PRO 6000 GPUs, and the GLM-5-NVFP4 model — is not just a collection of compatible versions. It is a testament to the engineering discipline required to make cutting-edge ML research work in practice. And the benchmark numbers — 20 tok/s per request, 995 tok/s at scale, zero errors — are not just data points. They are the quantitative proof that the stack works.## References

[1] "The Confirmation That Changed Everything: A Verification Step in the CUDAGraph Optimization Pipeline" — Analysis of message 4, the process verification checkpoint after force-killing the server.

[2] "The First Step of a Thousand: Initiating CUDAGraph Optimization for vLLM" — Analysis of message 1, the initial attempt to kill the server and begin the CUDAGraph workflow.

[3] "The Art of Force-Killing: A Single Bash Command and What It Reveals About Agentic Debugging" — Analysis of message 3, the escalation to SIGKILL after the initial kill failed.

[4] "The CUDAGraph Gamble: A Systematic Benchmarking Protocol for vLLM Inference Optimization" — Analysis of message 0, the user's four-step plan for CUDAGraph testing and benchmarking.

[5] "The Pivot Point: When CUDAGraph Meets a Missing Module" — Analysis of message 10, the correctness test that revealed a Python environment mismatch.

[6] "The 16-Minute Timeout: How a Flawed Health Check Masked a Successful CUDAGraph Deployment" — Analysis of message 6, the health check loop that timed out due to a grep pattern bug.

[7] "The Silent Server: When a Health Check Bug Masks Success" — Analysis of message 7, discovering the server was actually healthy despite the timeout.

[8] "Debugging Automation: How a Misplaced Grep Pattern Cost 16 Minutes in an ML Deployment Pipeline" — Analysis of message 8, the grep pattern fix.

[9] "Diagnosing the Silent Failure: A Pivot Point in CUDAGraph Debugging" — Analysis of message 9, scanning logs after the timeout.

[10] "Uncovering a Memory Allocation Failure in vLLM" — Analysis of message 20, the GPU memory allocation error.

[11] "When CUDAGraph Goes Silent: Debugging Garbage Output in vLLM's Graph Compilation Pipeline" — Analysis of message 12, confirming CUDAGraph garbage output across multiple prompts.

[12] "The Turning Point: When CUDAGraph Produces Garbage" — Analysis of message 11, the first discovery of CUDAGraph producing only spaces.

[13] "The CUDAGraph Verdict: When GPU Acceleration Produces Garbage" — Analysis of message 13, the decision to fall back to eager mode.

[14] "The Verification That Prevents Catastrophe: Analyzing a Process-Check Message in a CUDAGraph Debugging Session" — Analysis of message 14, verifying process state after fallback.

[15] "The Pivot: Restarting with --enforce-eager After CUDAGraph Failure" — Analysis of message 15, the restart command for eager mode.

[16] "The Silent Timeout: When Killing a Process Isn't Enough" — Analysis of message 16, the timeout during eager-mode server startup.

[17] "The Diagnostic Pivot: How a Single Curl Command Revealed a Server Crash" — Analysis of message 17, diagnosing the server crash.

[18] "The Diagnostic Tail: Unraveling a vLLM Server Crash Mid-Session" — Analysis of message 18, examining server logs after the crash.

[19] "The Server Crash That Wasn't What It Seemed: A Case Study in Diagnostic Debugging" — Analysis of message 19, further crash diagnosis.

[20] "The Debugger's Scalpel: Uncovering a Memory Allocation Failure in vLLM" — Analysis of message 20, the memory allocation failure root cause.

[21] "Diagnosing GPU Memory Persistence: The Critical Diagnostic Turn in vLLM Deployment" — Analysis of message 21, identifying residual GPU memory from orphaned processes.

[22] "The Critical Cleanup: How Killing Stale GPU Processes Saved a Model Deployment" — Analysis of message 22, force-killing residual GPU processes.

[23] "The Third Time's the Charm: Diagnosing GPU Memory Contention in a Failed vLLM Restart" — Analysis of message 23, verifying GPU memory cleanup.

[24] "The 445-Second Wait: A Deceptively Simple Health Check in ML Infrastructure Debugging" — Analysis of message 24, the successful eager-mode server startup.

[25] "The Moment of Truth: Validating Correctness After a CUDAGraph Failure" — Analysis of message 25, the correctness test confirming eager-mode output.

[26] "The Benchmarking Pivot: From CUDAGraph Failure to Performance Measurement" — Analysis of message 26, writing the benchmark script.

[27] "The Benchmark That Proved the Stack: Measuring GLM-5 Throughput Across Eight Blackwell GPUs" — Analysis of message 27, running the benchmark.

[28] "The Verdict on CUDAGraph: A Performance Autopsy of vLLM with GGUF MLA Models" — Analysis of message 28, the final benchmark report.

[29] "The CUDAGraph Verdict: A Performance Autopsy of vLLM with GGUF MLA Models" — Analysis of message 28, the final benchmark report and conclusions.