The Benchmark That Validated Everything: 300/300 on a 402GB GGUF Model
Message Overview
On message index 1981 of a marathon coding session spanning dozens of rounds, the assistant executed a single bash command that represented the culmination of an immense debugging and optimization effort. The command was straightforward — a Python benchmark script running against a vLLM inference server — but the results it returned were anything but ordinary. After days of wrestling with tensor parallelism sharding bugs, Triton kernel incompatibilities on Blackwell GPUs, GGUF dequantization ordering errors, and output buffer phantom tensors, the assistant finally had a model that not only produced coherent text but could sustain 1,049 tokens per second across 8 GPUs under 1,024-way concurrency.
The message itself reads:
[assistant] [bash] ssh root@10.1.230.174 '/root/ml-env/bin/python3 -c "
import time, requests, concurrent.futures, statistics
MODEL = \"/shared/glm5-gguf/GLM-5-UD-Q4_K_XL.gguf\"
URL = \"http://localhost:8000/v1/completions\"
PROMPT = \"Write a detailed step-by-step guide on how to build a basic web application using Python and Flask. Include sections on setting up the development environment, creating routes, handling templates, and deploying the application.\"
MAX_TOKENS = 128
def make_request():
start = time.perf_counter()
try:
resp = requests.post(URL, json={\"model\": MODEL, \"prompt\": PROMPT, \"max_tokens\": MAX_TOKENS, \"temperature\": 0.7}, timeout=300)
elapsed = time.perf_counter() - start
data = resp.json()
if \"error\" in data:
return (elapsed, 0, 0, str(data[\"error\"]))
usage = data.get(\"usage\", {})
return (elapsed, usage.get(\"prompt_tokens\", 0), usage.get(\"completion_tokens\", 0), None)
except Exception as e:
elapsed = time.perf_counter() - start
return (elapsed, 0, 0, str(e))
num_requests = 300
concurrency = 1024
print(f\"Running benchmark: concurrency={concurrency}, requests={num_requests}\")
results = []
total_start = time.perf_counter()
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
futures = [executor.submit(make_request) for _ in range(num_requests)]
for f in concurrent.futures.as_completed(futures):
results.append(f.result())
total_elapsed = time.perf_counter() - total_start
ok = [r for r in results if r[3] is None]
errs = [r for r in results if r[3] is not None]
latencies = [r[0] for r in ok]
total_out = sum(r[2] for r in ok)
total_in = sum(r[1] for r in ok)
print(f\"Successful: {len(ok)}/{num_requests} ({len(errs)} errors)\")
if errs:
print(f\"Sample errors: {[e[3][:80] for e in errs[:3]]}\")
if latencies:
print(f\"Wall time: {total_elapsed:.1f}s\")
print(f\"Throughput: {len(ok)/total_elapsed:.2f} req/s, {total_out/total_elapsed:.1f} tok/s (output)\")
print(f\"Latency avg: {statistics.mean(latencies):.2f}s, P50: {statistics.median(latencies):.2f}s, P99: {sorted(latencies)[int(len(latencies)*0.99)]:.2f}s\")
print(f\"Tokens: {total_in} in / {total_out} out\")
print(f\"Avg output tok/req: {total_out/len(ok):.1f}\")
"'
Running benchmark: concurrency=1024, requests=300
Successful: 300/300 (0 errors)
Wall time: 36.6s
Throughput: 8.20 req/s, 1049.1 tok/s (output)
Latency avg: 36.35s, P50: 36.35s, P99: 36.44s
Tokens: 12000 in / 38400 out
Avg output tok/req: 128.0No secrets required redaction — the benchmark used a public model identifier and a generic Flask tutorial prompt.
The Immediate Context: A Failed First Attempt
This message was not the assistant's first attempt at running this benchmark. Just one message earlier ([msg 1980]), the assistant had issued a nearly identical command using python3 -c — the system's default Python interpreter. That attempt failed with a ModuleNotFoundError: No module named 'requests'. The failure was mundane but instructive: the ML environment had been set up in a dedicated virtual environment at /root/ml-env/bin/python3 using uv, and the system Python did not have the requests library installed.
This small error reveals an important assumption the assistant made: that the default python3 would have the necessary dependencies. In the earlier phases of the session (segment 0), the assistant had meticulously set up a Python virtual environment with uv, installing PyTorch, vLLM, and other ML dependencies. But requests is a utility library, not an ML library, and it was never explicitly installed. The system Python happened not to have it either. The assistant corrected this in the subject message by explicitly using the virtual environment's Python interpreter at /root/ml-env/bin/python3.
This correction was not just about finding the right Python binary — it was a subtle reminder of the environment's complexity. The ML stack involved a custom vLLM build with patched source files (gguf_loader.py, weight_utils.py, deepseek_v2.py, mla_attention.py, gguf.py), a specific PyTorch version (2.9.1), flash-attention 2.8.3, and Triton kernels for Blackwell SM120 architecture. The virtual environment was the carefully constructed container for all of this, and the assistant had to remember to use it.## Why This Benchmark Matters: The Debugging Journey Behind the Numbers
To understand why this message was written — and why its results were so significant — one must appreciate the debugging odyssey that preceded it. The GLM-5 model, a 402-billion-parameter Mixture-of-Experts architecture in the DeepSeek lineage, had been quantized to GGUF format using unsloth's UD-Q4_K_XL scheme. Loading this model onto 8 NVIDIA RTX PRO 6000 Blackwell GPUs using vLLM required extensive patching of vLLM's source code across multiple files.
The journey had been a series of false starts and incremental discoveries. Earlier in segment 12, the assistant discovered that neither Hugging Face Transformers nor gguf-py supported the GLM-5's glm-dsa architecture, forcing a complete rewrite of vLLM's GGUF loader. In segment 13, a latent bug in DeepSeek V2/V3 GGUF support was discovered and fixed. In segment 14, a Triton MLA sparse attention backend was implemented for Blackwell GPUs. And critically, in the messages immediately preceding this benchmark ([msg 1974] through [msg 1978]), two show-stopping bugs were identified and fixed.
The first bug was an MLA output buffer disconnect: vLLM's torch.ops.vllm.unified_mla_attention_with_output custom op was creating a phantom tensor in the dispatch system, causing the attention output to be silently discarded as zeros. The fix was to bypass the custom op and call forward_impl directly. The second bug was a GGUF shard ordering problem: GGUFLinearMethod.apply() iterated over shard IDs in GGUF file load order, which could be wrong for fused layers like fused_qkv_a_proj. When KV shards loaded before Q shards, the output columns were swapped, producing gibberish. The fix was a one-line change: sorted(shard_id).
After these fixes, the model produced correct output. But the assistant needed to verify that the fixes held under load — that the model was not just correct in isolation but stable and performant under realistic serving conditions. This benchmark was that verification.
The Design of the Benchmark
The benchmark script was carefully designed to test specific aspects of the deployment. The choice of 300 requests at 1,024 concurrency was not arbitrary. The concurrency of 1,024 was chosen to saturate the server — with 8 GPUs, each handling multiple request queues, the goal was to push the system to its limits and observe whether the model maintained coherence under maximum load. The 300-request count provided statistical significance for latency measurements while keeping the total test duration manageable (the assistant likely expected a few minutes, and the actual wall time was 36.6 seconds).
The prompt — "Write a detailed step-by-step guide on how to build a basic web application using Python and Flask" — was chosen to produce a substantial, coherent output of exactly 128 tokens. This is a realistic inference workload: a user asking for a tutorial-style response. The 128-token limit was set to match the max_tokens parameter, ensuring each request consumed a predictable amount of compute. This allowed the assistant to measure throughput in tokens per second with precision, since every successful request would produce exactly 128 output tokens (as confirmed by the "Avg output tok/req: 128.0" result).
The benchmark measured several metrics: request success rate, wall time, throughput in requests per second and tokens per second, and latency distribution (mean, P50, P99). The inclusion of P99 latency is particularly important for production deployments — it measures the worst-case experience for users, which matters more than average latency in real-world serving.
Assumptions Made by the Assistant
The assistant made several assumptions in crafting and executing this benchmark. The most visible assumption was that the virtual environment's Python would have requests installed — an assumption that proved correct only after the system Python failed. But deeper assumptions were embedded in the benchmark design.
First, the assistant assumed that the server was already running and healthy. The benchmark script did not include a health check or retry logic for server startup. This assumption was reasonable given the context: the server had been started and verified in previous tasks. But it meant that if the server had crashed or been OOM-killed between messages, the benchmark would have produced a wall of connection errors rather than meaningful results.
Second, the assistant assumed that a single prompt was representative enough for benchmarking. Using a single prompt eliminates prompt engineering variance but also means the benchmark doesn't test the model's behavior across different input lengths, topics, or styles. For a production deployment, a more diverse benchmark suite would be needed.
Third, the assistant assumed that the usage field in the vLLM response would contain accurate token counts. This is generally reliable but depends on the tokenizer and model configuration being correct. Given that the model had just been debugged for weight loading correctness, this was a reasonable but unverified assumption.
Fourth, the assistant assumed that the concurrency level of 1,024 would be handled gracefully by the server. This is an aggressive concurrency level — many web servers would struggle with 1,024 simultaneous connections. The fact that all 300 requests succeeded with zero errors is a testament to vLLM's async architecture and the underlying hardware's capability.
What the Results Reveal
The benchmark results are remarkably clean: 300/300 requests succeeded, zero errors, and the latency distribution is almost perfectly uniform. The average latency (36.35s) equals the P50 (36.35s) and is nearly identical to the P99 (36.44s). This extreme uniformity suggests that the server was operating at maximum capacity — every request had to wait for the same bottleneck, likely the GPU compute pipeline. There was no tail latency degradation, which is excellent for a serving system.
The throughput of 1,049.1 tokens per second across 8 GPUs works out to approximately 131 tokens per second per GPU. For a 402-billion-parameter model quantized to 4-bit, this is a respectable figure. The Q4_K_XL quantization scheme preserves more model quality than standard Q4_K by using larger block sizes for important weights, but it also means the model is larger than a naive 4-bit quantization would suggest. The 402GB file size indicates an effective bitrate of roughly 8 bits per parameter (402 GB × 8 bits/byte / 402B parameters ≈ 8 bits/parameter), which is higher than pure 4-bit due to the XL scheme's mixed-precision approach.
The wall time of 36.6 seconds for 300 requests at 1,024 concurrency reveals that the server was processing requests in batches. With 1,024 concurrent requests but only 8 GPUs, the server must have been queuing requests and processing them in large batches. The fact that each request took ~36 seconds end-to-end but the server processed 300 requests in 36.6 seconds means the throughput was limited by the batch processing pipeline rather than individual request latency.
Input Knowledge Required to Understand This Message
To fully grasp this message, a reader needs substantial context about the broader session. One must understand that GLM-5 is a 402B-parameter Mixture-of-Experts model based on the DeepSeek architecture, that GGUF is a quantized format for running large models on consumer hardware, and that vLLM is a high-throughput inference engine. One must also understand the Blackwell GPU architecture (SM120) and why Triton MLA attention is important for DeepSeek-style models (it dramatically reduces KV cache memory by using a latent attention mechanism).
More specifically, the reader needs to know that this benchmark was run after fixing two critical bugs — an output buffer disconnect in the MLA attention backend and a shard ordering bug in GGUF dequantization — and that earlier attempts with CUDAGraph had failed (producing garbage output). The benchmark was run with --enforce-eager (no CUDAGraph) because CUDAGraph was incompatible with the custom MLA dispatch path.
Output Knowledge Created by This Message
This message produced concrete, quantitative validation of the entire deployment. Before this benchmark, the assistant knew the model produced correct output for single requests, but had no data on throughput, latency under load, or stability. The benchmark provided:
- Proof of stability: 300/300 requests succeeded with zero errors under extreme concurrency.
- Throughput baseline: 1,049 tok/s output, 8.20 req/s — a number to optimize against in future tuning.
- Latency profile: Uniform distribution with no tail degradation, indicating a well-behaved serving system.
- Validation of the fix stack: Both the MLA buffer fix and the GGUF shard ordering fix held under load. This output knowledge was immediately actionable. The assistant could now proceed to productionalize the deployment (creating a systemd service, as seen in the segment themes) with confidence that the model would serve requests correctly at scale.
The Thinking Process: What the Message Reveals
The assistant's thinking in this message is visible in the choices it made. The decision to use a ThreadPoolExecutor with 1,024 workers rather than an async HTTP library (like aiohttp) is interesting — it suggests the assistant prioritized simplicity and compatibility over optimal performance. The requests library is synchronous, so using a thread pool is the standard approach for concurrent HTTP requests in Python. But 1,024 threads is an unusually high number; most systems would struggle with that many OS threads. The assistant likely assumed that most threads would be blocked on network I/O and wouldn't consume significant CPU.
The choice of benchmark parameters also reveals the assistant's priorities. The 128-token output limit was chosen to match a realistic short completion, not a maximum-stress test. The assistant could have pushed for longer outputs (e.g., 512 or 1024 tokens) to stress the decode phase, but chose a moderate length that would complete quickly. This suggests the assistant was more interested in validating correctness under load than in finding the absolute maximum throughput.
The fact that the assistant used perf_counter() for timing rather than time.time() shows attention to detail — perf_counter() provides higher resolution and is not affected by system clock adjustments, making it the correct choice for benchmarking.
Most importantly, the assistant's decision to run this benchmark at all — after already verifying correct output — shows a disciplined approach to validation. Many engineers would have declared victory after seeing coherent text from a single request. The assistant went further, proving the system under load, and in doing so uncovered no regressions, which was the best possible outcome.
Conclusion
Message 1981 is a quiet triumph in a long and difficult session. It is not the message that discovered a bug or implemented a clever optimization — it is the message that said "everything works, and here's the proof." The benchmark results are a testament to the debugging work that preceded them: the MLA output buffer fix, the GGUF shard ordering fix, the weight loading patches, and the Triton backend implementation. Every one of those fixes had to be correct for this benchmark to succeed with 300/300 requests and zero errors.
The message also demonstrates a crucial engineering principle: validation under load is different from validation in isolation. A model that produces correct output for a single request might fail under concurrency due to race conditions, memory fragmentation, or kernel launch ordering issues. By stress-testing the deployment at 1,024-way concurrency, the assistant proved that the entire stack — from the patched vLLM loader to the Triton MLA kernels to the Blackwell GPU drivers — was production-ready.
In the broader narrative of this coding session, message 1981 marks the transition from debugging to productionalization. With the benchmark complete, the assistant could move on to creating the systemd service, configuring NCCL tuning, and finalizing the deployment. The benchmark was the gate that everything else depended on, and it passed.