Establishing Baseline Throughput: The Moment GLM-5-NVFP4 Proved Stable Under Concurrent Load
The Message
The subject of this article is message index 225 in a lengthy opencode session deploying the GLM-5-NVFP4 model — a 744-billion-parameter Mixture-of-Experts reasoning model — on eight NVIDIA RTX PRO 6000 Blackwell (SM120) GPUs. The message reads:
[assistant] Now we have a baseline: - 32 concurrent requests: 144 output tok/s, 304 total tok/s, 18.4 average concurrency - All 32 requests succeeded — no crashes under concurrent load
>
Let me check the server log to see the decode throughput under load, and then try pushing harder: [bash] ssh 10.1.230.175 'tail -40 ~/sglang-glm5.log' [2026-02-19 00:27:12] INFO: 127.0.0.1:33670 - "POST /generate HTTP/1.1" 200 OK [2026-02-19 00:27:12] INFO: 127.0.0.1:33754 - "POST /generate HTTP/1.1" 200 OK [2026-02-19 00:27:14] INFO: 127.0.0.1:33704 - "POST /generate HTTP/1.1" 200 OK [2026-02-19 00:27:15] INFO: 127.0.0.1:33828 - "POST /generate HTTP/1.1" 200 OK [2026-02-19 00:27:16 TP0] Decode batch, #running-req: 27, #token: 5760, token usage: 0.02, cuda graph: False, gen throughput (token/s): 170.24, #queue-req: 0 [2026-02-1...
At first glance, this appears to be a routine progress update — the assistant reporting numbers and checking a log file. But within the arc of this coding session, this message represents a critical inflection point. It is the moment the deployment transitions from "does it work at all?" to "how well does it work?" — from debugging a catastrophic NaN crash to establishing performance baselines and planning optimization. This article unpacks the reasoning, decisions, assumptions, and knowledge embedded in this single message.
The Preceding Crisis: Why This Message Matters
To understand why this message was written, one must appreciate what came immediately before it. The session had been consumed by a severe and persistent bug: the GLM-5-NVFP4 model, when deployed on SM120 Blackwell GPUs using SGLang, produced NaN (Not-a-Number) values during the decode phase — the step where the model generates output tokens one at a time. This is a catastrophic failure mode; the model's output becomes garbage, and the server often crashes or hangs.
The assistant had cycled through multiple attempted fixes. It tried different attention backends (flashinfer, flashmla), different KV cache dtypes (auto, bfloat16), and different NSA (Native Sparse Attention) decode backends (flashmla_kv, flashmla_sparse). All produced NaN during decode. The breakthrough came in message 215, when the assistant used --nsa-decode-backend trtllm --nsa-prefill-backend trtllm and got coherent output for the first time. Messages 216 and 217 confirmed the model was producing correct reasoning chains and answers — the fix worked.
But a single correct response does not constitute a working deployment. The system needed to handle concurrent requests, sustain throughput, and remain stable under load. Message 223 revealed a new problem: the bench_serving tool's OAI chat backend crashed when processing reasoning models because the model returns content: null (with the actual output in reasoning_content), and the tokenizer could not handle a null generated_text. The assistant pivoted to the sglang native backend, which worked. Message 224 ran a small-scale test (16 prompts, rate 2) yielding 50.55 tok/s — low but expected without CUDA graph optimization.
Message 225 is the report of the first meaningful multi-request benchmark: 32 concurrent requests at a higher request rate. This is the moment the deployment proves it can handle real-world usage patterns.
The Reasoning and Motivation
The assistant's primary motivation in this message is to establish a baseline. The word "baseline" appears prominently at the top. A baseline serves multiple purposes in performance engineering:
First, it validates that the system works under load. The assistant explicitly notes "All 32 requests succeeded — no crashes under concurrent load." This is not a trivial statement. Given that every previous attempt at decode had produced NaN, confirming that the trtllm NSA backend remains stable under concurrency is a significant milestone. The assistant is systematically eliminating the risk that the fix only works for single requests.
Second, a baseline provides a reference point for optimization. The assistant knows the current configuration is suboptimal: CUDA graphs are disabled (cuda graph: False in the server log), memory fraction is at 0.85, and no performance tuning has been applied. The 144 output tok/s figure is not the target — it is the starting point. Every subsequent tuning change (increasing memory fraction, enabling CUDA graphs, switching MoE backends) will be measured against this number.
Third, the assistant is triangulating between two data sources: the client-side benchmark tool reports 144 output tok/s, while the server-side log reports 170.24 tok/s at peak. This discrepancy is informative. The client-side metric averages over the entire test duration (including prefill time and queueing), while the server-side "gen throughput" measures only the decode batch throughput at a specific moment. Understanding this gap helps the assistant reason about where time is being spent — in prefill, in queueing, or in decode.
The assistant also signals its next step explicitly: "and then try pushing harder." This reveals a deliberate ramp-up strategy. The assistant is not jumping straight to maximum load; it is incrementally increasing concurrency (16 prompts → 32 → 64 → saturated) to observe how throughput scales and where bottlenecks emerge.
Decisions Made in This Message
Several decisions are visible, some explicit and some implicit:
Decision to use the sglang native backend for benchmarking. This was made in the previous message (224) after the OAI chat backend crashed. The native backend uses the /generate endpoint rather than the OpenAI-compatible /v1/chat/completions endpoint. This trades off compatibility for reliability — the native endpoint returns raw generated text without the reasoning_content wrapper that confuses the benchmark tool.
Decision to use 256-token input and 256-token output lengths. These are moderate values that allow the benchmark to complete quickly while still exercising both the prefill and decode phases. Longer sequences would stress the KV cache more but take longer to run; shorter sequences might not reveal throughput characteristics accurately.
Decision to check the server log. This is a deliberate choice to gather server-side metrics that the client-side benchmark cannot provide. The bench_serving tool with the native backend cannot report per-token timing (TTFT, TPOT, ITL) because the /generate endpoint returns results as a single non-streaming response. The server log fills this gap with its "Decode batch" lines showing instantaneous decode throughput, batch size, and token usage.
Decision to report both client-side and server-side metrics. The assistant presents the benchmark results (144/304 tok/s) and then immediately checks the server log (170.24 tok/s). This dual-perspective approach demonstrates a sophisticated understanding that no single metric tells the whole story.
Decision to continue pushing harder. Rather than stopping at 32 requests, the assistant explicitly plans to increase load. This reflects an understanding that 32 requests at rate 8 may not saturate the system — the average concurrency of 18.4 suggests there is headroom.
Assumptions and Their Implications
The message rests on several assumptions, most of which are reasonable but worth examining:
Assumption that the benchmark is representative. The random dataset generates synthetic prompts with random token sequences. Real user traffic would have different characteristics — variable input lengths, different topic distributions, different reasoning depths. The assistant implicitly assumes that random tokens exercise the model's throughput similarly to real text, which is generally true for throughput measurement but may miss memory or cache effects from real token distributions.
Assumption that 32 requests is sufficient for a baseline. With 8 GPUs running tensor parallelism, 32 requests means an average of 4 requests per GPU. This may not be enough to stress all components — the attention mechanism, the MoE routing, the all-reduce communication. The assistant acknowledges this by planning to push harder.
Assumption that the server log is accurate. The "gen throughput (token/s): 170.24" figure is a snapshot at one instant (00:27:16) with 27 running requests. It may not represent the average throughput over the entire benchmark run. The assistant treats it as a useful data point but does not over-interpret it.
Assumption that no silent errors occurred. All 32 requests returned HTTP 200 and completed without crashing, but the assistant does not verify that the outputs are semantically correct. Given that the model was producing correct answers in earlier single-request tests, this is a reasonable shortcut, but it does leave a gap — a regression that produces subtly wrong (but non-NaN) output would not be caught.
Assumption that the system is stable enough for further tuning. The assistant concludes "no crashes under concurrent load" and proceeds to optimization. This is a judgment call — the system could still have edge cases (e.g., memory fragmentation over time, long-context degradation) that only appear after hours of operation.
Input Knowledge Required
To fully understand this message, a reader needs knowledge spanning several domains:
Model architecture knowledge: GLM-5-NVFP4 is a 744B MoE model using NVFP4 quantization (4-bit floating point). It has 8 experts, uses Native Sparse Attention (NSA), and requires significant GPU memory — 80GB per GPU out of 97GB available, as shown in message 219. Understanding that this is a reasoning model (with explicit reasoning_content) is crucial to understanding why the OAI chat backend crashed.
SGLang architecture knowledge: SGLang is a serving system for large language models. Key concepts include tensor parallelism (TP=8, splitting the model across 8 GPUs), NSA backends (different implementations of the sparse attention kernel), MoE runner backends (how expert computation is dispatched), CUDA graphs (a performance optimization that captures GPU operations into reusable graphs), and memory management (mem-fraction-static controls how much GPU memory is reserved for the KV cache).
Benchmarking methodology: Understanding the difference between output tok/s (tokens generated by the model) and total tok/s (including input tokens), the concept of request rate vs. concurrency, and the limitations of different benchmark backends.
Server log interpretation: The "Decode batch" log line format includes running request count, token count, token usage ratio, CUDA graph status, generation throughput, and queue depth.
Hardware context: The system has 8 RTX PRO 6000 Blackwell GPUs (SM120 architecture, 96GB each) connected via PCIe (not NVLink) in a Proxmox VM. This PCIe-only interconnect becomes a critical bottleneck later in the session.
Output Knowledge Created
This message produces several pieces of new knowledge:
A validated baseline throughput figure: 144 output tok/s and 304 total tok/s for a 744B MoE model on 8 PCIe-connected Blackwell GPUs. This becomes the reference point for all subsequent tuning.
Confirmation of stability under concurrency: The trtllm NSA backend does not regress under load. This is the first time in the session that multi-request inference succeeds.
Server-side decode throughput: The log reveals 170.24 tok/s instantaneous decode throughput with 27 concurrent requests, providing a more granular view than the client-side average.
A working benchmark methodology: The combination of --backend sglang with the native /generate endpoint, using the HuggingFace model path as tokenizer, and random prompts with controlled input/output lengths, is established as a reliable testing protocol.
A documented discrepancy between client and server metrics: The gap between 144 tok/s (client average) and 170 tok/s (server snapshot) provides insight into where overhead exists — in prefill, queueing, or measurement methodology.
The Thinking Process
The assistant's reasoning is visible in the structure of the message. It follows a pattern: observe → interpret → plan next step.
The observation is the benchmark output: three numbers (output tok/s, total tok/s, average concurrency) plus a qualitative statement (all succeeded). The assistant does not simply report these numbers; it contextualizes them as a "baseline," signaling that they are provisional and will be compared against future results.
The interpretation happens implicitly through the decision to check the server log. The assistant recognizes that the client-side benchmark has blind spots — it cannot report per-token latency metrics with the native backend. The server log provides complementary data. The specific line the assistant extracts — "gen throughput (token/s): 170.24" — is the most relevant server-side metric for decode performance.
The plan is stated explicitly: "try pushing harder." This is not vague; the assistant has a clear ramp-up strategy. The next message (226) runs 64 prompts at rate 16, and the following message (227) runs 64 prompts at infinite rate (saturation). This stepwise approach is classic performance engineering — increase one variable at a time, observe the response, and identify the bottleneck.
The assistant also demonstrates a careful calibration of confidence. It does not declare victory. It does not say "the deployment is complete" or "performance is good." It says "now we have a baseline" — a measured, provisional statement that leaves room for the numbers to improve or reveal problems.
Conclusion
Message 225 is a quiet but pivotal moment in the GLM-5-NVFP4 deployment. It sits at the boundary between two phases: the debugging phase, where the question was "can the model generate coherent output at all?" and the optimization phase, where the question becomes "how can we make it faster?" The assistant's methodical approach — establishing a baseline, triangulating between data sources, and planning incremental load increases — reflects a disciplined engineering mindset. The message itself is brief, but the reasoning behind it is dense with domain knowledge, strategic thinking, and careful judgment. It is the moment the deployment stopped being a rescue mission and became a performance engineering project.