The Plateau at 2048: A Benchmarking Message That Revealed the Ceiling
Introduction
In the middle of an intensive optimization session for the GLM-5-NVFP4 model running on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, the assistant issued a seemingly routine command: push the concurrency to 2048 and measure throughput. Message [msg 902] is, on its surface, a single bash invocation and its output — a brief data point in a long chain of experiments. But this message is far more than a benchmark result. It is the moment the scaling curve bent, the point at which the assistant discovered the system's ceiling, and the catalyst for a deeper investigation into GPU utilization, kernel efficiency, and server configuration limits. Understanding this message requires unpacking the reasoning that led to it, the assumptions baked into the test, the knowledge it produced, and the thinking process visible in its execution.
The Message Itself
The assistant wrote:
Good — now let me push to 2048 to see if we scale further:
>
``bash ssh root@10.1.230.174 'source /root/ml-env/bin/activate && echo "=== CONCURRENCY 2048 ===" && python3 -m sglang.bench_serving \ --backend sglang --host 127.0.0.1 --port 8000 \ --model glm-5 --tokenizer lukealonso/GLM-5-NVFP4 \ --dataset-name random --random-input-len 128 --random-output-len 128 \ --num-prompts 2048 --request-rate 999 \ 2>&1 | grep -E "Output token throughput|Total token throughput|Peak output|Mean TPOT|Concurrency:"' ``
>
`` === CONCURRENCY 2048 === Output token throughput (tok/s): 1639.69 Peak output token throughput (tok/s): 3593.00 Total token throughput (tok/s): 3249.48 Concurrency: 1254.38 Mean TPOT (ms): 527.24 ``
The command is straightforward: use SGLang's built-in bench_serving tool to send 2048 prompts (each with 128 random input tokens and requesting 128 output tokens) at an effectively infinite request rate (--request-rate 999), then extract the key performance metrics. The results show output throughput of ~1,640 tok/s, peak output of ~3,593 tok/s, total throughput of ~3,249 tok/s, an actual achieved concurrency of ~1,254, and a mean time-per-output-token (TPOT) of 527 ms.
Why This Message Was Written: The Reasoning and Motivation
The assistant did not arrive at this test arbitrarily. The message is the culmination of a carefully structured benchmarking sequence that began in [msg 900], where the assistant declared: "Let me run benchmarks at multiple concurrency levels in parallel — single, 256, 512, 1024, and 2048." This was a deliberate plan to map the throughput scaling curve of the GLM-5-NVFP4 model across the full range of concurrency the server could handle.
The motivation was rooted in a critical discovery made earlier in the session: the model was compute-bound, not communication-bound. The assistant had confirmed this by benchmarking a TP4+PP2 configuration and finding it ~2× slower than TP8 ([msg 899]). This meant that the bottleneck was not inter-GPU communication (allreduce latency) but rather the raw computational efficiency of the FP4 GEMM kernels on the Blackwell SM120 architecture. If the model was compute-bound, then the only way to increase throughput was to increase the batch size per expert — which meant pushing concurrency higher.
The assistant's reasoning, visible in the analysis at [msg 891], was precise: with 256 experts and 8 activated per token, each expert sees approximately batch_size * 8 / 256 tokens. At 512 concurrent requests, that's ~16 tokens per expert; at 1024, ~32 tokens per expert. These tiny GEMM operations (16×2048×6144) have an arithmetic intensity of roughly 16 ops/byte — about 72× below what is needed to be compute-bound. The only lever available was concurrency: more tokens in flight meant larger per-expert batches, which meant higher GPU utilization and better throughput.
The assistant had already seen promising scaling from 256 to 512 to 1024 in [msg 901]. At 256 concurrency, output throughput was 704.64 tok/s. At 512, it rose to 1,105.82 tok/s. At 1024, it reached 1,601.28 tok/s. The curve was still climbing. The natural next question — the one that drove [msg 902] — was: does it keep scaling to 2048?
How Decisions Were Made
Several decisions are embedded in this message, each revealing the assistant's methodology.
The decision to test 2048 concurrency. This was not a random number. The assistant had already tested 1, 256, 512, and 1024. The sequence follows a doubling pattern, which is a standard benchmarking practice: test at geometrically increasing loads to identify the inflection point where throughput saturates. The choice of 2048 specifically was also informed by the server configuration — the --max-running-requests 1024 parameter set in the launch script at [msg 889]. The assistant likely suspected that 2048 would exceed this limit, but testing it would reveal exactly what happens at the boundary.
The choice of benchmark parameters. The assistant used --random-input-len 128 --random-output-len 128, matching the previous benchmarks for comparability. The --request-rate 999 setting effectively disables rate limiting, sending all prompts as fast as possible — this is a "burst" test designed to measure peak server capacity rather than sustained throughput under a controlled arrival rate. The --dataset-name random flag generates synthetic data, avoiding any dataset-specific effects. These choices reflect a commitment to controlled, reproducible measurement.
The decision to grep for specific metrics. Rather than capturing the full benchmark output, the assistant filtered for "Output token throughput|Total token throughput|Peak output|Mean TPOT|Concurrency:" — the five metrics most relevant to the scaling analysis. This is an efficiency choice: the assistant is iterating rapidly through experiments and needs concise, comparable data points.
The decision to run this as a single bash command via SSH. The assistant could have written a Python script, used the SGLang client library, or run the benchmark locally. Instead, it chose to execute directly on the remote server via SSH, capturing output inline. This reflects the session's workflow: the assistant is operating on a remote machine (the LXC container on the Proxmox host) and has been interacting with it through SSH throughout the session.
Assumptions Embedded in the Test
Every benchmark rests on assumptions, and [msg 902] is no exception.
The assumption that throughput would continue to scale. The assistant's language — "let me push to 2048 to see if we scale further" — reveals an expectation that scaling might continue. The results at 256, 512, and 1024 showed increasing throughput, though with diminishing returns: from 704 to 1,105 (57% increase), then to 1,601 (45% increase). The assistant likely expected 2048 to show further gains, perhaps reaching ~2,000 tok/s or more.
The assumption that the server could handle 2048 concurrent requests. The server was launched with --max-running-requests 1024, which caps the number of requests that can be actively processed. The assistant may have assumed this limit would be hit gracefully — that excess requests would queue and the benchmark would measure queuing overhead — or may have been deliberately testing what happens beyond the configured limit.
The assumption that the benchmark tool would handle 2048 prompts correctly. The bench_serving tool sends all prompts concurrently and measures end-to-end performance. At 2048 prompts, the tool itself could become a bottleneck — memory usage, connection handling, and result aggregation all scale with the number of prompts. The assistant implicitly assumed the tool would remain reliable at this scale.
The assumption that 128-token input and output lengths were representative. The model's performance characteristics may differ at longer or shorter sequence lengths. The assistant chose 128 tokens for consistency with earlier benchmarks, but this implicitly assumes that the scaling behavior observed at 128 tokens generalizes.
Mistakes and Incorrect Assumptions
The most significant revelation of [msg 902] is that the assumption of continued scaling was incorrect — or at least incomplete. The output shows 1,639.69 tok/s at 2048 concurrency, compared to 1,601.28 tok/s at 1024. That is only a ~2.4% improvement, well within the noise of measurement variance. The system had plateaued.
The assistant recognized this immediately in the very next message ([msg 903]), writing: "We're plateauing — 2048 only gives 1,640 tok/s vs 1,601 at 1024. We hit the KV cache limit (max_running_requests=1024)." The actual achieved concurrency reported by the benchmark was 1,254.38 — meaning the server never actually processed 2048 requests concurrently. The --max-running-requests 1024 limit capped the effective concurrency, and the remaining requests sat in the queue, contributing to latency (the TPOT jumped from ~300ms at 1024 to 527ms at 2048) but not to throughput.
This reveals an incorrect assumption about the relationship between --num-prompts and --max-running-requests. The assistant assumed that sending 2048 prompts would stress-test the server's ability to handle high concurrency, but the server's internal limit prevented it from ever reaching that concurrency level. The benchmark measured queuing overhead more than throughput capacity.
A subtler issue is that the assistant did not first increase --max-running-requests before testing 2048 concurrency. The server was configured with --max-running-requests 1024 in the launch script at [msg 889]. If the goal was to test scaling to 2048, the logical first step would have been to restart the server with --max-running-requests 2048 and then benchmark. The assistant skipped this step, perhaps because restarting the server would take ~90 seconds (the observed startup time), and the assistant was iterating rapidly.
Another potential oversight: the assistant did not check whether the GPU memory could support 2048 concurrent requests. The KV cache size is proportional to max-running-requests × max-sequence-length × num-layers × hidden-dim × kv-heads × 2 bytes. For a model of this scale, 2048 concurrent requests might exceed the 96 GB of VRAM per GPU, causing OOM errors. The assistant implicitly assumed memory was sufficient, but the plateau may also reflect memory pressure.
Input Knowledge Required to Understand This Message
To fully grasp [msg 902], a reader needs substantial context from the preceding conversation.
Knowledge of the hardware setup. The server runs on 8× NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB VRAM, connected via PCIe in a Proxmox VM with virtualization-induced latency. The GPUs have a 600W TDP and a theoretical FP4 peak of 3,700 TFLOPS (sparse) or 1,850 TFLOPS (dense).
Knowledge of the model architecture. GLM-5-NVFP4 is a Mixture-of-Experts (MoE) model with 256 experts, 8 activated per token. It uses FP4 quantization (modelopt_fp4). The model is deployed via SGLang with FlashInfer backends for attention and MoE computation.
Knowledge of the server configuration. The server was launched with --max-running-requests 1024, --tp-size 8, --mem-fraction-static 0.92, --disable-cuda-graph, and --disable-radix-cache. These parameters directly affect the benchmark results.
Knowledge of the previous benchmark results. The assistant had already established a scaling curve: 10.22 tok/s at concurrency 1, 704.64 at 256, 1,105.82 at 512, and 1,601.28 at 1024. The 2048 result is meaningless without this context.
Knowledge of the compute-bound diagnosis. The assistant had determined that the model is compute-bound, not communication-bound, based on the TP4+PP2 comparison and the FP4 GEMM kernel efficiency analysis. This diagnosis motivated the concurrency scaling experiments.
Knowledge of the kernel tile limitations. The assistant had discovered that only 128×128 tile configurations work on SM120; the larger M128×N256 and M256×N128 tiles fail due to shared memory constraints. This means each kernel tile processes only 128×128×K operations, limiting computational density.
Output Knowledge Created by This Message
[msg 902] produced several pieces of knowledge that shaped the subsequent direction of the optimization effort.
The throughput ceiling. The most concrete output is the discovery that throughput plateaus at approximately 1,640 tok/s (output) / 3,249 tok/s (total) for this configuration. This became the baseline against which all future optimizations would be measured.
The concurrency ceiling. The benchmark revealed that the server's effective concurrency caps at ~1,254, far below the 2,048 prompts sent. This directly implicates the --max-running-requests 1024 limit and motivated the assistant to increase it in later experiments.
The latency explosion. The TPOT jumped from ~300ms at 1024 concurrency to 527ms at 2048, even though throughput barely increased. This reveals that beyond the server's capacity, additional requests primarily accumulate queueing delay rather than being processed. This is classic queueing theory behavior: once utilization approaches 100%, latency goes to infinity while throughput stabilizes.
The need to increase max-running-requests. The plateau directly motivated the assistant to investigate server parameter tuning. In the following messages ([msg 903] and beyond), the assistant checked GPU power draw (finding only ~235W per GPU out of 600W TDP), then began systematically tuning --max-running-requests and --num-continuous-decode-steps to push the ceiling higher.
The validation of the benchmarking methodology. The fact that the 2048 result so cleanly matches the 1024 result (within 2.4%) validates that the benchmark is producing consistent, reproducible measurements. This is important for the credibility of all subsequent optimization experiments.
The Thinking Process Visible in the Message
Although [msg 902] is brief, the thinking process is visible in its structure and timing.
The assistant begins with "Good — now let me push to 2048 to see if we scale further." The word "Good" acknowledges the successful completion of the previous benchmarks (256, 512, 1024) and signals a transition to the next experiment. "Push" is an apt verb — it conveys the idea of testing the system's limits, of applying pressure to find the breaking point.
The command itself is carefully constructed. The assistant uses && to chain commands, ensuring the echo runs before the benchmark. The 2>&1 redirect merges stderr into stdout, capturing any error messages. The grep -E pipeline extracts only the essential metrics. These details reveal a methodical approach: the assistant is not just running a command but engineering the output for easy parsing.
The choice of --request-rate 999 is particularly telling. This is not a realistic request rate — no production system would sustain 999 requests per second for long. But the assistant is not testing realistic scenarios; it is testing the server's maximum capacity under ideal conditions. This is a stress test, not a performance characterization. The thinking is: "Remove all rate-limiting constraints and see how fast the server can go."
The fact that the assistant did not immediately restart the server with higher --max-running-requests before testing 2048 reveals a trade-off in the thinking process. The assistant could have spent 90 seconds restarting the server with a higher limit, then benchmarked. Instead, it chose to benchmark first with the existing configuration, accepting that the results would be constrained. This suggests the assistant was thinking: "Let me first see what happens at 2048 with the current config. If it plateaus, I'll know the limit matters and can adjust. If it scales, I'll know the limit wasn't binding." This is a valid experimental strategy — test first, diagnose the bottleneck, then fix it.
Broader Significance in the Optimization Journey
[msg 902] sits at a critical inflection point in the optimization session. Before this message, the assistant had been exploring the problem space: diagnosing the compute-bound nature of the workload, analyzing kernel tile configurations, understanding the SM120 architecture limitations. After this message, the assistant pivots to solution mode: increasing --max-running-requests, tuning --num-continuous-decode-steps, exploring expert parallelism, and eventually achieving a 28% throughput improvement to reach 2,095 output tok/s and 4,151 total tok/s (as documented in the segment summary).
The plateau at 2048 was the reality check that forced the assistant to confront the server configuration limits. Without this data point, the assistant might have continued assuming that higher concurrency would yield linear scaling. The benchmark revealed that the system had a bottleneck — not in the GPUs' compute capacity, but in the server's request scheduling and KV cache management. This shifted the optimization focus from kernel-level improvements (which required modifying CUTLASS or FlashInfer source code) to server-level parameter tuning (which required only configuration changes).
Conclusion
Message [msg 902] is a masterclass in systematic benchmarking. In a single command, it encapsulates the assistant's methodology: test at the boundary, measure precisely, and let the data speak. The result — a throughput plateau at ~1,640 tok/s — was not what the assistant hoped for, but it was exactly what the assistant needed to see. It revealed the --max-running-requests limit as the binding constraint, motivated the subsequent parameter tuning, and established the baseline against which all future improvements would be measured.
The message also illustrates a deeper principle of performance optimization: you cannot optimize what you have not measured. The assistant did not assume that 2048 would scale; it tested. It did not guess at the bottleneck; it benchmarked. And when the data showed a plateau, it did not ignore it or rationalize it away; it used it as a signal to change direction. This is the essence of evidence-driven engineering, and [msg 902] is a perfect example of a small message that made a large difference.