The Plateau at 912 Tok/s: A Diagnostic Benchmarking Pivot in GLM-5 Inference Optimization

Introduction

In the long arc of deploying a 744-billion-parameter Mixture-of-Experts (MoE) model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, there comes a moment when the numbers stop climbing. Message 656 in this opencode session captures precisely such a moment — a transitional message where the assistant, having just achieved 879.87 total tokens per second at 256 concurrent requests, pushes further to 512 concurrency and observes only marginal improvement to 912.70 tok/s. This message is not about a breakthrough; it is about the recognition that a breakthrough is needed elsewhere. It is a diagnostic pivot, a quiet admission that the current configuration has hit a ceiling, and that the path forward lies in the deeper, more complex work of MoE kernel tuning.

The Message in Full

The assistant writes:

256 concurrency gave us 879.87 total tok/s — getting closer to 1k! Let me find the model config and read more of FINDINGS.md for the MoE tuning instructions.

This is followed by three tool calls executed in parallel:

A bash command to find the GLM-5 model's config.json on disk: find /shared/huggingface/hub/ -name 'config.json' -path '*GLM-5*'
A bash command to run the sglang benchmark at 512 concurrency with random 256-token inputs and 128-token outputs
A read command to continue reading the local research repository's FINDINGS.md file (starting from line 100) The results returned are: - The model config is located at /shared/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/config.json - The 512-concurrency benchmark yields 912.70 total tok/s (612.52 input, 300.18 output, peak output 474) - The FINDINGS.md content at line 100+ discusses Triton shared memory exhaustion crashes on longer requests

Why This Message Was Written: The Reasoning and Motivation

To understand why this message exists, we must understand the context that precedes it. The assistant has been working for hours — across multiple segments of this conversation — to deploy the GLM-5-NVFP4 model on a Proxmox-hosted LXC container with 8 Blackwell GPUs. The journey has been arduous: fixing CUDA initialization by disabling HMM in the nvidia_uvm module, patching sglang for SM120 shared memory constraints, resolving flash-attn build issues, and migrating from a KVM VM to an LXC container to eliminate VFIO/IOMMU barriers to GPU P2P communication.

The immediate prior message (msg 655) had set a clear agenda: "Let me look at the GLM-5 model config to confirm MoE parameters, and also check our prior FINDINGS.md for the exact tuning commands, while running the benchmarks in parallel." This was the plan. Message 656 is the execution of that plan — but with a critical shift in tone.

The first benchmark at 256 concurrency (run in msg 655) yielded 879.87 tok/s. The assistant's reaction — "getting closer to 1k!" — is cautiously optimistic. But the real motivation for this message is the recognition that 879 tok/s is still below the 1k+ target, and that simply throwing more concurrency at the problem may not be the solution. The 512-concurrency benchmark confirms this suspicion: the gain from 256 to 512 concurrency is only about 33 tok/s (from 879 to 912), a mere 3.7% improvement. This is a classic sign of hitting a throughput ceiling — the system is saturating some bottleneck that is not request concurrency.

The assistant's motivation is thus diagnostic: it needs to understand why throughput is plateauing, and it needs the model's architectural parameters (expert count, hidden size, intermediate size) to begin the MoE kernel tuning process. The parallel execution of the find command, the benchmark, and the FINDINGS.md read reflects an efficient, multi-threaded approach to gathering the information needed for the next phase of optimization.

How Decisions Were Made

This message is notable for what it reveals about the assistant's decision-making process. Three decisions are implicit in the tool calls:

Decision 1: Run the 512-concurrency benchmark. The assistant could have jumped straight to MoE tuning, but instead chose to first characterize the system's scaling behavior. This is a sound engineering practice: before investing time in complex kernel optimization, confirm that the bottleneck is not simply insufficient concurrency. The 256→512 result (only 3.7% gain) provides strong evidence that the bottleneck lies elsewhere — likely in the MoE kernel execution or inter-GPU communication.

Decision 2: Locate the model config via find rather than the earlier broken path. In msg 655, the assistant had attempted to read the config using a shell glob pattern ($(ls .../snapshots/)) which failed because the snapshots directory didn't exist at the expected path. The find command is a more robust approach that searches the entire huggingface cache tree. This decision reflects learning from a previous failure.

Decision 3: Read FINDINGS.md starting from line 100. The assistant had already read the beginning of FINDINGS.md in msg 655. By starting at line 100, it skips the introductory material and dives into the section about "Configuration 2: SGLang with triton..." which discusses shared memory crashes. This is a targeted read — the assistant is looking for specific information about MoE kernel tuning procedures and SM120-specific issues.

Assumptions Made

Several assumptions underpin this message:

Assumption 1: The throughput ceiling is due to MoE kernel performance. The assistant's pivot to reading FINDINGS.md and locating the model config assumes that kernel tuning is the next lever to pull. This is a reasonable assumption given the prior research (the Kimi K2 deployment achieved 5,816 tok/s on similar hardware with tuned MoE kernels), but it is not yet proven. The bottleneck could also be in the allreduce communication, the attention backend, or the CPU-side scheduling.

Assumption 2: The existing FINDINGS.md contains applicable knowledge for GLM-5. The prior research was conducted on Kimi K2 (GLM-4.7 architecture) which has E=161 experts. GLM-5 has E=256 experts — a substantially different configuration. The assistant assumes that the tuning methodology transfers even if the specific configs do not.

Assumption 3: The model config path found by the find command is correct and accessible. The path includes a snapshot hash (6944a23f9ffb9668ac970901526c6cc72c34f4e2), which is a standard Hugging Face cache structure. The assistant assumes this path will work for subsequent Python code that reads the config.

Assumption 4: Higher concurrency will eventually show diminishing returns. The assistant implicitly assumes that the system has a finite throughput capacity and that concurrency scaling will saturate. The 512 benchmark confirms this assumption, but the assistant does not yet know the exact saturation point or the primary limiting factor.

Mistakes or Incorrect Assumptions

The 512-concurrency benchmark may be misleading due to benchmark duration. The benchmark ran with --request-rate inf, meaning requests are dispatched as fast as possible. At 512 prompts with 256 input + 128 output tokens each, the benchmark completes relatively quickly. The "Peak concurrent requests: 512" line confirms that all 512 requests were in flight simultaneously. However, the total token throughput of 912 tok/s is an aggregate over the entire benchmark run, which may include warm-up and cool-down phases that dilute the peak throughput. The true peak sustainable throughput might be higher than 912 tok/s if measured over a longer steady-state period.

The assumption that FINDINGS.md knowledge transfers may be optimistic. The prior research achieved 5,816 tok/s on Kimi K2, but that was with extensive MoE kernel autotuning specific to that model's expert count (E=161) and intermediate size. GLM-5 has E=256 experts and a different architecture (glm_moe_dsa with DeepSeek Sparse Attention). The tuning procedure may need significant adaptation, and the assistant may need to regenerate all configs from scratch rather than adapting existing ones.

The assistant may be overestimating the impact of MoE tuning. While MoE kernel tuning was the single biggest performance win for Kimi K2, the GLM-5 deployment has additional complexities: NVFP4 quantization, DeepSeek Sparse Attention, and the flashinfer allreduce fusion that the assistant later discovers is unsupported on SM120. The 912 tok/s ceiling may have multiple contributing factors, and fixing only the MoE kernels may not yield the expected gains.

Input Knowledge Required to Understand This Message

To fully grasp this message, one needs:

Knowledge of the hardware stack: Eight NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture, 96GB each), connected via PCIe with NODE-level P2P within NUMA domains and SYS-level across NUMA domains. The GPUs have 100KB shared memory per SM (versus 228KB on datacenter Blackwell), which constrains Triton kernel block sizes.

Knowledge of the model architecture: GLM-5-NVFP4 is a 744B-parameter MoE model with 256 experts, 8 activated per token, using NVFP4 quantization for expert weights and BF16 for attention. It uses DeepSeek Sparse Attention (DSA) and has 78 layers (3 dense + 75 MoE).

Knowledge of the sglang inference stack: The server is running with tensor parallelism 8, flashinfer attention backend, trtllm NSA backends, flashinfer_cutlass MoE runner, and flashinfer allreduce fusion enabled. The benchmark tool (sglang.bench_serving) measures throughput at various concurrency levels.

Knowledge of the prior research context: The local repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ contains extensive findings from a previous deployment of Kimi K2 (a similar MoE model) on the same hardware, including MoE kernel tuning procedures and performance benchmarks.

Knowledge of the session's history: The assistant has already resolved CUDA initialization issues, confirmed P2P GPU access in the LXC container, and run benchmarks at 32, 64, and 128 concurrency (438, 757, and 806 tok/s respectively). The current 879→912 tok/s results represent incremental progress.

Output Knowledge Created by This Message

This message produces several pieces of actionable knowledge:

The 512-concurrency throughput baseline: 912.70 total tok/s (612.52 input, 300.18 output). This is the highest concurrency tested so far and establishes that the system does not scale linearly beyond 256 concurrent requests. The peak output token throughput of 474 tok/s is also recorded.

The model config file location: /shared/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/config.json. This path is now known and can be used in subsequent Python scripts to read the model's architectural parameters (num_experts, hidden_size, moe_intermediate_size, etc.).

Confirmation that the FINDINGS.md research is accessible and relevant: The read operation confirms that the file contains information about SM120 shared memory crashes and MoE kernel configuration, which the assistant will need for the tuning phase.

Evidence of diminishing returns from concurrency scaling: The 256→512 gain of only 33 tok/s (3.7%) strongly suggests that the bottleneck is not request concurrency but rather kernel execution efficiency or inter-GPU communication.

The Thinking Process Visible in the Reasoning

The assistant's reasoning is visible in the structure and sequencing of the message. The opening line — "256 concurrency gave us 879.87 total tok/s — getting closer to 1k!" — reveals an optimistic but measured assessment. The assistant is tracking progress against the user's target of 1k+ tok/s, and 879 is tantalizingly close. Yet the tone is not triumphant; it is transitional. The "Let me find the model config and read more of FINDINGS.md" signals a shift from brute-force concurrency scaling to targeted kernel optimization.

The parallel execution of three tool calls reveals a multi-pronged reasoning strategy:

Benchmark to characterize: Push concurrency to 512 to see if the ceiling is higher
Find to enable: Locate the model config to extract architectural parameters
Read to learn: Consult prior research for tuning methodology This is not a message that makes a single decision; it is a message that gathers the information needed for the next set of decisions. The assistant is in a diagnostic loop — it has observed a symptom (plateauing throughput), formulated a hypothesis (MoE kernel bottleneck), and is now collecting the data needed to test that hypothesis. The fact that the 512 benchmark returns only 912 tok/s — a mere 33 tok/s improvement over 256 concurrency — is itself a piece of reasoning evidence. The assistant does not comment on this result explicitly in the message (the benchmark output appears as a tool result), but the implication is clear: the concurrency lever is nearly exhausted. The real work lies ahead in kernel tuning.

Conclusion

Message 656 is a quiet but pivotal moment in the GLM-5 deployment. It is the moment when the assistant recognizes that the easy gains have been harvested and the hard work must begin. The 912 tok/s plateau is not a failure — it is a signal. It tells the assistant that the system's throughput is constrained not by how many requests it can juggle, but by how efficiently it can execute the MoE computation that dominates each forward pass. The message's true significance lies not in the numbers it reports, but in the pivot it initiates: from concurrency scaling to kernel tuning, from brute force to surgical optimization. The assistant has found the ceiling; now it must find the lever to break through it.