The Strategic Pivot: Planning the Next Phase of GLM-5 Inference Optimization

Introduction

In any complex engineering endeavor, there comes a moment when the team must step back, assess the current state, and chart a new course. Message 652 of this opencode session represents exactly such a moment. After successfully migrating a massive 744B-parameter Mixture-of-Experts model (GLM-5-NVFP4) from a KVM virtual machine to an LXC container to achieve bare-metal GPU topology—a journey spanning dozens of messages, kernel panics, CUDA initialization bugs, and PCIe topology investigations—the assistant confronts a sobering reality: despite all that effort, performance is essentially unchanged. The message is brief, barely a paragraph of reasoning followed by a structured todo list, but it encapsulates a critical strategic decision that shapes the next hundred messages of work. This article examines that message in depth: why it was written, what assumptions it encodes, what knowledge it draws upon, and what it sets in motion.

The Context: From KVM to LXC — Why Performance Still Lags

To understand message 652, one must first appreciate the journey that preceded it. The session began with deploying GLM-5-NVFP4 on a Proxmox-hosted KVM virtual machine with 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The initial performance was disappointing: approximately 485 tok/s peak at 64 concurrency, and a mere 11 tok/s for single-stream inference. The root cause was identified as VFIO/IOMMU overhead: the KVM hypervisor's PCIe passthrough mechanism placed each GPU in its own IOMMU group, preventing direct peer-to-peer (P2P) DMA between GPUs. Instead of the fast NVLink-like P2P transfers that modern multi-GPU inference depends on, all cross-GPU communication was forced through the host's PCIe root complex (PHB topology), adding ~13µs of latency per transaction.

The solution seemed clear: eliminate VFIO entirely by migrating to an LXC container, which shares the host kernel and thus presents a bare-metal GPU topology to the CUDA runtime. This required fixing a CUDA initialization bug (cuInit error 3) caused by NVIDIA's HMM (Heterogeneous Memory Management) feature, patching the kernel module parameters with uvm_disable_hmm=1, and carefully configuring the LXC container's device mounts. After this multi-hour effort, the assistant confirmed that the LXC container indeed showed NODE/SYS GPU topology (the bare-metal topology) with P2P bandwidth of 53.76 GB/s same-NUMA and 40.24 GB/s cross-NUMA—a dramatic improvement over the KVM VM's PHB-limited topology.

Yet when the assistant ran benchmarks, the results were deflating. At 32 concurrency: 438 tok/s. At 64: 757 tok/s. At 128: 806 tok/s. Single-stream: still ~11 tok/s. The LXC container was essentially matching the KVM VM, not surpassing it. The better P2P bandwidth was not translating into higher throughput.

This is the precise moment captured in message 652. The assistant has just received these disappointing benchmark results (messages 646–648) and the user's open-ended prompt: "Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed" (msg 651). The assistant must now decide what to do.

The Message Itself — A Strategic Pivot Point

Here is the complete text of message 652:

Looking at the current state, we have a running server but performance is below target (~800 tok/s peak vs 1k+ goal, ~11 tok/s single-stream vs >100 tok/s goal). The most promising next steps from our research are:

>

1. Tune MoE kernels for GLM-5 (E=256) on SM120 — this was the single biggest performance win for Kimi K2 2. Test higher concurrency levels (256, 512, 1024) — we only tested up to 128 3. Try TP4+PP2 — showed benefits for Kimi K2

>

Let me start by checking the current state of things and then work on MoE kernel tuning.

>

[todowrite] {"todos":[{"content":"Check current server status and GPU utilization in LXC container","priority":"high","status":"in_progress"},{"content":"Run higher concurrency benchmarks (256, 512, 1024) to find throughput ceiling","priority":"high","status":"pending"},{"content":"Tune MoE kernels for GLM-5 (E=256, N=192) on SM120 RTX PRO 6000","priority":"high","status":"pending"},{"content":"Restart server with tuned MoE configs and ..."}]}

The message is structured in three parts: a situational assessment, a prioritized list of interventions drawn from prior research, and a structured todo list with explicit status tracking. The tone is measured and analytical. There is no frustration or panic despite the disappointing results—just a calm re-evaluation and a decision to proceed with the most promising avenue.

Reasoning and Decision-Making Process

The assistant's reasoning is visible in the way it prioritizes. Three candidate interventions are identified, all drawn from the local research repository at /home/theuser/glm-kimi-sm120-rtx6000bw/, which contains extensive findings from deploying a similar model (Kimi K2-Thinking NVFP4) on the same RTX PRO 6000 hardware. The assistant explicitly references this prior work as the basis for its decisions.

The first and highest-priority intervention is MoE kernel tuning. This is not a guess—it is grounded in the prior research, which achieved 5,816 tok/s on Kimi K2 at 2048 concurrency. The key insight from that work was that the default MoE kernel configurations in sglang are tuned for datacenter GPUs (H100/B200 with SM90/SM100 architecture) and perform poorly on consumer Blackwell (SM120) hardware. The SM120 architecture has only 100KB of shared memory per SM compared to 228KB on SM100, meaning Triton kernels designed for the larger shared memory will either crash or run suboptimally. The prior research included a tuning script (sglang/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py) that generates hardware-specific MoE kernel configurations by benchmarking different block sizes and launch parameters. The assistant's reasoning is straightforward: if MoE tuning was the single biggest performance win for Kimi K2, it should be the primary lever for GLM-5 as well.

The second intervention—testing higher concurrency—is a simpler diagnostic. The benchmarks so far only went up to 128 concurrent requests, and the throughput curve was still climbing (from 438 tok/s at 32 to 806 tok/s at 128). It is possible that the system simply needs more concurrent requests to saturate the GPUs. The assistant plans to test at 256, 512, and 1024 concurrency to find the true throughput ceiling.

The third intervention—TP4+PP2 (Tensor Parallelism 4 + Pipeline Parallelism 2)—is a more speculative architectural change. The prior research found that for short-context workloads, splitting the 8 GPUs into two groups of 4 (TP4) with pipeline parallelism between them (PP2) can outperform a single TP8 group. This is counterintuitive because TP8 has more GPUs collaborating on each layer, but the allreduce overhead scales quadratically with the number of GPUs in the TP group. For short sequences where the computation-to-communication ratio is low, the reduced allreduce cost of TP4 can outweigh the pipeline bubble overhead of PP2. The assistant flags this as a future option but does not prioritize it immediately.

The todo list reveals the assistant's execution plan: first check the current server status (is it still running? are GPUs idle?), then run higher concurrency benchmarks to establish the baseline, then tune the MoE kernels, and finally restart the server with the tuned configurations. This is a rational, staged approach: gather data before making changes, then apply the most impactful change first.

Assumptions Underlying the Plan

Message 652 rests on several key assumptions, some of which prove correct and others that turn out to be incomplete.

The central assumption is that MoE kernel tuning is the primary bottleneck. The assistant believes that the default MoE configurations are suboptimal for SM120 and that generating hardware-specific configs will unlock significant throughput. This assumption is reasonable—the prior Kimi K2 work demonstrated exactly this pattern—but it implicitly assumes that the MoE computation is the dominant term in the inference latency budget. If other components (attention, allreduce, or the NVFP4 quantization path) are equally constrained, MoE tuning alone may not suffice.

A second assumption is that the Kimi K2 experience transfers directly to GLM-5. Both models use a similar MoE architecture (DeepSeek-style with grouped experts), but GLM-5 has 256 experts (E=256) with 8 activated per token, while Kimi K2 had 161 experts (E=161). The expert count affects the MoE kernel's launch parameters—the number of expert groups, the group size, and the routing patterns all change. The existing tuned configs in the research repo are for E=161 and cannot be reused. The assistant recognizes this ("need new ones for GLM-5 E=256") but assumes that running the same tuning script with the new parameters will produce equally effective configs.

A third assumption is that higher concurrency will continue to improve throughput. The benchmarks show throughput still rising at 128 concurrency (806 tok/s, up from 757 at 64), so the assistant expects further gains at 256, 512, and 1024. This assumes that the server's --max-running-requests 64 setting is not artificially capping concurrency and that the GPUs have sufficient memory for the additional KV cache. The model has ~495K tokens of KV cache capacity (FP8), and at 256 input + 128 output tokens per request, the system should support hundreds of concurrent requests before hitting memory limits.

A more subtle assumption is that the LXC container's better P2P bandwidth is not the limiting factor. The assistant has already confirmed that P2P is working (53 GB/s same-NUMA) and that the topology is bare-metal. The fact that throughput did not improve versus the KVM VM suggests that P2P bandwidth was not the bottleneck in the first place—or that the bottleneck has shifted to something else. The assistant implicitly assumes that the bottleneck is now in the compute kernels (MoE) rather than in communication.

Input Knowledge Required

To understand message 652, a reader needs substantial context from the preceding segments. This includes:

Hardware knowledge: The system has 8× NVIDIA RTX PRO 6000 Blackwell GPUs (SM120 architecture, 96GB VRAM each), an AMD EPYC 9335 CPU with 64 cores/128 threads across 2 NUMA nodes, and ~516GB RAM. The GPUs are connected via PCIe Gen5, with GPUs 0–3 on NUMA node 0 and GPUs 4–7 on NUMA node 1.

Model knowledge: GLM-5-NVFP4 is a 744B-parameter Mixture-of-Experts model with 256 experts (8 activated per token), using NVFP4 quantization (expert MLPs in FP4, attention in BF16). It uses DeepSeek Sparse Attention (DSA) and has 78 layers (3 dense + 75 MoE). The model consumes ~405GB on disk and loads ~63GB per GPU across 8 GPUs.

Infrastructure knowledge: The system runs Proxmox VE with a privileged LXC container (CT 129) that has direct GPU device mounts. The Python environment uses torch 2.9.1+cu128, flashinfer 0.6.3, sgl_kernel 0.3.21, and transformers 5.2.0. The sglang source is built from the GitHub main branch with PR #14311 (SM120 shared memory fix) already merged.

Prior research knowledge: The local repository at /home/theuser/glm-kimi-sm120-rtx6000bw/ contains 871 lines of findings from deploying Kimi K2 on the same hardware, including MoE tuning instructions, TP4+PP2 results, and performance comparison tables. The key finding is that MoE kernel tuning was the single biggest performance lever for Kimi K2.

Benchmark data: The current throughput is ~800 tok/s peak at 128 concurrency, with ~11 tok/s single-stream. The target is 1,000+ tok/s total and >100 tok/s single-stream.

Output Knowledge Created

Message 652 creates a structured plan that drives the next phase of the session. The todo list is not merely decorative—it is a machine-readable specification that the assistant will use to track progress across subsequent messages. Each todo item has a status (in_progress, pending) and a priority (high), creating a lightweight project management system within the conversation.

The message also establishes a decision framework for the assistant's own future actions. By explicitly ranking the three interventions, the assistant creates a fallback hierarchy: if MoE tuning does not yield sufficient gains, try higher concurrency; if that also falls short, attempt TP4+PP2. This prevents the assistant from getting stuck in analysis paralysis—it has a clear next action (check server status) and a clear escalation path.

Furthermore, the message creates shared context between the assistant and the user. The user's prompt was open-ended ("Continue if you have next steps, or stop and ask for clarification"), and the assistant's response demonstrates that it has a coherent theory of the problem and a concrete plan. This builds trust and allows the user to step back and let the assistant execute autonomously.

What Actually Happened Next

The chunk summary for segment 6 reveals that the assistant's plan was partially successful but ultimately incomplete. The higher concurrency benchmarks (step 2) were indeed run, and they showed significant improvement: throughput jumped from ~880 tok/s to ~1,950 tok/s at 256 concurrency, ~2,800 at 512, and ~3,740 at 1024 concurrency. This was achieved by enabling FlashInfer CUTLASS MoE autotune for SM120 (a form of MoE tuning, step 1) and raising --max-running-requests from 64 to 1024 (step 2). The assistant also added --disable-cuda-graph and --disable-radix-cache, which were not in the original plan but proved necessary.

However, the assistant then discovered a deeper problem: GPU power draw remained around 250W out of a 600W TDP, indicating severe underutilization. The root cause was that FlashInfer's allreduce fusion was disabled on SM120 because the underlying TRT-LLM communication kernels only supported SM90/SM100. Attempts to patch flashinfer to add SM120 support caused performance to collapse to 236 tok/s, suggesting synchronization issues. This bottleneck was not anticipated in message 652—the assistant assumed MoE kernels were the primary constraint, but the real ceiling turned out to be the allreduce fusion path.

Conclusion

Message 652 is a deceptively simple message that serves as the strategic pivot point for an entire phase of work. It is the moment when the assistant synthesizes dozens of prior messages, acknowledges that the LXC migration did not yield the expected gains, and charts a new course based on prior research. The message's brevity belies its importance: it encodes assumptions about where the bottleneck lies, draws on extensive prior knowledge, and creates a structured execution plan that drives the next hundred messages. While some of those assumptions proved incomplete—the allreduce fusion bottleneck was not anticipated—the message's value lies not in being perfectly prescient but in providing a clear, actionable framework for iterative optimization. It is a masterclass in how to respond to disappointing results: assess calmly, prioritize based on evidence, execute methodically, and be prepared to learn from what the next round of data reveals.