The Commit That Captured a Plateau: Quantifying the MoE Throughput Lever

In the middle of a sprawling, multi-session effort to build a native speculative decoding engine for the Kimi K2.6 model on NVIDIA Blackwell GPUs, there arrives a message that at first glance appears to be mere housekeeping: a git commit followed by a service health check. But message [msg 12013] is far more than that. It is the moment when a week of careful benchmarking, kernel development, and performance analysis crystallizes into a single, actionable insight — and the moment when the assistant must balance the demands of research with the practical responsibility of keeping a production service running.

The message reads in full:

[bash] cd /home/theuser/glm-kimi-sm120-rtx6000bw/kdtree-engine git add -A && git -c user.name=theuser -c user.email=theuser@localhost commit -q -m "Phase 3: quantify INT4 Marlin MoE GEMM throughput lever at K2.6 scale on PRO 6000

>

- bench_marlin_moe.py: drives SGLang's real fused_marlin_moe (moe_wna16_marlin_gemm, the kernel K2.6 uses) with K2.6 MoE dims (E=384, hidden 7168, inter 2048, top-8, INT4, group 32) across M. - Result (single GPU, full E=384): time plateaus ~7ms for M=256..1024 while us/token drops 150->6.8 (4x). Two regimes: weight-streaming-bound (M<256, time ~ active experts) vs compute-bound plateau (M>=256, all experts streamed once, extra tokens nearly free). - Confirms + refines the thesis: drive verify batch M=streamsq_len into the plateau (>=256) to amortize the expert-weight read; throughput then scales with streams at flat MoE cost. Sweet spot: budget 32-64 + 8-16 streams. C=1 can't reach the plateau (stays weight-streaming-bound). TP-8 caveat: per-GPU time ~ /8. Writeup in docs/BENCHMARKS_CT200.md s7." 2>&1 | tail -2 echo "=== poll service recovery (model reload ~min) ===" for i in 1 2 3 4 5 6; do sleep 25; r=$(timeout 15 ssh -o StrictHostKeyChecking=no root@10.1.2.200 'curl -s --max-time 10 http://127.0.0.1:30001/v1/completions -H "Content-Type: application/json" -d "{\"model\":\"/root/models/Kimi-K2.6\",\"prompt\":\"hi\",\"max_tokens\":3,\"temperature\":0}" 2>/dev/null | head -c 60' 2>/dev/null); if echo "$r" | grep -q choices; then echo "READY after ~$((i25))s: $r"; break; else echo " not ready (${i})"; fi; done === poll service recovery (model reload ~min) === not ready (1) not ready (2) not ready (3) not ready (4)

The message is bisected into two acts. The first is a git commit that documents the central finding of Phase 3 of the DDTree engine project. The second is a polling loop that checks whether the production SGLang service — temporarily stopped to free GPU memory for the benchmark — has recovered. The two acts are connected by a single thread of responsibility: the assistant generated knowledge, preserved it, and then immediately turned to restoring the infrastructure that made that knowledge possible.

Why This Message Was Written

The message exists because of a chain of decisions that began several rounds earlier. The assistant had been building a native C/C++/CUDA speculative decoding engine for the Kimi K2.6 model, a massive 1-trillion-parameter mixture-of-experts (MoE) transformer. A critical open question was how the MoE layer's performance scales with batch size — specifically, how many tokens need to be batched together in the DDTree verify phase before the per-token cost of the MoE computation becomes negligible. This is the "throughput lever" that determines how many streams and how much draft budget the engine should use.

To answer this, the assistant wrote bench_marlin_moe.py, a benchmark that drives SGLang's real fused_marlin_moe kernel — the same INT4 Marlin GEMM kernel that the production K2.6 service uses — at the model's actual dimensions: 384 experts, hidden dimension 7168, intermediate dimension 2048, top-8 routing, INT4 quantization with group size 32. The benchmark sweeps the token batch size M from 1 to 1024 and measures the kernel's latency.

But running this benchmark required stopping the production service. The Marlin-repacked weights for all 384 experts occupy roughly 8.5 GB, and during the repacking process the peak memory usage reaches approximately 16.8 GB — far exceeding the ~9 GB of free memory available on any GPU while the service is running. The assistant made a deliberate decision: stop the service, run the benchmark at full scale on a freed GPU, then restart. This was a calculated risk, since restarting the K2.6 service takes 6–10 minutes due to model loading, JIT compilation of Triton kernels, and CUDA graph capture.

The benchmark completed successfully. The results were dramatic and immediately useful. Message [msg 12013] is the moment those results were committed to the repository and the service recovery was initiated.

The Commit as a Document of Discovery

The commit message in [msg 12013] is unusually detailed for a git commit, and for good reason: it encapsulates the core finding that will drive the next phase of the engine's design. The message distills the benchmark into two regimes:

Regime 1 — Weight-streaming-bound (M < 256): At small batch sizes, the MoE kernel's latency is proportional to the number of active experts. Each token activates only its top-8 experts, so the GPU spends most of its time streaming expert weights from HBM into the compute units. The per-token cost is high — 150 microseconds at M=1.

Regime 2 — Compute-bound plateau (M ≥ 256): Once the batch size reaches approximately 256 tokens, all 384 experts have been activated at least once, and their weights are already in flight. Adding more tokens to the batch costs almost nothing. The per-token cost drops to 6.8 microseconds at M=1024 — a 22× improvement over the M=1 case. The total step time plateaus at approximately 7 milliseconds, remaining flat even as M doubles from 256 to 1024.

This is the kind of finding that changes the design of a system. Before the benchmark, the assistant had been targeting "1–10 streams" as a plausible operating range. The benchmark revealed that this range would barely enter the plateau — 10 streams with a budget of 32 (sequence length 33) gives M = 10 × 33 = 330, just barely crossing the 256 threshold. The sweet spot, the commit message declares, is "budget 32-64 + 8-16 streams," which pushes M into the 264–1040 range where the MoE cost is fully amortized.

The commit also includes an important caveat: the benchmark was run on a single GPU, but the production service uses tensor parallelism across 8 GPUs (TP8). Under TP8, each GPU holds one-eighth of the expert weights, so the per-GPU MoE cost is approximately one-eighth of the single-GPU measurement. The plateau shape remains the same, but the absolute times shrink. This caveat prevents the single-GPU numbers from being misinterpreted as the actual per-layer cost in the production system.

The Service Recovery Poll: Practical Responsibility

The second half of the message is a polling loop that checks whether the SGLang service has recovered after being stopped for the benchmark. The assistant uses a for loop with 25-second sleeps and a curl command that sends a minimal completion request to the service's HTTP endpoint. If the response contains the word "choices," the service is considered ready.

The polling reveals that the service is not ready after 4 cycles (100 seconds). The shell tool then terminates the command after exceeding a 120-second timeout. This is not a failure — the assistant correctly anticipated that model reload takes 6–10 minutes, and the 120-second timeout was simply too short for the polling loop to complete. The next message in the conversation ([msg 12014]) shows the assistant checking the systemd journal to monitor the actual reload progress, confirming that the service is still loading.

This polling loop reveals an important assumption: the assistant assumed the service might recover within the 120-second window, or at least that partial progress would be visible. In practice, the model loading process is opaque to HTTP polling — the service doesn't start listening on port 30001 until the model is fully loaded and all 8 GPUs have synchronized their weight shards. The "not ready" responses are expected behavior, not a sign of trouble.

The Deeper Technical Context

To fully appreciate what [msg 12013] accomplishes, one must understand the problem it solves. The DDTree speculative decoding engine works by having a small "drafter" model propose multiple candidate tokens in parallel (organized as a tree), and then the large "target" model verifies all of them in a single forward pass. The verify pass processes M tokens simultaneously, where M = number_of_streams × sequence_length. The throughput of the system depends critically on how the verify pass's cost scales with M.

Before this benchmark, the scaling behavior of the INT4 Marlin MoE kernel at K2.6 dimensions was unknown. The assistant had theoretical models suggesting that weight streaming would dominate at small M and that a compute-bound plateau would appear at larger M, but the exact transition point and the shape of the curve were uncertain. The benchmark provided the numbers: the transition occurs at M ≈ 256, and the plateau is remarkably flat from M=256 to M=1024.

This finding has direct engineering implications. It means that the DDTree engine should be designed to push M into the 256–1024 range during the verify phase. This can be achieved by increasing the number of streams (each stream generates a separate draft path) or by increasing the draft budget (the length of each draft path). The sweet spot identified in the commit — budget 32–64 with 8–16 streams — gives M values of 264–1040, squarely in the plateau.

The finding also explains why C=1 (autoregressive decoding without speculation) is fundamentally disadvantaged: with M=1, every step operates in the weight-streaming regime, paying the full 150 µs per token in MoE cost alone. Speculative decoding with a large enough verify batch can reduce this to under 7 µs per token — a 20× improvement in the MoE component alone.

Input Knowledge and Output Knowledge

To understand this message, a reader needs substantial background knowledge: the architecture of mixture-of-experts transformers, the INT4 Marlin quantization format and its GEMM kernel, the concept of tensor parallelism, the DDTree speculative decoding algorithm, and the operational details of SGLang's service deployment with systemd. The message assumes familiarity with the K2.6 model dimensions (E=384, hidden=7168, inter=2048) and the benchmark methodology that produced the numbers.

The message creates several forms of output knowledge. First, it produces a permanent, version-controlled record of the benchmark finding in the git repository. Second, it refines the design target for the DDTree engine: the sweet spot of budget 32–64 with 8–16 streams. Third, it provides a quantitative basis for the claim that C=1 autoregressive decoding cannot reach the plateau regime. Fourth, it documents the TP-8 caveat, preventing the single-GPU numbers from being misinterpreted.

Conclusion

Message [msg 12013] is a transition point in the larger narrative of building a speculative decoding engine. It is the moment when measurement yields to design, when analysis yields to action. The benchmark results are committed, the design targets are refined, and the assistant turns to the practical task of restoring the production service. The polling loop's timeout is a reminder that even carefully planned operations have unpredictable durations — but the core work is done. The plateau has been found, measured, and documented. The throughput lever is quantified. The engine's design now has a solid empirical foundation.