The Confirmation Benchmark: Reproducibility as a Scientific Discipline in ML Systems Optimization

Introduction

In the midst of an intensive benchmarking campaign to maximize throughput of the Kimi K2.6 Mixture-of-Experts model on 8× RTX PRO 6000 Blackwell GPUs connected via PCIe, the assistant sends what appears at first glance to be a routine message: a bash command executing a Python script that runs a few API calls and prints throughput numbers. But this message—message index 11534 in the conversation—is far more than a simple benchmark. It is a carefully calibrated scientific control experiment, a reproducibility check performed after a failed optimization attempt, and a methodological commitment that reveals how the assistant thinks about systems optimization.

The message reads in full:

[assistant] [bash] # Quick confirm the best config reproduces, then try EP4 python3 -u - <<'PY' import json, time, urllib.request, concurrent.futures

CT200 = "10.1.2.200" PORT = 30001 MODEL = "/root/models/Kimi-K2.6"

def api(prompt, max_tokens=2048, timeout=600): payload = json.dumps({"model": MODEL, "messages": [{"role":"user","content":prompt}], "temperature": 0, "max_tokens": max_tokens}) req = urllib.request.Request(f"http://{CT200}:{PORT}/v1/chat/completions", data=payload.encode(), headers={"Content-Type":"application/json"}) t0 = time.perf_counter() with urllib.request.urlopen(req, timeout=timeout) as r: data = json.load(r) wall = time.perf_counter() - t0 ct = data.get("usage", {}).get("completion_tokens", 0) return ct/wall if wall > 0 else 0, wall, ct

print("Warmup...", flush=True) api("Say OK", 16, 120); api("Say OK", 16, 120)

prompts = ["Write quicksort in Python.", "Explain relativity.", "Redis guide with code.", "Write a haiku.", "JSON parser in Python."]

print("=== EP8 best (confirm) ===", flush=True) ts,_,_ = api(prompts[0], 512); print(f" C=1: {ts:.1f} tok/s", flush=True) for C in [32, 64, 128, 192, 256]: t0 = time.perf_counter() with concurrent.futures.ThreadPoolExecutor(min(C,256)) as pool: futs = [pool.submit(api, prompts[i%5], 2048, 900) for i in range(C)] results = [f.result() for f in concurrent.futures.as_completed(futs)] tw = time.perf_counter() - t0 total_toks = sum(r[2] for r in results) print(f" C={C:>3}: {total_toks/tw:>8.1f} tok/s wall={tw:.1f}s", flush=True) print("Done.", flush=True) PY Warmup... === EP8 best (confirm) === C=1: 65.2 tok/s C= 32: 624.1 tok/s wall=84.1s C= 64: 935.6 tok/s wall=107.8s C=128: 1416.9 tok/s wall=142.3s C=192: 1441.9 tok/s wall=212.2s C=256: 1492.5 tok/s wall=272.5s Done.

The output confirms that the EP8 configuration—expert parallelism across 8 GPUs with max_running_requests=256 and num_continuous_decode_steps=8—reproduces its previously observed performance. Single-request throughput is a solid 65.2 tok/s, and aggregate throughput scales to 1492.5 tok/s at high concurrency. The config is stable. The baseline is firm. The next experiment can begin.

The Context That Made This Message Necessary

To understand why this message exists, one must understand what happened immediately before it. The assistant had been systematically exploring parallelism strategies for the Kimi K2.6 model, a 384-expert MoE architecture with Multi-head Latent Attention (MLA) running in INT4 precision. Earlier benchmarking had established that expert parallelism (EP8) dramatically outperformed pure tensor parallelism (TP8) at low concurrency—65 tok/s versus 26 tok/s—by eliminating the expensive AllReduce operations across PCIe for every MoE layer. At high concurrency, EP8 reached approximately 1538 tok/s peak throughput.

The assistant then attempted to push further. In message 11529, it deployed an "EP8 tuned v2" configuration that added two modifications: increasing num_continuous_decode_steps from 8 to 16, and enabling --enable-eplb (expert load balancing). The EPLB feature attempts to dynamically rebalance expert routing to reduce the load imbalance inherent in MoE systems, where some GPUs may receive more tokens than others because the router selects only 8 of 384 experts per token.

The result was a disaster. As the assistant's reasoning in message 11532 documents: "EPLB hurt performance significantly—C=1 went from 65 to 7.7 tok/s in one test (huge variance), and peak throughput dropped from 1538 to 862 tok/s. The EPLB overhead (tracking expert distributions, rebalancing) is too expensive for this workload."

This failure is instructive. Expert load balancing is a theoretically sound idea—MoE systems suffer from routing imbalance, and rebalancing should help—but the implementation overhead in SGLang's EPLB was greater than the benefit. The tracking and rebalancing introduced synchronization points and CPU-side computation that destroyed the very throughput it was meant to improve. The assistant's reasoning correctly identified this: "Let me go back to the best config: EP8 with maxreq=256, continuous=8, no EPLB."

But simply reverting is not enough. The assistant needed to confirm that the revert actually restored the previous performance level. This is the scientific method applied to systems engineering: after a perturbation (the failed EPLB experiment), one must verify that the system returns to its prior state before proceeding with the next perturbation.

Why This Message Was Written: The Reasoning and Motivation

The comment at the top of the bash command says it all: "Quick confirm the best config reproduces, then try EP4." This message serves two purposes simultaneously.

First, it is a reproducibility check. The assistant had just stopped the EPLB-tuned service, deployed a new service unit (sglang-k26-ep8-best.service), waited for it to become ready (message 11533 shows a 150-second loading period), and now needs to verify that the service is actually serving requests at the expected performance level. This is not trivial—SGLang services can fail silently, load weights incorrectly, or suffer from Triton compilation cache issues that degrade performance on the first few requests. The warmup phase (two "Say OK" requests) addresses the compilation cache issue, but only the benchmark confirms the full configuration.

Second, it is a baseline measurement for the next experiment. The assistant's plan is to try EP4 (expert parallelism with 4 groups, where each group contains 2 GPUs doing tensor parallelism internally). To evaluate whether EP4 is better or worse than EP8, the assistant needs a reliable measurement of EP8's performance under the current conditions. The previous EP8 measurements (from messages 11527 and 11531) were taken under slightly different conditions—different prompt sets, different concurrency levels, and crucially, before the EPLB experiment had potentially disturbed the system state (e.g., GPU memory fragmentation, NCCL connection state, Triton cache state). A fresh baseline eliminates these confounds.

The choice of concurrency levels—32, 64, 128, 192, 256—reveals the assistant's understanding of the scaling curve. These are not arbitrary numbers. The assistant knows from previous sweeps that throughput scales roughly linearly up to C=128, then begins to plateau as the scheduler becomes saturated and PCIe bandwidth is fully utilized. Testing at C=32 and C=64 captures the linear regime; C=128 captures the knee of the curve; C=192 and C=256 capture the saturation regime. This allows the assistant to verify not just the peak throughput but the entire shape of the scaling relationship.

How Decisions Were Made Within This Message

Several design decisions are embedded in the benchmark script itself.

The choice of prompts: The assistant uses five prompts covering different domains (sorting algorithm, physics explanation, database guide, poetry, parsing). This diversity is intentional—it prevents the benchmark from overfitting to a single prompt pattern (e.g., very short outputs that inflate throughput numbers). The prompts are also deliberately short on the input side, minimizing prefill time and focusing the measurement on decode throughput.

The choice of max_tokens: Single-request throughput is measured at max_tokens=512, while concurrency sweeps use max_tokens=2048. This is a subtle but important distinction. At C=1, the assistant wants a quick measurement (512 tokens takes ~8 seconds at 65 tok/s), while at high concurrency, longer generations (2048 tokens) allow the system to reach steady-state scheduling behavior. Short generations at high concurrency would be dominated by prefill overhead and request startup latency, not decode throughput.

The timeout of 900 seconds: Each concurrency level runs for up to 15 minutes. This is generous but necessary—at C=256 with 2048 tokens per request, the total generation load is 256 × 2048 = 524,288 tokens, which at ~1500 tok/s takes about 350 seconds. The 900-second timeout provides a comfortable margin for cold-start effects and scheduler queueing delays.

The use of as_completed: The script uses concurrent.futures.as_completed to collect results as they finish, rather than waiting for all results in submission order. This is the correct approach for throughput measurement—it minimizes the impact of straggler requests on the aggregate timing. However, the wall time measurement starts before all requests are submitted (the ThreadPoolExecutor submits them sequentially in a loop), which introduces a small systematic error: the wall time includes the submission time of all requests. For C=256, this submission overhead is negligible (microseconds per request), but it's worth noting as a methodological detail.

Assumptions Made by the Assistant

Every benchmark rests on assumptions, and this one is no exception.

The assistant assumes that two warmup requests are sufficient to bring the system to steady state. For a service using Triton JIT compilation, the first request to each kernel shape triggers compilation, which can take seconds. Two "Say OK" requests (16 tokens each) may not exercise all kernel shapes that appear in the actual benchmark (e.g., different sequence lengths, different batch sizes). If the benchmark encounters a novel shape, the first request at that shape will include compilation latency, potentially depressing the measured throughput.

The assistant assumes that the service is in the same state as before the EPLB experiment. But GPU memory fragmentation, NCCL connection state, and CUDA graph caches may have been affected by the service restart. The 150-second loading period (message 11533) suggests a full model load, which should reset GPU memory state, but NCCL connections to peer GPUs may need to re-establish, and the first few collective operations may incur higher latency.

The assistant assumes that 5 prompts cycled across concurrent requests provide a representative workload. In production, a serving system would see a much wider variety of prompt lengths and generation lengths. The benchmark's uniformity (all requests generate 2048 tokens) may favor certain scheduler behaviors (e.g., batching strategies that work well for homogeneous workloads) that would not generalize to heterogeneous workloads.

The assistant assumes that the concurrent.futures approach accurately measures server throughput. In reality, the client-side measurement includes network latency, HTTP parsing overhead, and any queueing delay at the load balancer (if any). For a local network connection to a server on the same machine (via SSH tunnel or direct connection), these overheads are small, but they are not zero.

Input Knowledge Required to Understand This Message

A reader needs substantial background to fully grasp what this message is doing.

Model architecture knowledge: Understanding that Kimi K2.6 is a Mixture-of-Experts model with 384 experts, MLA attention, and INT4 quantization is essential. Without this, the significance of EP8 versus TP8—and why expert parallelism avoids AllReduce on MoE layers—is incomprehensible.

Parallelism strategy knowledge: The reader must understand the difference between tensor parallelism (splitting individual layers' weights across GPUs), expert parallelism (assigning different experts to different GPUs), and pipeline parallelism (splitting layers across GPUs). The tradeoffs—communication overhead, load imbalance, GPU utilization—are the entire subject of this benchmarking campaign.

Hardware knowledge: The RTX PRO 6000 Blackwell GPUs are connected via PCIe Gen5, not NVLink. This is the critical bottleneck: PCIe has much lower bandwidth and higher latency than NVLink, making AllReduce operations across 8 GPUs extremely expensive. This is why EP8 wins—it replaces AllReduce with point-to-point All-to-All communication, which is cheaper on PCIe.

SGLang configuration knowledge: Flags like --num-continuous-decode-steps, --max-running-requests, --ep-size, and --enable-eplb are SGLang-specific. The assistant's understanding of what each flag does—and how they interact—is the product of extensive experimentation.

Benchmark methodology knowledge: The distinction between single-request throughput (tok/s at C=1) and aggregate throughput (total tok/s across concurrent requests) is fundamental. The shape of the scaling curve (linear then saturating) reveals the system's bottleneck. The choice of concurrency levels and token counts reflects an understanding of where the bottleneck lies.

Output Knowledge Created by This Message

This message produces several pieces of knowledge that advance the optimization campaign.

Confirmation of reproducibility: The EP8 configuration delivers 65.2 tok/s at C=1, consistent with the previous measurement of ~65 tok/s. This confirms that the service deployment is reliable and the configuration is deterministic.

Updated scaling curve: The aggregate throughput numbers—624.1 at C=32, 935.6 at C=64, 1416.9 at C=128, 1441.9 at C=192, 1492.5 at C=256—show a clear saturation pattern. Throughput increases by ~300 tok/s between each step up to C=128, then only ~25 tok/s between C=128 and C=192, and ~50 tok/s between C=192 and C=256. This suggests the system is near its maximum throughput at C=128, with diminishing returns beyond.

Baseline for EP4 comparison: The assistant can now run the same benchmark with EP4 and compare directly. If EP4 achieves higher throughput at any concurrency level, it becomes the new best config. If not, the assistant knows EP8 is the winner.

Evidence of PCIe bottleneck: The saturation at ~1500 tok/s across 8 GPUs implies each GPU is doing approximately 187 tok/s. Given that each token requires 8 expert FFN evaluations (each 7168→2048→7168 in INT4) plus attention and shared expert computation, the throughput is likely limited by PCIe All-to-All bandwidth rather than compute capacity. This is a crucial insight for the next phase of optimization.

The Thinking Process Visible in This Message

The assistant's thinking is visible in the structure and content of the message, even though the message itself contains no explicit reasoning text (unlike message 11532 which has a "## Agent Reasoning" section).

The most telling artifact is the comment: "Quick confirm the best config reproduces, then try EP4." This reveals a two-phase experimental protocol that the assistant follows implicitly:

Phase 1 (this message): Establish a clean baseline by confirming the known-good configuration.
Phase 2 (next message): Apply the treatment (EP4) and measure the effect. This is the classic structure of a controlled experiment, applied to systems optimization. The assistant could have simply deployed EP4 directly and compared to the previous EP8 numbers from hours earlier. But it chose to re-measure EP8 first, demonstrating an understanding that system state drifts over time and that fair comparisons require contemporaneous measurements. The choice to measure at multiple concurrency levels—rather than just peak throughput—shows a multi-dimensional evaluation philosophy. The assistant cares not just about the maximum throughput number, but about the entire operating curve. A configuration that peaks higher but has worse latency at low concurrency might be worse for a mixed workload. By measuring the full curve, the assistant preserves the ability to make nuanced tradeoff decisions later. The warmup phase—two requests of 16 tokens each—shows an understanding of Triton JIT compilation behavior. The first request to any kernel shape triggers compilation, which can take seconds and would distort throughput measurements. By warming up with short requests, the assistant ensures that the common kernel shapes are compiled before the benchmark begins. The use of flush=True in print statements reveals an awareness of the buffered output problem in Python subprocesses. Without flushing, the benchmark output might be interleaved with bash output or lost if the process is interrupted. This is a small detail, but it shows careful engineering.

Broader Significance

This message, for all its apparent simplicity, embodies a methodological approach that distinguishes expert systems optimization from ad-hoc tinkering. The assistant treats the ML serving system as an experimental apparatus, applies the scientific method (hypothesis → experiment → measurement → analysis), and maintains rigorous controls.

The failed EPLB experiment that preceded this message is not a mistake to be hidden—it is a productive failure that generated knowledge. The assistant learned that EPLB's overhead exceeds its benefit for this workload, and this knowledge is incorporated into the system's design. The confirmation benchmark then ensures that the system returns to its known-good state before the next experiment.

This cycle—perturb, measure, revert, confirm, perturb again—is the engine of systematic optimization. Each cycle produces a data point on the performance landscape. Over many cycles, the assistant builds a map of that landscape, identifying ridges (good configurations) and valleys (bad ones). The message at index 11534 is one of those confirmation steps, but it is also a data point: EP8 at this moment in time, on this hardware, with this model, delivers 1492.5 tok/s at C=256. That number is now part of the map.

The next message in the conversation will likely deploy EP4 and measure its performance. The comparison will be clean because the baseline was just re-measured. And whether EP4 wins or loses, the knowledge will be incorporated, and the cycle will continue.