When Optimization Backfires: A Case Study in Benchmarking SGLang's EPLB and Continuous Decode Steps

The Message

The following bash command was executed by the assistant to benchmark an updated SGLang deployment of the Kimi K2.6 model:

python3 -u - <<'PY'
import json, time, urllib.request, concurrent.futures

CT200 = "10.1.2.200"
PORT = 30001
MODEL = "/root/models/Kimi-K2.6"

def api(prompt, max_tokens=512, timeout=600):
    payload = json.dumps({"model": MODEL,
        "messages": [{"role":"user","content":prompt}],
        "temperature": 0, "max_tokens": max_tokens})
    req = urllib.request.Request(f"http://{CT200}:{PORT}/v1/chat/completions",
                                 data=payload.encode(), headers={"Content-Type":"application/json"})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        data = json.load(r)
    wall = time.perf_counter() - t0
    ct = data.get("usage", {}).get("completion_tokens", 0)
    return ct/wall if wall > 0 else 0, wall, ct

print("Warmup...", flush=True)
api("Say OK", 16, 120); api("Say OK", 16, 120)

prompts = ["Write a Python quicksort implementation with detailed comments.",
    "Explain the theory of relativity in 3 sentences.",
    "What is 2+2? Answer with just the number.",
    "Write a haiku about programming.",
    "Write a Python JSON parser that handles nested objects."]

print("\n=== EP8 v2 (continuous=16, eplb) ===", flush=True)
print("C=1:", flush=True)
for p in prompts[:3]:
    ts, w, ct = api(p, 512)
    print(f"  {ts:.1f} tok/s | {ct} tok/{w:.1f}s", flush=True)

print("\nSweep:", flush=True)
for C in [1, 8, 32, 64, 96, 128, 192, 256, 384]:
    try:
        t0 = time.perf_counter()
        with concurrent.futures.ThreadPoolExecutor(min(C, 256)) as pool:
            futs = [pool.submit(api, prompts[i%5], 2048, 900) for i in range(C)]
            done, notdone = concurrent.futures.wait(futs, timeout=900)
            results = [f.result() for f in done]
        tw = time.perf_counter() - t0
        total_toks = sum(r[2] for r in results)
        agg = total_toks / tw
        avg_tok = total_toks // max(len(results),1)
        print(f"  C={C:>3}: agg={agg:>8.1f} tok/s  wall={tw:>6.1f}s  avg={avg_tok}tok  ok={len(results)} fail={len(notdone)}", flush=True)
        if len(notdone) > 0: break
    except Exception as e:
        print(f"  C={C:>3}: ERROR - {e}", flush=True)
        break
print("\nDone.", flush=True)
PY

The output it produced told a stark story:

=== EP8 v2 (continuous=16, eplb) ===
C=1:
  63.4 tok/s | 512 tok/8.1s
  7.7 tok/s | 512 tok/66.6s
  56.7 tok/s | 40 tok/0.7s

Sweep:
  C=  1: agg=    13.3 tok/s  wall= 152.0s  avg=2022tok  ok=1 fail=0
  C=  8: agg=    46.7 tok/s  wall= 166.2s  avg=970tok  ok=8 fail=0
  C= 32: agg=   204.1 tok/s  wall= 196.0s  avg=1250tok  ok=32 fail=0
  C= 64: agg=   356.6 tok/s  wall= 219.3s  avg=1222tok  ok=64 fail=0
  C= 96: agg=   513.5 tok/s  wall= 235.8s  avg=1261tok  ok=96 fail=0
  C=128: agg= ...

The output is truncated—the benchmark was still running when the message was captured—but the pattern was already unmistakable: something had gone catastrophically wrong. The second single-request test produced a mere 7.7 tokens per second, taking 66.6 seconds to generate 512 tokens. The aggregate throughput at C=1 had collapsed from 61.3 tok/s in the previous run to 13.3 tok/s. This was not a marginal regression; it was a 4.6× slowdown.## Context: The Road to EP8 v2

To understand why this message matters, we need to trace the chain of reasoning that led to it. The assistant had been systematically benchmarking parallelism strategies for the Kimi K2.6 model on an 8× RTX PRO 6000 Blackwell system connected via PCIe. Earlier in the session ([msg 11523]), the assistant had theorized about optimal configurations, analyzing the tradeoffs between tensor parallelism (TP), expert parallelism (EP), and pipeline parallelism (PP) on PCIe-bound hardware.

The key insight was that EP8—where each of the 8 GPUs holds 48 complete experts locally—eliminated the expensive AllReduce communication that pure TP required for every MoE layer. At C=1 (single concurrent request), EP8 achieved 65 tok/s versus TP8's 26 tok/s, a 2.5× improvement. The initial "EP8 tuned" run ([msg 11527]) with max-running-requests=256 and num-continuous-decode-steps=8 had shown promising results: 65 tok/s at C=1 and scaling to 345 tok/s at C=16.

The user then asked the assistant to try additional optimizations ([msg 11524]). The assistant investigated DeepEP (an optimized All-to-All backend for MoE) but found it wasn't installed ([msg 11528]). As an alternative, the assistant turned to two other levers:

  1. EPLB (Expert Load Balancing): An SGLang feature that attempts to balance the routing of tokens across expert GPUs, reducing the load imbalance that can plague expert-parallel systems. In EP8, each token activates 8 of 384 experts, and those 8 experts are distributed across 8 GPUs. If routing is skewed, some GPUs get more work than others, creating a bottleneck.
  2. Increased continuous decode steps: The assistant doubled num-continuous-decode-steps from 8 to 16. This parameter controls how many decode steps the engine processes in a single batch before yielding control back to the scheduler. Higher values reduce Python overhead and scheduler latency by amortizing it across more tokens, but they also increase the risk of stragglers—if one sequence finishes early, the others must wait. The assistant deployed this "EP8 v2" configuration ([msg 11529]) with --num-continuous-decode-steps 16 --enable-eplb, restarted the service, waited for it to become ready ([msg 11530]), and then ran the benchmark that is the subject of this article.

The Benchmark Harness

The Python script embedded in the bash command is a carefully constructed benchmarking tool. It defines an api() function that sends a chat completion request to the SGLang server and measures throughput. The function uses urllib.request (the standard library HTTP client) to POST to the OpenAI-compatible endpoint, parses the JSON response, and extracts the completion_tokens count from the usage metadata. It returns tokens per second, wall time, and token count.

The script performs two warmup requests ("Say OK" with 16 max tokens) to ensure the server has completed any lazy initialization—Triton kernel compilation, CUDA graph captures, or memory allocation that might distort the first real measurement.

The benchmark has two phases. First, it sends three prompts individually at C=1 with max_tokens=512 to measure single-request latency and throughput. Then it runs a concurrency sweep across increasing concurrency levels (C=1, 8, 32, 64, 96, 128, 192, 256, 384) with max_tokens=2048, using a ThreadPoolExecutor to issue requests in parallel. The sweep measures aggregate throughput (total tokens generated across all requests divided by total wall time), which is the key metric for production serving.

Notably, the benchmark uses only 3 prompts for the single-request phase (instead of 5 in the previous run) and a slightly different concurrency sweep list (starting at 1, skipping 4 and 16). This is a minor inconsistency that slightly reduces the comparability of results, but the overall methodology is sound.

The Results: A Dramatic Regression

The output reveals a catastrophic performance collapse. The second single-request test produced 7.7 tok/s—a 10× degradation from the previous run's 65 tok/s. The aggregate throughput at C=1 fell to 13.3 tok/s from 61.3 tok/s. Even at higher concurrency, the numbers were far below expectations: at C=96, the EP8 v2 managed 513.5 tok/s, whereas the previous EP8 tuned run had already reached 345 tok/s at just C=16.

The most telling data point is the second single-request test: 512 tokens in 66.6 seconds. This is not a normal slowdown. Something fundamentally broke between the two runs. The 7.7 tok/s figure is closer to what one might expect from a single GPU running the full model without any parallelism—or from a configuration where the server is thrashing, swapping, or hitting a pathological scheduling behavior.

The first single-request test (63.4 tok/s) and the third (56.7 tok/s) are closer to the previous run's 65 tok/s, but the aggregate C=1 result of 13.3 tok/s suggests that the single long-running request dominated the measurement. The aggregate throughput at C=1 is computed by dividing total tokens (2022) by total wall time (152.0s). Since the benchmark submitted only one request at C=1, the 152-second wall time is dominated by the slow request.

What Went Wrong? Diagnosing the Regression

The assistant did not have the opportunity to diagnose the regression within this message—the output is truncated, and the next message in the conversation would contain the full results and the assistant's analysis. But we can reason about the likely causes.

Hypothesis 1: EPLB Overhead

Expert Load Balancing (EPLB) is designed to redistribute tokens across expert GPUs to balance the workload. However, on PCIe-connected GPUs, the balancing itself requires communication. If EPLB introduces additional synchronization points or data movement, it could degrade throughput—especially at low concurrency where the overhead is not amortized across many tokens.

The fact that the first request (63.4 tok/s) was reasonable while the second (7.7 tok/s) was abysmal suggests something non-deterministic. EPLB might have triggered a rebalancing event during the second request, causing a stall while expert assignments were recomputed and communicated.

Hypothesis 2: Continuous Decode Step Interaction

Doubling num-continuous-decode-steps from 8 to 16 could interact badly with EPLB or with the scheduler. If the engine commits to 16 decode steps but EPLB needs to rebalance after each step, the continuous decode loop might be repeatedly interrupted, negating the benefit of batching. Alternatively, if one request in the batch finishes early (e.g., the "What is 2+2?" prompt which naturally generates very few tokens), the remaining requests must wait for the full 16 steps to complete, wasting GPU cycles.

Hypothesis 3: CUDA Graph Invalidation

SGLang uses CUDA graphs to accelerate the decode path by capturing a sequence of GPU operations and replaying them with minimal CPU overhead. However, CUDA graphs are fragile—they must be recaptured if the model configuration changes. If EPLB changes the expert routing pattern, it might invalidate the captured CUDA graph, forcing a costly recapture during the request. The 66-second request could be explained by a graph recapture event that serializes kernel launches.

Hypothesis 4: Memory Pressure

The --mem-fraction-static 0.88 setting reserves 88% of GPU memory for the model and KV cache. With max-running-requests=256 and context-length=32768, the KV cache could be large. If EPLB requires additional memory for its routing tables or if the continuous decode loop holds more intermediate state, the server might be hitting memory limits and swapping to host memory.

The Thinking Process Visible in the Message

The message itself is a tool call—a bash command that executes a Python script. The assistant's reasoning is not directly visible in the message text because the assistant does not include an explicit reasoning block in this particular message. However, the structure of the benchmark reveals the assistant's thinking:

  1. The assistant chose to benchmark rather than assume the new config would be better. This is a hallmark of rigorous engineering: when making changes to a system, measure the impact rather than trusting intuition. The previous run had shown strong results, and the assistant could have declared victory. Instead, it ran a controlled experiment.
  2. The benchmark design reflects lessons from previous runs. The warmup phase, the use of multiple prompts to avoid caching effects, the concurrency sweep, and the timeout handling all indicate accumulated experience with benchmarking this particular system.
  3. The assistant used the same benchmark structure as the previous run ([msg 11527]), enabling direct comparison. This consistency is crucial for scientific measurement.
  4. The choice of concurrency levels shows an expectation of improvement. The sweep goes up to C=384, higher than the previous run's C=512, suggesting the assistant believed the new configuration might sustain higher throughput before saturating.

Assumptions Made by the Assistant

Several assumptions are embedded in this message:

  1. EPLB would improve throughput, or at least not harm it. The assistant enabled EPLB without testing it in isolation from the continuous decode step change. This confounded the two variables—if the result is bad, we don't know which change caused it.
  2. The service restart would cleanly apply the new configuration. The assistant stopped the old service, wrote a new systemd unit, and started it. But systemd service files can have subtle issues—environment variables not being picked up, working directory problems, or file descriptor leaks from the previous process.
  3. The warmup requests were sufficient to reach steady state. Two "Say OK" requests with 16 tokens each may not trigger all the kernel compilation paths needed for the full 2048-token generation. If CUDA graph capture happens lazily during the first real request, that request will be anomalously slow.
  4. The benchmark script correctly handles all edge cases. The api() function uses urllib.request.urlopen which is a blocking call. In the ThreadPoolExecutor, blocking I/O calls are fine, but the function doesn't handle HTTP errors, connection resets, or JSON parsing errors gracefully. If the server returns an error response, the script would raise an exception and potentially abort the entire sweep.
  5. The network path is not a bottleneck. The benchmark runs from a client machine (not the CT200 server itself) and sends requests over the network. At high concurrency, the network could become a bottleneck, especially if many requests are in flight simultaneously. The script doesn't measure network latency separately.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. SGLang architecture knowledge: Understanding of tensor parallelism, expert parallelism, continuous decode steps, and EPLB. The message assumes familiarity with these concepts and their configuration flags.
  2. Kimi K2.6 model architecture: Knowledge that this is a Mixture-of-Experts model with 384 experts, using MLA (Multi-head Latent Attention) with compressed KV projections, and INT4 quantized weights via Marlin kernels.
  3. Hardware constraints: Understanding that 8× RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 have limited inter-GPU bandwidth compared to NVLink, making communication topology a critical factor.
  4. Benchmarking methodology: Familiarity with throughput measurement, concurrency sweeps, warmup phases, and the distinction between single-request latency and aggregate throughput.
  5. Python standard library: Knowledge of urllib.request, concurrent.futures, json, and time.perf_counter for understanding the benchmark code.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. Empirical evidence that EPLB + continuous=16 degrades performance on this hardware. Whether the cause is EPLB overhead, continuous step interaction, or both, the 4.6× regression at C=1 is a clear signal that the combination is harmful.
  2. A baseline for comparison. Even though the v2 config performed worse, the benchmark provides a quantitative measurement that can be compared to the v1 run. This enables the assistant to make data-driven decisions about which optimizations to pursue.
  3. A reproducible benchmark methodology. The Python script is self-contained and can be re-run against any SGLang server, providing a template for future benchmarking.
  4. A diagnostic signal in the variance. The fact that one single-request test was 10× slower than the others is itself useful information—it suggests a non-deterministic performance pathology that warrants investigation.

Conclusion

This message captures a pivotal moment in a larger optimization journey. The assistant, having achieved strong results with EP8 tuned, attempted to push further with EPLB and increased continuous decode steps. The benchmark revealed that these changes backfired dramatically—at least in combination. The 7.7 tok/s result is a stark reminder that optimization is not monotonic: adding features and tuning parameters does not always improve performance, and the interaction between seemingly independent optimizations can produce surprising regressions.

The true value of this message lies not in the disappointing numbers themselves, but in the disciplined engineering practice it represents. The assistant could have assumed the new configuration was better and moved on. Instead, it measured. It built a benchmark, ran it, and captured the results. This creates a feedback loop that enables the next round of diagnosis and improvement. In the subsequent messages of the conversation, the assistant would analyze these results, identify the root cause, and iterate toward a better configuration.

For the reader, this message is a case study in the importance of benchmarking every change, the danger of confounded variables, and the reality that inference optimization at scale is an empirical science—not a matter of intuition.