The Baseline: Measuring Single-Request Throughput in a Pipeline-Parallel LLM Deployment

Introduction

In the complex world of large language model (LLM) serving, few measurements are as fundamental—or as deceptively simple—as the single-request throughput benchmark. When an AI assistant responds to a user's request to "rerun C=1 test" after aborting a concurrency sweep, the resulting message ([msg 11505]) appears at first glance to be a routine execution: fire off a few prompts, collect some numbers, report back. But this message is anything but routine. It is the capstone of an intensive debugging session that uncovered a critical configuration bottleneck in SGLang's pipeline parallelism implementation, and it represents a deliberate act of measurement discipline—establishing a clean baseline before the next phase of optimization.

This article examines message [msg 11505] in depth: why it was written, what it reveals about the state of the system, the assumptions baked into its methodology, and how it fits into the broader narrative of deploying the Kimi K2.6 model with speculative decoding across multiple GPU platforms.

The Message

The assistant's response is a single tool call—a bash command that executes an inline Python script against the PP8 (pipeline parallelism with 8 stages) service running on the CT200 machine at 10.1.2.200:30001:

python3 -u - <<'PY'
import json, time, urllib.request

CT200 = "10.1.2.200"
PORT = 30001
MODEL = "/root/models/Kimi-K2.6"

def api(prompt, max_tokens=512, timeout=300):
    payload = json.dumps({"model": MODEL,
        "messages": [{"role":"user","content":prompt}],
        "temperature": 0, "max_tokens": max_tokens})
    req = urllib.request.Request(f"http://{CT200}:{PORT}/v1/chat/completions",
                                 data=payload.encode(), headers={"Content-Type":"application/json"})
    t0 = time.perf_counter()
    with urllib.request.urlopen(req, timeout=timeout) as r:
        data = json.load(r)
    wall = time.perf_counter() - t0
    ct = data.get("usage", {}).get("completion_tokens", 0)
    return ct/wall if wall > 0 else 0, wall, ct

print("Warmup...", flush=True)
api("Say OK", 16, 120)

prompts = [
    "Write a Python quicksort implementation with detailed comments.",
    "Explain the theory of relativity in 3 sentences.",
    "What is 2+2? Answer with just the number.",
    "Write a haiku about programming.",
    "Write a Python JSON parser that handles nested objects. Return only code.",
]

print("\n=== K2.6 PP8 C=1 (max=512) ===", flush=True)
for p in prompts:
    ts, w, ct = api(p, 512)
    print(f"  {ts:.1f} tok/s | {ct} tok in {w:.1f}s", flush=True)

print("\nDone.", flush=True)
PY

The output is clean and consistent:

Warmup...

=== K2.6 PP8 C=1 (max=512) ===
  37.0 tok/s | 512 tok in 13.8s
  36.9 tok/s | 482 tok in 13.1s
  35.5 tok/s | 47 tok in 1.3s
  36.9 tok/s | 512 tok in 13.9s
  36.8 tok/s | 512 tok in 13.9s

Done.

Five prompts, five measurements, all clustering tightly around 37 tokens per second. The consistency is remarkable—four of five results fall within 0.2 tok/s of each other, and the one outlier (35.5 tok/s) corresponds to the shortest generation (47 tokens), where startup overhead dominates.

Why This Message Was Written: The Debugging Backstory

To understand why the assistant is running a C=1 benchmark at this precise moment, we must rewind through the preceding messages. The context reveals a multi-hour debugging saga focused on pipeline parallelism performance.

Earlier in the session ([msg 11495]), the assistant had been investigating why PP8 was underperforming on the CT200 machine—an 8× RTX PRO 6000 Blackwell system connected via PCIe. The initial PP8 configuration was yielding only ~248 tok/s at C=32, compared to TP8's ~577 tok/s. This was puzzling because pipeline parallelism should theoretically excel at high concurrency by overlapping computation across stages.

The assistant's reasoning in [msg 11495] is a masterclass in diagnostic thinking. It examines the logs and finds a critical clue: despite 32 concurrent API requests, the scheduler was showing only #running-req: 8 per stage. All eight PP stages were processing the same eight requests in lockstep, rather than being staggered with different micro-batches at different stages. This meant the pipeline wasn't actually pipelining—it was behaving like data parallelism with a serial bottleneck.

The breakthrough came in [msg 11499] when the assistant traced the issue to SGLang's auto-computation of pp_max_micro_batch_size. The relevant source code was:

pp_max_micro_batch_size = max(max_running_requests // pp_size, 1)

With max_running_requests=64 and pp_size=8, this yielded pp_max_micro_batch_size = 8. Each micro-batch could hold at most 8 requests. Even though there were 16 micro-batch slots (from pp_async_batch_depth=8 and pp_size=8), the per-micro-batch limit of 8 requests meant the pipeline stages were never fully utilized. The assistant fixed this by explicitly setting --pp-max-micro-batch-size 64 in the service configuration.

After this fix, the assistant ran a concurrency sweep ([msg 11501]) showing improved scaling: C=32 achieved 250.9 tok/s, C=48 reached 316.2 tok/s (though the C=64 test was still running when the timeout hit). Then in [msg 11503], the assistant pushed further with a shorter max_tokens=2048 to get faster results, showing C=64 at 396 tok/s before the user aborted the run.

The user's instruction—"kill that, but rerun C=1 test"—is a request for a clean baseline measurement. After all the debugging and reconfiguration, the user wants to know: what is the fundamental single-request throughput of this system? This is the message that answers that question.

Methodology and Assumptions

The benchmark script in [msg 11505] embodies several deliberate methodological choices that reveal the assistant's understanding of reliable measurement:

Temperature=0: By setting temperature to zero, the assistant ensures deterministic output. This eliminates variance from sampling, making the benchmark reproducible. It also means the model generates greedily, which typically produces shorter, more predictable sequences.

max_tokens=512: This is a deliberate choice for quick turnaround. Earlier benchmarks used 4096 or 2048 tokens, but for a C=1 baseline, 512 tokens is sufficient to measure steady-state throughput without waiting minutes per request. The results confirm this: each long generation completes in ~13-14 seconds.

Warmup request: The script sends a trivial "Say OK" request before the real benchmarks. This is crucial for LLM serving because the first request often incurs cold-start overhead—CUDA graph capture, Triton kernel compilation, and KV cache initialization. By warming up, the assistant ensures the measured results reflect steady-state performance.

Prompt diversity: The five prompts span different domains (coding, physics, math, poetry) and response lengths. The "What is 2+2?" prompt produces only 47 tokens, testing the overhead floor, while the coding prompts generate the full 512 tokens, testing sustained throughput.

Assumption of service readiness: The script does not check whether the service is healthy before running—it simply connects and sends requests. This assumes the service is already running and properly configured from the previous systemctl start command. If the service had crashed or was still loading, the script would have failed with a connection error.

Assumption of representative workload: The assistant assumes that 512-token generations with these specific prompts are representative of the model's single-request performance. In reality, throughput can vary with prompt length, response length, and the specific computational paths taken through the model. The tight clustering of results (36.8-37.0 tok/s for long generations) suggests this assumption is reasonable for this model and hardware.

What the Results Reveal

The measured throughput of ~37 tok/s at C=1 is a significant data point in the optimization journey. Comparing with earlier results:

• TP8 (tensor parallelism, 8 GPUs): Earlier in the session, TP8 achieved ~26.3 tok/s at C=1. PP8's 37 tok/s represents a 40% improvement over TP8 for single-request latency.
• Earlier PP8 (before micro-batch fix): The initial PP8 configuration achieved ~34.3 tok/s at C=1. The fix to pp_max_micro_batch_size improved single-request throughput by about 8%, suggesting the micro-batch size affects not just concurrency scaling but also per-request efficiency—possibly by enabling better CUDA graph utilization or reducing scheduling overhead. The 37 tok/s figure also serves as the denominator for speedup calculations when speculative decoding (DFlash/DDTree) is later deployed. In subsequent chunks (Chunk 2 of this segment), the assistant deploys DFlash speculative decoding and achieves up to 2.15× speedup over the autoregressive baseline. The C=1 baseline is the reference point for that calculation.

Input Knowledge and Output Knowledge

Input knowledge required to understand this message includes:

• The architecture of SGLang's pipeline parallelism (PP splits the model across GPUs, each stage processes a micro-batch in sequence)
• The concept of pp_max_micro_batch_size and how it limits per-micro-batch capacity
• The history of the debugging session: the auto-computation bug, the fix, and the concurrency sweep that preceded this message
• The hardware configuration: 8× RTX PRO 6000 Blackwell GPUs connected via PCIe on the CT200 machine
• The model: Kimi K2.6, a Mixture-of-Experts architecture with ~260B parameters (or similar scale) Output knowledge created by this message includes:
• A confirmed single-request throughput baseline of ~37 tok/s for PP8 with the micro-batch fix
• Evidence that the fix did not regress single-request performance (it actually improved slightly from 34.3 tok/s)
• A reproducible benchmark script that can be used for future comparisons
• Confidence that the system is stable and properly configured for the next phase of optimization

The Thinking Process

While the message itself contains no explicit reasoning block—it is a direct tool call responding to a user request—the thinking process is visible in the choices made. The assistant could have simply run a single prompt and reported one number. Instead, it constructed a mini-benchmark with five diverse prompts, a warmup, and careful parameter selection.

The choice of temperature=0 is particularly telling. In earlier benchmarks ([msg 11501]), the assistant used temperature=0.6, top_p=0.95 for more realistic sampling. Switching to greedy decoding for the C=1 test suggests the assistant prioritizes reproducibility and clean measurement over realism. This is the right call for a baseline: you want to measure the system's capability, not the stochastic variation of sampling.

The prompt selection also reveals deliberate thinking. The "What is 2+2?" prompt serves as a minimum-latency probe—it completes in 1.3 seconds and produces only 47 tokens, revealing the overhead of prompt processing, KV cache initialization, and the first few decode steps. The longer prompts measure sustained throughput. The spread between 35.5 tok/s (short) and 37.0 tok/s (long) quantifies the overhead: about 4% of throughput is lost to non-decode overhead in short generations.

Broader Significance

This message sits at a pivot point in the session. The preceding messages were diagnostic—finding the bottleneck, fixing it, and exploring concurrency scaling. This message establishes the new baseline. The subsequent messages (Chunk 2 of Segment 64) will deploy speculative decoding, benchmark across NVLink-connected B300 GPUs, and produce the comprehensive DDTREE_FINDINGS_REPORT.md.

The 37 tok/s baseline is the "before" measurement. When DFlash speculative decoding later achieves 86 tok/s at C=1 on the same hardware, the speedup is calculated against this 37 tok/s autoregressive baseline. Without this message, that speedup would be uncertain—was it 86 vs the old 34.3, or 86 vs the fixed 37.0? The assistant's discipline in re-establishing the baseline after every configuration change ensures that all subsequent comparisons are valid.

In the broader narrative of the opencode session, this message exemplifies a key engineering virtue: measure before optimizing. The assistant could have jumped straight from fixing the micro-batch bug to deploying speculative decoding. Instead, it took the time to re-measure the baseline, ensuring that future improvements would have a solid reference point. This discipline is what separates rigorous engineering from guesswork.

Conclusion

Message [msg 11505] is a seemingly simple benchmark that carries the weight of an entire debugging saga. It answers the user's request for a clean C=1 measurement after a critical configuration fix, establishes a reproducible baseline of ~37 tok/s for PP8 on 8× RTX PRO 6000 GPUs, and demonstrates the assistant's commitment to disciplined measurement methodology. The tight clustering of results—five prompts yielding nearly identical throughput—confirms that the system is stable, properly configured, and ready for the next phase of optimization. In the journey from diagnosing pipeline bubbles to deploying speculative decoding, this message is the calm before the storm: the moment when the engineer checks their instruments, confirms they're calibrated, and prepares for the next experiment.