The Turning Point: Benchmarking EAGLE-3 Speculative Decoding Under Parallel Load

Introduction

In the long and winding journey to optimize inference on an 8× NVIDIA RTX PRO 6000 Blackwell GPU system running the massive Kimi-K2.5 INT4 model (a 1-trillion-parameter Mixture-of-Experts architecture), a single message marks the moment when months of infrastructure work finally paid off with actionable data. Message [msg 5417] is deceptively simple: it contains a single bash command that runs a parallel throughput benchmark against a running SGLang server, and the output shows throughput measurements at eight different concurrency levels. But this message is the culmination of a grueling optimization saga—a saga that involved upgrading the entire CUDA stack from version 12.8 to 13.0, patching SGLang's source code in four separate files to support Blackwell (SM120) GPUs, debugging ABI incompatibilities between PyTorch nightly builds and pre-compiled kernels, and transforming EAGLE-3 speculative decoding from a net-negative 54.1 tok/s (40% slower than baseline) into a net-positive 96.1 tok/s (3.8% faster).

The benchmark results in this message would go on to shape the entire next phase of the project, revealing a critical insight: EAGLE-3 speculation helps at low concurrency but becomes a liability when the GPUs are already saturated. This discovery directly led to the design of a dynamic speculation policy—a load-aware system that disables speculation when it hurts and enables it when it helps.

The Message in Full

The assistant executed the following command on the remote server running the EAGLE-3 + flashinfer fusion configuration:

ssh root@10.1.230.174 '~/ml-env/bin/python3 /tmp/benchmark_parallel.py --server-url http://localhost:30000 --concurrency 1 2 5 10 30 70 100 250 --max-tokens 512 2>&1'

The benchmark script, benchmark_parallel.py, had been written moments earlier in [msg 5415] and copied to the container via SCP. It was a custom tool designed specifically for this task—the existing benchmark_eagle3.py script only did serial (single-stream) benchmarking. The new script sent concurrent requests at varying concurrency levels, measuring total system throughput (tokens per second across all requests), per-request throughput, and latency percentiles.

The output revealed the following results (as documented in the subsequent context message [msg 5421]):

| Concurrency | Throughput (tok/s) | Per-Req (tok/s) | Avg Latency (s) | |---|---|---|---| | 1 | 77.5 | 77.7 | 6.6 | | 2 | 125.1 | 62.9 | 8.2 | | 5 | 183.2 | 37.3 | 13.8 | | 10 | 239.6 | 24.8 | 20.8 | | 30 | 299.5 | 10.4 | 49.6 | | 70 | 337.7 | 5.7 | 96.2 | | 100 | 338.8 | 4.1 | 134.3 | | 250 | 340.9 | 2.9 | 243.6 |

The data tells a clear story: throughput scales nearly linearly from C=1 (77.5 tok/s) to C=10 (239.6 tok/s), then begins to saturate. By C=70, the system hits a ceiling around 338-340 tok/s, and adding more concurrency yields almost no additional throughput. Meanwhile, per-request throughput collapses from 77.7 tok/s at C=1 to just 2.9 tok/s at C=250, and latency balloons from 6.6 seconds to over 4 minutes.

Why This Message Was Written: The Motivation and Context

To understand why this benchmark was run, we need to trace the conversation's trajectory. The project had been through several distinct phases:

  1. Environment setup: Installing NVIDIA drivers, CUDA toolkits, PyTorch, and resolving flash-attn build issues across multiple sessions.
  2. Baseline optimization: Getting the Kimi-K2.5 INT4 model running at 89.5 tok/s on CUDA 12.8, then improving to 92.6 tok/s after the CUDA 13 upgrade.
  3. EAGLE-3 debugging: The assistant spent extensive effort fixing hidden state wiring, adding delegation methods to the KimiK25 model class, and patching the EAGLE-3 worker to properly interface with the target model.
  4. The CUDA 13 breakthrough: Upgrading to CUDA 13.0.1 unblocked FlashInfer allreduce fusion and Torch symmetric memory on SM120 (Blackwell), which dramatically reduced the verify pass latency. This transformed EAGLE-3 from a 54.1 tok/s liability into a 96.1 tok/s asset.
  5. The user's question: In [msg 5413], the user simply asked: "Try benchmarking parallel inference C=2,5,10,30,70,100,250." This was a natural next step—after proving that EAGLE-3 speculation worked in the single-stream case, the user wanted to understand how it behaved under realistic multi-user load. The assistant's response in [msg 5414] immediately recognized the significance of this request. It read the existing benchmark script, found it was serial-only, and wrote a new parallel benchmark from scratch. The thinking was: "I need a parallel/concurrent benchmark that measures total throughput (output tok/s across all concurrent requests)." This was not a trivial script—it needed to manage concurrent HTTP requests, collect timing data, compute aggregate statistics, and handle error cases gracefully.

How Decisions Were Made

Several key decisions are embedded in this message:

Choice of concurrency levels: The assistant used the exact levels the user requested (1, 2, 5, 10, 30, 70, 100, 250), but also added C=1 to establish a baseline single-stream reference. This was a wise addition—it allowed direct comparison with the earlier single-stream benchmark results (which had shown 96.1 tok/s for EAGLE-3). The discrepancy between the C=1 result here (77.5 tok/s) and the earlier single-stream result (96.1 tok/s) is notable and deserves analysis: the earlier benchmark used max_tokens=2048 while this one used max_tokens=512, and the benchmark methodology differed (parallel framework vs sequential).

Choice of max_tokens=512: The assistant chose 512 tokens per request rather than the 2048 used in earlier benchmarks. This is a reasonable choice for a parallel benchmark—shorter requests complete faster, allowing more measurements in less time, and 512 tokens is a realistic generation length for many production use cases. However, it does mean the results are not directly comparable to the earlier 2048-token benchmarks.

Benchmark methodology: The script sent 30 requests per concurrency level, with 5 warmup requests. This is a reasonable methodology—30 samples provide enough data for stable averages, and the warmup ensures the server's CUDA graphs and memory pools are initialized before measurement begins. The script measured total throughput (aggregate tok/s across all concurrent requests), per-request throughput, and latency percentiles (avg, p50, p99).

Server configuration: The benchmark was run against the EAGLE-3 server with flashinfer allreduce fusion enabled, configured with --speculative-num-steps 2, --speculative-eagle-topk 4, --speculative-num-draft-tokens 16, and --mem-fraction-static 0.88. This was the culmination of all the optimization work—the assistant deliberately chose to benchmark the best-known configuration first.

Assumptions Made

The assistant made several assumptions in this message:

That the server would handle concurrent requests correctly: EAGLE-3 speculative decoding with flashinfer fusion had only been tested in single-stream mode. The assistant assumed the batching and scheduling logic would work correctly under concurrent load without deadlocks, race conditions, or memory corruption. This assumption proved correct—all 30 requests succeeded at every concurrency level ("ok: 30/30").

That 512 max_tokens is representative: The assistant assumed that results at 512 tokens would generalize to other generation lengths. This is a reasonable assumption for throughput scaling behavior, but the absolute numbers would differ for longer generations (which have more opportunity for speculation to help or hurt).

That 30 requests per concurrency level is sufficient: With 30 samples, the assistant assumed the averages would be stable. The tight p50/p99 latency ratios (e.g., 6.6/7.2 at C=1) suggest this was a good assumption—the system showed consistent behavior.

That the benchmark script itself wouldn't become a bottleneck: At high concurrency (C=250), the client machine sending 250 concurrent HTTP requests could itself become a bottleneck. The assistant implicitly assumed the client (running on the same machine as the server) had sufficient resources to manage these connections.

Mistakes and Incorrect Assumptions

The C=1 result doesn't match earlier single-stream benchmarks: The parallel benchmark shows 77.5 tok/s at C=1, while the earlier single-stream benchmark showed 96.1 tok/s (average over 10 runs). This ~20% discrepancy is significant and unexplained in the message. Possible causes include: different max_tokens (512 vs 2048), different benchmark methodology (parallel framework overhead vs sequential), or server state differences (the server had been running for some time and might have been in a different memory pool state). The assistant did not comment on this discrepancy in the message.

The benchmark doesn't compare against baseline (no speculation): The most critical comparison—EAGLE-3 vs baseline at each concurrency level—is absent from this message. The assistant ran only the EAGLE-3 configuration. The user's next message ([msg 5418]) explicitly asks about this: "Can we tell (or mod) sglang to disable speculation at certain concurrency? Probably need to bench w/o speculation to decide a threshold." The assistant acknowledges this gap in [msg 5419], adding a todo to benchmark the baseline.

No variance reporting: The output shows only averages, not standard deviations or min/max ranges. Given the earlier single-stream benchmarks showed significant variance (87.5 to 103.5 tok/s range), it would have been valuable to see the spread at each concurrency level. The p50/p99 latency numbers provide some insight into tail latency, but throughput variance is unreported.

Input Knowledge Required

To fully understand this message, one needs:

Understanding of speculative decoding: EAGLE-3 is a speculative decoding algorithm where a smaller "draft" model proposes multiple tokens, and the large "target" model verifies them in parallel. The key tradeoff is between the speedup from generating multiple tokens per forward pass and the overhead of running both models. The verify step is communication-intensive because it requires all-reduce operations across all 8 GPUs for each of the 61 layers.

Knowledge of the CUDA 13 upgrade context: The message builds directly on the CUDA 13 upgrade that unblocked FlashInfer allreduce fusion. Without this context, the benchmark results would seem disconnected from the months of effort that preceded them.

Understanding of concurrency scaling in LLM serving: The throughput curve—near-linear at low concurrency, saturating at high concurrency—is characteristic of GPU-bound systems. At low concurrency, the GPU has idle capacity that can absorb additional requests. At high concurrency, the GPU is fully utilized and throughput plateaus while latency continues to increase.

Familiarity with SGLang architecture: The benchmark targets SGLang's HTTP API at port 30000, which is the v0.5.9 default. Understanding that SGLang uses continuous batching and CUDA graph optimization helps interpret why the server behaves differently at different concurrency levels.

Output Knowledge Created

This message produced several valuable pieces of knowledge:

The saturation point for EAGLE-3 speculation: The system saturates at approximately 340 tok/s, reached around C=70. This is a hard upper bound determined by the GPU compute capacity and the all-reduce communication overhead across 8 PCIe-connected GPUs.

The crossover hypothesis: The data strongly suggests that EAGLE-3 speculation, which helps at low concurrency, would become a net negative at high concurrency where the GPU is already saturated. The speculation overhead (running the draft model, managing additional memory, extra scheduling complexity) consumes resources that could otherwise be used for additional batch processing. This hypothesis directly motivated the next phase of work: benchmarking without speculation to find the exact crossover point, then implementing dynamic speculation disable.

The latency explosion at high concurrency: At C=250, average latency reaches 243.6 seconds (over 4 minutes) for a 512-token generation. This is a critical observation for production deployment—it means the system cannot handle more than ~70 concurrent requests without unacceptable latency, regardless of speculation strategy.

The diminishing returns of concurrency: Adding more concurrency beyond C=10 yields rapidly diminishing throughput gains. The jump from C=10 to C=30 adds only 60 tok/s (25% gain for 3× more concurrency), and from C=30 to C=70 adds only 38 tok/s (13% gain for 2.3× more concurrency). This has direct implications for capacity planning and pricing.

The Thinking Process Visible in the Message

While the message itself is just a bash command and its output, the thinking process is visible in the surrounding context. In [msg 5414], the assistant reads the existing benchmark script, evaluates it, and decides: "This is a serial benchmark. I need a parallel/concurrent benchmark." This shows a clear understanding of the task requirements and a willingness to build new tools rather than repurpose inadequate ones.

The assistant then writes benchmark_parallel.py ([msg 5415]) with careful attention to methodology: warmup requests, multiple runs per concurrency level, comprehensive metrics (throughput, per-request throughput, latency percentiles, error counts). The script is designed to be robust and informative.

The choice to benchmark only the EAGLE-3 configuration (not baseline) reflects the assistant's prioritization: the user asked for EAGLE-3 benchmarks, and the assistant delivered them. The baseline comparison would come later, in response to the user's follow-up question.

The assistant also shows awareness of the server state: it runs the benchmark against the existing EAGLE-3 server (started in [msg 5401]) without restarting it. This assumes the server is in a steady state and has been running long enough for CUDA graph caches to be warm.

The Broader Significance

This message represents the transition from "does it work?" to "how well does it work under realistic conditions?" The earlier benchmarks (single-stream, 2048 tokens) were necessary to validate that the EAGLE-3 speculative decoding pipeline was functioning correctly after all the patches and upgrades. But production inference serving is never single-stream—it's a constant stream of requests from multiple users, and the system's behavior under concurrent load is what ultimately determines its usefulness.

The data from this benchmark directly informed the next major feature: dynamic speculation disable based on server load. The user's immediate follow-up ([msg 5418]) shows they grasped the implication instantly: "Can we tell (or mod) sglang to disable speculation at certain concurrency?" The assistant's response ([msg 5419]) formalizes this into a todo list: benchmark baseline at the same concurrency levels, find the crossover point, investigate SGLang's code for dynamic speculation mechanisms, and test reduced speculation configurations.

In essence, this message is the pivot point. Before it, the project was about making EAGLE-3 work at all. After it, the project became about deploying EAGLE-3 intelligently—using it where it helps, disabling it where it hurts, and building the infrastructure to make that decision automatically.

Conclusion

Message [msg 5417] appears, on the surface, to be a routine benchmark execution. But in the context of the broader conversation, it is a decisive moment. It transforms months of infrastructure work into actionable performance data, reveals the critical saturation behavior that would define the next phase of the project, and demonstrates the assistant's ability to build custom tools for specific measurement tasks. The parallel throughput benchmark is not just a test—it is the bridge between "making it work" and "making it work well in production." The data it produced would go on to shape the architecture of a dynamic speculation system, proving that sometimes the most valuable output of a single message is not the code it runs but the questions it answers—and the new questions it raises.