The 128 Concurrency Decision: Data-Driven Optimization in an EAGLE-3 Training Pipeline

"Now let me also bump the concurrency to 128 (matches the optimal throughput point from profiling):"

This single sentence, spoken by an AI assistant during an opencode coding session, is deceptively brief. On its surface, it describes a simple parameter change in a Python script. But within the context of a sprawling, multi-day effort to deploy and optimize a 1-trillion-parameter language model on eight NVIDIA Blackwell GPUs, this message represents something far more significant: the precise moment when accumulated empirical knowledge crystallizes into a concrete optimization decision.

The message is message [msg 2935] in a long conversation spanning dozens of rounds, multiple sub-agents, and countless tool calls. It is the third edit to the same file in a single round, following fixes to the reasoning tokenization logic and a change to the sample count target. The assistant is modifying 01b_generate_synthetic.py, a script that generates synthetic training data for an EAGLE-3 speculative decoding draft model by sending prompts to a running vLLM server and capturing the model's responses. The parameter being changed is the concurrency level — the number of simultaneous inference requests the script will send to the server.

The Context: A Pipeline Under Construction

To understand why this message matters, one must appreciate the scale of the project. The assistant is building a complete EAGLE-3 speculative decoding pipeline for Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts model with 384 routed experts and a unique MLA (Multi-head Latent Attention) architecture. The goal is to train a small 2.6-billion-parameter draft model that can predict the large model's outputs, enabling speculative decoding — a technique where the draft model generates candidate tokens that the large model verifies in parallel, potentially doubling or tripling throughput.

The pipeline has multiple stages: generate synthetic training data via inference, extract hidden states from the running model, build vocabulary mappings, train the draft model, and finally deploy it with speculative decoding. Each stage is a complex script with its own parameters, failure modes, and optimization opportunities.

The synthetic data generation script (01b_generate_synthetic.py) is the first production stage. It reads prompts from a dataset, sends them to the vLLM server running Kimi-K2.5, captures the model's responses (including the reasoning/thinking tokens that precede the final answer), and saves them for later processing. The script had already been fixed twice in this round: once to properly wrap reasoning content with thinking and response tokens (matching the model's actual token stream), and once to change the default sample count from 25,000 to 10,000 (the target for the local "hero run").

The Decision: Why C=128?

The concurrency parameter controls how many inference requests are sent to the vLLM server simultaneously. Too low, and the GPUs are underutilized — the server processes requests one at a time, leaving compute capacity idle. Too high, and requests queue up, causing latency spikes and potentially overwhelming the server's scheduler. The optimal concurrency is the point where throughput plateaus — where adding more parallel requests no longer increases the total tokens generated per second.

The assistant's reference to "the optimal throughput point from profiling" points back to earlier work documented in segments 18 and 19 of the conversation. In those segments, the assistant conducted a comprehensive benchmarking campaign, measuring Kimi-K2.5 INT4 throughput at various concurrency levels. The results were clear: peak throughput of approximately 1,536 tokens per second was achieved at a concurrency of 128. Below that, throughput scaled roughly linearly with concurrency as the GPUs were fed more work. Above that, throughput plateaued or even decreased as queueing overhead dominated.

By setting the concurrency to 128, the assistant is aligning the data generation script with the server's known performance characteristics. This is not a guess or a heuristic — it is a direct application of empirical measurement to a production parameter.

Input Knowledge: What the Assistant Needed to Know

This seemingly trivial edit required a substantial body of accumulated knowledge:

  1. The profiling results from the benchmarking campaign: The assistant had previously run extensive throughput tests at multiple concurrency levels (C=1, C=16, C=32, C=64, C=128, C=256) and identified C=128 as the saturation point. Without this data, any concurrency choice would be arbitrary.
  2. The hardware topology: The system has 8× NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe Gen5 with no NVLink. This PCIe interconnect is the dominant bottleneck (51.5% of decode time is AllReduce), which means throughput scaling is limited by communication, not compute. The optimal concurrency reflects this bottleneck.
  3. The server's capacity: The vLLM server was already running and serving requests. The assistant knew it could handle C=128 concurrent requests based on the earlier benchmarks.
  4. The script's architecture: The assistant knew that the script used asyncio with asyncio.gather() to dispatch concurrent requests, and that the concurrency parameter directly controlled the number of simultaneous API calls.
  5. The pipeline's time budget: The assistant had estimated the total pipeline time at 9-11 hours for 10K samples. Using C=128 was essential to meeting this estimate — lower concurrency would have extended the inference phase from ~3-5 hours to potentially 10+ hours.

Output Knowledge: What Changed

The edit itself is invisible in the message — no diff is shown, only the confirmation "Edit applied successfully." But the effect is clear: the --concurrency argument's default value in 01b_generate_synthetic.py was changed from whatever it was (likely a conservative default like 32 or 64) to 128.

This change has cascading effects:

Assumptions and Potential Pitfalls

The assistant makes several assumptions with this change:

  1. The profiling results are still valid: The assistant assumes that the throughput characteristics measured during benchmarking still hold. If the server's workload has changed (e.g., different prompt lengths, different sampling parameters), the optimal concurrency might differ.
  2. The server can handle sustained C=128 load: The benchmarks were likely run for short durations. Sustained high-concurrency inference for 3-5 hours could expose thermal throttling, memory fragmentation, or other degradation that short benchmarks miss.
  3. The script's concurrency mechanism is efficient: The assistant assumes that asyncio.gather() with 128 concurrent coroutines will not introduce its own overhead (e.g., Python's event loop becoming saturated with 128 concurrent HTTP connections).
  4. The network and API layer can keep up: 128 concurrent requests mean 128 simultaneous HTTP connections to the vLLM server. The server's HTTP worker pool, the container's network stack, and the local machine's ability to process responses all need to handle this load.
  5. The dataset has enough diversity: If many prompts produce very long or very short responses, the concurrency might need adjustment. The assistant assumes the average case holds. None of these assumptions are obviously wrong — they are grounded in empirical data and reasonable engineering judgment. But they are assumptions nonetheless, and they could be invalidated by real-world conditions.

The Thinking Process: A Study in Data-Driven Optimization

What makes this message remarkable is what it reveals about the assistant's thinking process, even though the reasoning is compressed into a single sentence.

The assistant is working through a prioritized list of edits to the script. First came the critical bug fix (reasoning tokenization), then the target adjustment (10K samples), and finally the performance optimization (C=128). This ordering shows clear priority: correctness first, scope second, speed third.

The reference to "optimal throughput point from profiling" demonstrates that the assistant maintains a mental model of the system's performance characteristics and applies them proactively. It is not just fixing bugs — it is optimizing the pipeline based on previously collected data. This is the hallmark of a systematic approach to engineering: measure, understand, then optimize.

The LSP errors displayed after the edit are dismissed as irrelevant — they are "could not be resolved" errors for torch, datasets, transformers, and openai, which are all installed on the container but not in the local development environment. The assistant correctly identifies these as false positives and proceeds.

The Broader Significance

This message, for all its brevity, captures a fundamental truth about large-scale ML engineering: the difference between a working pipeline and an efficient one is often a handful of well-chosen parameters, each grounded in empirical measurement. The assistant could have left the concurrency at its default and the pipeline would have worked. But it would have taken longer, consumed more wall-clock time, and potentially missed the window for the next stage.

The C=128 decision is one of dozens, perhaps hundreds, of similar decisions that collectively determine whether a complex pipeline runs in hours or days. It is the product of profiling, benchmarking, and systematic reasoning — the invisible work that separates a hack from a production system.

When the synthetic data generation eventually completes, producing 10,000 samples of Kimi-K2.5's actual reasoning outputs, the concurrency parameter will be a silent contributor to that success. No one will look at the script and say "ah, C=128, that was the right call." But the data will be there, generated in hours instead of days, because someone — or something — remembered the profiling results and applied them.