Three Challenges, One Pipeline: Reasoning Capture, Throughput Optimization, and Pragmatic Capping in an EAGLE-3 Training Data Pipeline

Introduction

Building a synthetic data generation pipeline for a trillion-parameter Mixture-of-Experts model like Kimi-K2.5 is never a straightforward task. When that pipeline must produce training data for an EAGLE-3 speculative decoding drafter — where every token matters and faithful reproduction of the base model's reasoning traces is essential — the engineering challenges compound rapidly. This segment of the opencode session documents exactly such a journey: a multi-day effort spanning approximately 107 messages, during which the assistant and user navigated three distinct but interconnected crises that threatened to derail the entire project.

The pipeline in question processes roughly 88,000 prompts across eight B-series datasets (B1 through B8) and two pre-tokenized A-series datasets, generating responses via an SGLang inference server running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs. The goal is to produce high-quality training data that faithfully captures the model's reasoning traces and tool-calling behavior, at sufficient throughput to complete the generation in hours rather than days. The three challenges that emerged were:

  1. A reasoning capture bug that silently stripped the model's chain-of-thought from saved responses, corrupting the training data at its source.
  2. A throughput bottleneck that threatened to stretch a 57-hour generation timeline into something even more untenable, requiring deep investigation into KV cache behavior and server configuration.
  3. A dataset size capping problem that forced a redefinition of the generation problem itself — from "generate all the data" to "generate enough data" — with a pragmatic per-dataset token budget. Each challenge was resolved through a combination of diagnostic rigor, architectural pivots, and careful optimization. This article traces the full arc of this segment, examining how each crisis was identified, investigated, and resolved, and what broader lessons emerge for anyone building large-scale inference pipelines for training data generation.

Part I: The Reasoning Capture Bug — When Faithful Data Hinges on an API Choice

The Discovery

The first crisis was discovered almost by accident. In message [msg 3729], the assistant sampled 100 responses from the running inference pipeline and found something alarming: the reasoning field was empty for every single sample, even though the model was clearly generating reasoning content (the thinking blocks were present in the raw output). The usage.completion_tokens field counted all tokens (including reasoning), but the saved content field only captured the visible part of the response. The reasoning content was being silently dropped.

The user confirmed the problem in message [msg 3746], pasting two JSON records from the pipeline's output showing that reasoning was an empty string while the content field contained what should have been reasoning text. The user's diagnosis was precise: "Seems again we're not capturing reasoning correctly (this was also a bug previously fixed in 10k version)."

The Root Cause

The root cause was a configuration gap. The SGLang inference server had been launched without the --reasoning-parser flag. Without this flag, SGLang does not split the model's output into reasoning_content and content fields. Instead, it returns everything — including the thinking text — embedded in the content field, with reasoning_content set to null. The run_inference.py script, written to use OpenAI's chat completions API format, attempted to extract reasoning via getattr(msg, "reasoning", None), which returned an empty string.

The assistant initially attempted to fix this by restarting the server with --reasoning-parser kimi_k2. But this led to a deeper discovery. When the assistant tested the /generate endpoint (SGLang's native API), it found that even with the reasoning parser enabled, the output_ids field was missing the thinking token (token ID 163606). The reasoning parser was stripping it from both the text output AND the raw token IDs.

The Critical Insight

This was the critical insight. The thinking token (163606) was already being appended by apply_chat_template as the last token of the prompt — the model never generates it, it's part of the input. The model's generation begins after thinking, and when it finishes reasoning, it naturally emits the response token (163607) to transition to its final answer. The reasoning parser was unnecessary and actively harmful.

The decision was made in message [msg 3788]: abandon the OpenAI-compatible API entirely and use SGLang's /generate endpoint with raw input_ids/output_ids. The assistant wrote: "Right — for EAGLE-3 training we want the exact token sequence. Either: 1. Correct parsers that properly separate reasoning/content/tool_calls so we can reconstruct perfectly, 2. Or skip all parsing and get raw token IDs directly. Option 2 is cleaner."

The Rewrite

The rewrite of run_inference.py in message [msg 3800] implemented this pivot. The key changes were:

The Side Effect

This seemingly clean fix had an unintended consequence. The new response format placed completion_tokens, prompt_tokens, and output_ids at the top level of the JSON response, rather than nested inside a usage object within a choices array. The monitoring dashboard — specifically stats_collector.py and monitor.py — was still parsing the old nested format, causing token counts to silently default to zero. This monitoring blind spot would only be discovered later when the user reported that the interactive monitor UI didn't show token counts.

Part II: The Throughput Optimization Odyssey — Three Levers and a Rollback

The 57-Hour Wake-Up Call

Even before the reasoning capture bug was fully resolved, a separate crisis was brewing. In message [msg 3728], the assistant checked on the running inference pipeline and did the math: at 26 completions per minute, with 88,088 prompts across 8 datasets, the total generation time would be approximately 57 hours.

The assistant's response was methodical. It wrote a Python script inline within an SSH command to compute per-dataset estimates: B1_glaive at 10,000 prompts (~6.4 hours), B2_opencodeinstruct at 14,714 prompts (~9.4 hours), B3_magicoder at 10,000 prompts (~6.4 hours), and so on, totaling ~57 hours. This was a sobering number that launched one of the most detailed inference optimization investigations in the entire session.

The KV Cache Bottleneck

The initial diagnosis revealed that the server was achieving approximately 780–890 tok/s, but the effective throughput was constrained by KV cache saturation. With --mem-fraction-static 0.85, the server could hold approximately 116,171 tokens in its KV cache across all 8 GPUs. With B2_opencodeinstruct's prompts averaging 4,000+ tokens, only about 35–50 concurrent requests could run simultaneously while 100+ sat queued. The server's raw generation throughput was healthy, but the scheduler couldn't admit new requests because there was no room for their KV cache entries.

The assistant systematically explored three optimization levers, each with its own failure mode and lesson.

Lever 1: Increasing Memory Fraction

Raising --mem-fraction-static from 0.85 to 0.93 doubled the token capacity to 231,120 tokens — a promising improvement. But this configuration left only 0.82 GB of free GPU memory per rank after CUDA graph capture. When the inference pipeline launched with aggressive concurrency (150 short requests), a large prefill batch triggered an OOM crash. The assistant learned that memory fraction tuning requires not just maximizing capacity but also preserving headroom for transient allocations during prefill and CUDA graph operations.

Lever 2: Hierarchical Cache

The assistant then turned to SGLang's --enable-hierarchical-cache feature, which spills evicted KV cache entries from GPU memory to host RAM. This seemed like the ideal solution: the machine had 449 GB of host RAM, and allocating even a fraction of it to KV cache overflow would dramatically expand effective capacity.

But the first attempt with --hicache-size 300 failed catastrophically. The hicache size parameter is specified per TP rank, not total. With 8 tensor-parallel ranks, each attempting to allocate 300 GB, the server tried to consume 2.4 TB of host memory, causing an OOM that consumed 439 GB of the 449 GB available. The assistant had to recover by killing processes from the Proxmox host level, as the container's GPU memory was stuck due to kernel context staleness.

The fix came from reading SGLang's source code (memory_pool_host.py and hiradix_cache.py) to confirm the per-rank semantics. With 445 GB available after cleanup, approximately 40 GB needed for model loading overhead, and a safety margin, the safe per-rank size was approximately 48 GB. The corrected launch with --hicache-size 48 succeeded, yielding 8 × 48 = 384 GB of host RAM for KV cache overflow.

But then a deeper insight emerged. The assistant observed that throughput, after an initial peak of 1,317 tok/s, collapsed to 606 tok/s with only 34 active requests and 115 queued. Token usage was at 0.94 — the GPU KV cache was nearly full. The hierarchical cache, it turned out, helps with evicted radix cache entries for prefix reuse, but for actively generating requests, the KV data must live on GPU memory. Hicache does not increase the number of concurrent decode slots — it only helps when new requests share prefixes with previously completed ones. For a workload where each request has a unique, long prompt, prefix reuse is minimal, and the hicache provides little benefit during steady-state decode.

This was the critical diagnostic breakthrough. The real bottleneck was not memory efficiency or scheduling — it was raw GPU KV cache capacity. With 159,277 GPU token slots (at mem_fraction_static=0.88) and 4,000 tokens per request, the server could fit at most ~40 concurrent decodes. Each decode generated tokens at roughly 17 tok/s, yielding a theoretical maximum of ~680 tok/s — exactly what was observed.

Lever 3: KV Cache Quantization

With the root cause identified, the solution was clear: reduce the memory per token. SGLang supported --kv-cache-dtype fp8_e4m3, which stores KV cache entries in 8-bit floating point instead of 16-bit, halving memory requirements. The assistant launched with all levers combined: mem_fraction_static=0.90, kv-cache-dtype=fp8_e4m3, and hicache=48GB. The results were spectacular: max_total_num_tokens=376,029 — a 3.2× increase over the baseline — with 134–150 concurrent requests and 1,300–1,450 tok/s generation throughput.

The Rollback: Quality Over Throughput

But then came the user's intervention: "Uh don't do fp8 kv cache, that degrades the model noticeably." In seven words, the user vetoed the single most impactful optimization. The Kimi-K2.5 model is already quantized to INT4 and uses Multi-head Latent Attention (MLA), which compresses the KV cache into a low-dimensional latent space. Adding FP8 quantization on top of these existing compressions risked compounding information loss, producing training data with degraded quality that would propagate into the EAGLE-3 drafter.

The assistant's response was immediate and professional: no debate, no data-driven justification attempt — just acceptance and action. The FP8 server was killed, zombie GPU processes were cleaned up, and a new server was launched with mem_fraction_static=0.88, bf16 KV cache, and hierarchical cache retained. The result: 159,277 max tokens, approximately 930 tok/s throughput, and a stable configuration that the user trusted.

The final throughput analysis was a masterclass in accepting hardware reality: "930 tok/s, 72 running, 79 queued, token usage 0.96. Hitting KV ceiling again with the long B2 sequences... This is the hardware limit for this model+GPU combo on long sequences." The assistant calculated that at approximately 0.4–0.5 req/s effective on B2 with 4K average tokens, the pipeline was delivering approximately 1,600–2,000 tok/s effective throughput — "meaningfully better than the 400 tok/s we started with."

Part III: The Pragmatic Cap — When Optimization Meets Reality

The 10 Million Token Question

With the server stabilized at approximately 930 tok/s, the assistant faced a sobering calculation: generating all 88,000 samples across all datasets would take approximately 57 hours. This was the moment when the user's strategic insight transformed the entire approach.

The user proposed a simple but powerful constraint: "Maybe in each category instead doing the whhole thing just do 10M tokens per category?" This was not a technical optimization — it was a redefinition of the problem. Instead of "generate all the data," the problem became "generate enough data." The user understood that beyond a certain scale, additional training data from any single category yields diminishing returns, and that preserving diversity across categories was more valuable than maximizing volume in any one category.

The assistant immediately recognized the wisdom in this approach and calculated the implications. With 8 categories × 10M tokens = 80M tokens total, and the current throughput of approximately 930 tok/s, the estimated generation time dropped to 17–26 hours — a 2–3× improvement over the uncapped estimate. The assistant computed per-dataset sample counts by dividing 10M by each dataset's average token length, producing a precise plan: B1 needed 6,341 of its 10,000 samples, B2 needed 2,636 of its 14,714, and so on.

The Implementation Journey

The assistant implemented the cap through a series of surgical edits to run_inference.py. The initial approach was a simple --max-samples flag that truncated the prompt list to a fixed number per dataset. This was a pragmatic shortcut: instead of implementing a token-count cap (which would require pre-computing token lengths for all prompts or dynamically stopping mid-batch), the assistant used a sample-count proxy.

But this proxy failed for B2_opencodeinstruct, which averaged 3,793 tokens per response. Seven thousand samples at that average yields 26.6 million tokens — more than 2.6× the intended 10M budget. The user caught this in real-time: "Still inferecing B2 even tho it's now >10M." This observation triggered a pivot to a proper token-budget implementation.

The token budget required threading a max_total_tokens parameter through every layer of the pipeline: the async coroutine managing concurrent requests (run_dataset_inference), the dataset-level orchestrator (process_dataset), the resume logic that counts tokens from prior runs, the "already complete" early-exit check, and the CLI argument parser. Each edit was applied in sequence, with the assistant reading the relevant code sections before each modification. The final edit — fixing the "already complete" check to compare against token budget rather than sample count — was the last stitch that made the entire system consistent.

The Monitoring Blind Spot

The format change from the OpenAI-compatible API to the /generate endpoint had a side effect that was only discovered when the user reported: "Btw the interactive monitor.py UI doesn't show token counts." The stats collector (stats_collector.py) was still parsing the old nested usage.completion_tokens format, while the new format placed completion_tokens at the top level. The result was that avg_comp and total_comp were silently defaulting to zero — the dashboard rendered without token counts, but without any error or warning.

The assistant fixed both stats_collector.py and monitor.py to handle both formats, ensuring backward compatibility with any legacy data. This fix restored observability just in time for the user to notice the B2 budget overshoot — a testament to the tight feedback loop between monitoring infrastructure and operational awareness.

Themes and Broader Lessons

Several themes emerge from this segment that are worth highlighting for anyone building large-scale inference pipelines.

The Cascade of Consequences

Every architectural decision in this session created ripples that had to be managed downstream. The switch to the /generate endpoint fixed the reasoning capture bug but broke the stats collector. The FP8 KV cache optimization tripled throughput but was rejected on quality grounds. The sample-count cap was simple to implement but failed for high-token datasets. Each fix revealed the next problem, creating a natural chain of discovery and resolution. This cascade is not a sign of poor engineering — it is the natural consequence of working with complex, interconnected systems where no change is truly isolated.

The Primacy of Quality

The user's rejection of FP8 KV cache — despite its dramatic throughput improvement — established a clear hierarchy of priorities: data quality trumps generation speed. This is a crucial principle for training data pipelines, where the downstream cost of degraded quality (a weaker drafter model) far exceeds the upstream cost of slower generation. The assistant's willingness to accept this constraint without debate, and to pivot immediately to a quality-preserving configuration, demonstrates a mature understanding of the project's true objective.

The Value of Observability

The stats collector fix was not just about fixing a bug — it was about restoring the feedback loop that enabled the user to catch the B2 budget overshoot. Without token counts visible in the dashboard, the pipeline would have silently exceeded its budget, and the user would have discovered the issue only when it was too late to correct. Observability is not a luxury; it is the mechanism by which operators maintain awareness of system behavior and catch problems before they become crises.

Pragmatic Engineering Under Constraints

The entire arc of this segment is a story of working within limits: GPU memory limits, host RAM limits, quality constraints, time budgets. The assistant's response to each constraint was not to fight it but to understand it, quantify it, and work within it. The final configuration — bf16 KV at mem_fraction_static=0.88 with hierarchical cache and a 10M token per-dataset cap — is not the theoretical optimum, but it is a working system that produces useful training data at a predictable rate.

The Human-in-the-Loop Advantage

Throughout this segment, the user played a critical role that no automated system could have replicated. The user spotted the empty reasoning field, questioned whether short responses were a problem, vetoed the FP8 KV cache on quality grounds, caught the B2 budget overshoot, and demanded verification of tool-calling behavior. These interventions caught issues that automated checks might have missed. The collaboration between human judgment and machine execution was the key to navigating all three crises successfully.

Conclusion

The EAGLE-3 training pipeline that emerged from this segment was fundamentally different from the one that entered it. The original pipeline used OpenAI's chat completions API, had no reasoning capture, achieved approximately 600 tok/s throughput, and had no limits on generation time. The final pipeline used SGLang's raw /generate endpoint, captured exact token sequences including reasoning and tool-call special tokens, achieved approximately 930–1,350 tok/s throughput, and had a per-dataset token budget.

More importantly, the team now had a deep understanding of their system. They knew how SGLang's reasoning parser interacted with the chat template. They knew the KV cache limits of their 8-GPU setup. They knew the token distribution characteristics of each dataset. They had verification scripts to confirm data integrity. And they had a monitoring infrastructure that could surface token counts and catch budget overshoots.

The three challenges of this segment — reasoning capture, throughput optimization, and pragmatic capping — were not setbacks. They were learning opportunities that forced the team to understand their system at a deeper level. Each challenge revealed an assumption that needed to be questioned, a measurement that needed to be taken, or a design that needed to be rethought. The pipeline that emerged was not just faster and more correct — it was better understood, better documented, and better equipped to handle the next challenge that would inevitably arise.

In the high-stakes world of training speculative decoding drafters for trillion-parameter models, this kind of deep understanding is not a luxury — it is a necessity. The three challenges of this segment were the crucible in which that understanding was forged.