The SWE Agent Anomaly: Understanding Short Completions in EAGLE-3 Training Data Generation

Introduction

In the middle of a large-scale data generation pipeline for training an EAGLE-3 speculative decoding drafter, a seemingly minor observation from the user triggered a revealing exchange. The user noticed that the B8_sweagent dataset — part of a batch of eight datasets being processed through the OpenRouter API — showed unusually short average completion lengths. "Swe Agent long seems like very short avg completions, no? Or just quick to toolcalls there?" they asked in [msg 4078]. The assistant's response, captured in [msg 4079], is a masterclass in real-time data characterization: it explains why the numbers look the way they do, demonstrates an evolving understanding of the dataset as it processes, and reinforces the soundness of the overall pipeline. This message, though brief, sits at a critical juncture where data generation is nearly complete and the pipeline is about to transition into the compute-intensive hidden state extraction phase.

The Message in Full

The assistant's response reads:

Good observation. SWE agent trajectories are very prompt-heavy (long multi-turn conversations with tool outputs) but the model's individual responses are short — it's typically emitting a brief reasoning + a tool call or a short code patch. The avg is climbing as the longer-running requests finish (288→1264 tok/response so far), so the initial burst was the quick ones completing first.

>

Let me check if it finished and get the final summary: [bash] sleep 60 && ssh root@10.1.230.174 'tail -30 /data/eagle3/synth_100k/logs/inference_openrouter.log' 2>/dev/null 2950/3572 (0 err, 2950 ok) 25.6 req/s avg_comp=183tok tot=0.5M ~$1.2 ETA=0.0h 3000/3572 (0 err, 3000 ok) 25.2 req/s avg_comp=185tok tot=0.6M ~$1.2 ETA=0.0h 3050/3572 (0 err, 3050 ok) 24.8 req/s avg_comp=187tok tot=0.6M ~$1.3 ETA=0.0h 3100/3572 (0 err, 3100 ok) 24.4 req/s avg_comp=190tok tot=0.6M ~$1.3 ETA=0.0h 3150/3572 (0 err, 3150 ok) 24.2 req/s avg_comp=192tok tot=0.6M ~$1.3 ETA=0.0h 3200/3572 (0 err, 3200 ok) 22.2 req/s avg_comp=208tok tot=0.7M ~$1.5 ETA=0.0h 3250...

Why This Message Was Written: Context and Motivation

To understand why this message exists, one must understand the pipeline that produced it. The assistant had been running a massive data generation effort for EAGLE-3 training, using OpenRouter API to generate responses across eight datasets (B1 through B8). The pipeline had pivoted from local GPU inference to OpenRouter after earlier attempts at local inference proved too slow or unreliable for the scale needed. The script run_inference_openrouter.py was built to handle 2000 concurrent requests, with provider routing, resume support, and careful token budget tracking.

By the time we reach [msg 4079], the pipeline has already processed B1 through B7. B8_sweagent — the SWE agent trajectories dataset — is the last one running. The user, monitoring progress, sees an anomaly: the average completion length for B8 appears very short compared to earlier datasets. Earlier datasets like B3_magicoder averaged around 2604 tokens per response. B8_sweagent is showing averages around 288 tokens initially, then climbing to 183-208 tokens in the log tail shown. This looks like something might be wrong — are the responses being truncated? Is the model failing to generate proper outputs?

The user's question is reasonable. When you're spending real money on API calls (the total cost for this run was approximately $86), seeing average completion lengths an order of magnitude shorter than expected naturally raises concern. The user wants to know: is this a bug, or is this expected behavior?

The Reasoning: Why SWE Agent Trajectories Are Different

The assistant's explanation reveals a deep understanding of the data being generated. SWE agent (Software Engineering agent) trajectories are fundamentally different from the other datasets in the pipeline. While datasets like B3_magicoder or B5_openthoughts contain standalone reasoning chains or code generation tasks, SWE agent data consists of multi-turn conversations where an AI agent interacts with a software engineering environment.

The key insight is the asymmetry between prompt length and response length. SWE agent prompts are extremely long — they contain the full conversation history, tool outputs, code context, and environmental observations accumulated over multiple turns. A single prompt might be thousands of tokens long. But the model's response in each turn is typically brief: a short reasoning snippet followed by a tool call (like edit_file, bash, or view_file), or a short code patch. The model doesn't need to regenerate the entire conversation — it only needs to decide what action to take next.

This is why the average completion length is low: the denominator includes the full multi-turn trajectory, but each individual response is short. The assistant correctly identifies that "the model's individual responses are short — it's typically emitting a brief reasoning + a tool call or a short code patch."

The Evolving Average: A Window into Real-Time Processing

One of the most interesting aspects of this message is the assistant's observation that "the avg is climbing as the longer-running requests finish (288→1264 tok/response so far)." This reveals a subtle bias in how streaming averages work in a concurrent system.

When 2000 requests are launched simultaneously, the fastest ones complete first. These are the requests with the shortest responses — perhaps single-turn interactions or very brief tool calls. As these quick requests drain from the queue, the remaining pool consists of progressively longer-running requests. The running average therefore climbs over time, not because the underlying distribution is changing, but because the sampling bias shifts as the queue empties.

The assistant notes the average has climbed from 288 to 1264 tokens per response, and the log output shows it continuing to climb (183→185→187→190→192→208 tokens in successive batches). This is the "survivor bias" of concurrent request processing: the short ones finish first, so the average of the remaining pool is always higher than the overall average.

This observation also serves as a subtle confirmation that nothing is wrong. If the average were staying flat at 288, that might indicate a systematic truncation issue. But the fact that it's steadily climbing is exactly what you'd expect from a healthy pipeline processing a heterogeneous dataset.

Assumptions Made

The message rests on several key assumptions:

Assumption 1: SWE agent structure is known. The assistant assumes that SWE agent trajectories follow a predictable pattern of long prompts with short responses. This is a reasonable assumption based on the structure of the SWE-agent benchmark and similar datasets, but it's worth noting that the assistant hasn't actually inspected the raw B8 data yet — it's reasoning from domain knowledge.

Assumption 2: The short completions are genuine, not truncated. The assistant assumes that the OpenRouter API is returning complete responses, not cutting them off. This assumption is supported by the earlier validation work ([msg 4059], [msg 4060]) where 1637 OpenRouter responses were structurally validated with zero issues. The token count audit ([msg 4061]) showed an average difference of only 1.0 tokens (0.04%) between OpenRouter's reported completion_tokens and the reconstructed token IDs. So the assistant has strong empirical grounds to trust the data.

Assumption 3: The climbing average is a selection effect, not a change in behavior. The assistant assumes that the increasing average is due to longer-running requests completing later, not because the model is somehow generating longer responses over time. This is a reasonable statistical interpretation.

Assumption 4: The pipeline is working correctly. The assistant assumes that the 0 errors reported in the log (the log shows "0 err" consistently for B8) means the pipeline is healthy. Earlier datasets had error bursts (B6 showed ~958 errors from initial rate limiting), but B8 has been error-free.

Potential Mistakes and Limitations

While the assistant's reasoning is sound, there are a few points worth examining critically:

The average range discrepancy. The assistant states the average has climbed from 288 to 1264 tokens per response, but the log output shown in the same message shows averages in the 183-208 range. This discrepancy suggests the 1264 figure might have been from an earlier check (perhaps [msg 4075] showed 288, and the assistant extrapolated). The log output in [msg 4079] shows the average is actually still climbing slowly from 183 to 208 — it hasn't reached 1264 yet. This could be a minor imprecision in the assistant's reporting, or the 1264 figure might be a cumulative average across all completed requests rather than the running average of the current batch.

The "longer-running requests" framing. The assistant says "the longer-running requests finish" as if the later-completing requests are inherently longer. But in a concurrent system with 2000-way parallelism, the order of completion is also affected by network latency, provider routing, and queue depth at the API endpoint. A request with a 200-token response that gets routed to a slow provider could finish after a request with a 500-token response that hits a fast provider. The assistant implicitly assumes that completion time is dominated by response length, which is approximately true but not perfectly so.

No structural validation of B8 specifically. The assistant validated B3 (1637 responses) and B4 (25 responses) structurally, but hasn't validated B8. The assumption that B8 follows the same pattern is reasonable given that the same script and API parameters are used, but SWE agent data might have unique edge cases — for example, tool call tokens appearing in the output, or the model generating next tokens that could interfere with the response separator detection.

Input Knowledge Required

To fully understand this message, one needs:

1. Knowledge of the EAGLE-3 training pipeline. The message is the culmination of a long data generation effort. Understanding that the assistant is generating training data for speculative decoding — where a lightweight "drafter" model predicts the base model's hidden states — is essential.
2. Understanding of SWE-agent benchmarks. SWE agent (Software Engineering agent) is a benchmark where AI agents interact with real GitHub repositories to fix bugs. The trajectories are multi-turn conversations with tool calls. Knowing this structure is key to understanding why responses are short.
3. Familiarity with OpenRouter API and concurrent request handling. The pipeline uses 2000 concurrent requests, and the log output shows request rates, error counts, token totals, and cost estimates. Understanding how concurrent processing creates selection bias in running averages is important.
4. Context of the earlier validation work. The message builds on extensive earlier work validating that OpenRouter responses reconstruct correctly ([msg 4059]-[msg 4062]). Without that context, the assistant's confidence in the data might seem unwarranted.
5. Knowledge of the Kimi-K2.5 tokenizer. The special tokens ( response = 163607, <|im_end|> = 163586, thinking = 163606) and their encoding behavior are critical to understanding why the reconstruction works.

Output Knowledge Created

This message produces several valuable outputs:

1. Confirmation that B8_sweagent behavior is expected. The short completion lengths are not a bug but a natural consequence of SWE agent data structure. This prevents unnecessary debugging or re-running.
2. Characterization of SWE agent data. The message provides a clear description of the data: prompt-heavy, short individual responses, multi-turn structure with tool calls. This characterization is useful for downstream processing decisions (e.g., how to handle sequence length limits during hidden state extraction).
3. Evidence of pipeline health. The log output shows 0 errors, steady throughput (~25 req/s), and a climbing average — all signs of a healthy pipeline. The cost tracking (~$1.5 total for B8 so far) confirms the budget is on track.
4. A model of real-time average behavior. The observation about averages climbing as short requests complete first is a useful piece of operational knowledge for anyone running concurrent API pipelines.
5. Transition readiness. The message implicitly signals that B8 is nearly complete (3250/3572 done, ETA=0.0h), setting the stage for the next phase: merging datasets, shuffling, and hidden state extraction.

The Thinking Process

The assistant's thinking process in this message reveals several layers of reasoning:

Layer 1: Pattern recognition. The user flags an anomaly (short completions). The assistant immediately recognizes the pattern from domain knowledge: SWE agent trajectories have this structure. This isn't a learned fact from this specific run — it's a general property of the data type.

Layer 2: Statistical reasoning. The assistant doesn't just explain the static property; it notes the dynamic behavior (the average is climbing). This shows awareness of how concurrent processing creates sampling bias. The assistant is actively monitoring the trend, not just looking at the current value.

Layer 3: Confirmation through action. Rather than just theorizing, the assistant runs a command to check the latest progress. The log output confirms the pattern: the average is indeed climbing (183→208 over the visible range), and the error count is zero. This turns the explanation from speculation into verified observation.

Layer 4: Cost awareness. The assistant tracks the cost (~$1.5 for B8 so far) and implicitly compares it to the remaining budget. This shows the assistant is operating within resource constraints, not just generating data indiscriminately.

Layer 5: Pipeline awareness. The message ends with the log showing 3250/3572 complete. The assistant knows that once this finishes, the entire data generation phase is done, and the pipeline transitions to merging and hidden state extraction. This message is effectively the closing chapter of the data generation phase.

Conclusion

The message in [msg 4079] is a small but revealing moment in a complex pipeline. It demonstrates how domain knowledge, statistical reasoning, and operational monitoring combine to produce confidence in a data generation process. The user's question — a reasonable concern about anomalously short completions — is met with an explanation that is simultaneously specific (this is how SWE agent data works), general (averages climb as short requests finish first), and actionable (let me check the latest numbers). The message serves as a bridge between the data generation phase and the upcoming hidden state extraction phase, confirming that the data is correct and the pipeline is healthy.

In the broader context of the EAGLE-3 training effort, this message represents the final moments of a $86, 33-minute data generation run that produced approximately 40,000 samples and 138.4 million tokens. The careful validation, structural checks, and real-time monitoring that preceded this message — and the confident explanation it enables — are what separate a well-engineered pipeline from a fragile one. When the user sees an anomaly, the assistant doesn't panic or restart; it understands the data well enough to explain why the numbers look the way they do.