The Cost of Non-Streaming Output: Losing 2,700 Samples to a Design Flaw

Introduction

In the course of a complex machine learning pipeline spanning model deployment, inference serving, and EAGLE-3 speculative decoding training, a single short message from the assistant reveals a painful but instructive moment: the realization that 2,700 successfully completed inference samples had been permanently lost. The message, indexed as <msg id=2912>, reads in full:

Zero — the old script only wrote the file at the end, not streaming. So the 2700 completed samples were lost when we killed it. The new script streams to disk as each result comes in.

>

Let me also verify httpx is available: [bash] ssh root@10.1.230.174 '/root/ml-env/bin/python3 -c "import httpx; print(httpx.__version__)"' 0.28.1

This brief exchange—a single line of analysis followed by a verification command—encapsulates a cascade of decisions, assumptions, and their consequences. It is the moment where an invisible design flaw becomes visible, and where the assistant pivots from recovery to prevention.

The Broader Context: Building a Synthetic Training Dataset

To understand why this message matters, one must understand the pipeline in which it sits. The team had been working for days to deploy and optimize inference for Kimi-K2.5, a 1-trillion-parameter Mixture-of-Experts model, on a machine with 8 NVIDIA Blackwell RTX PRO 6000 GPUs. After extensive profiling and benchmarking (see [chunk 22.0]), they had achieved stable throughput of approximately 1,500–1,600 tokens per second serving the INT4 quantized variant of the model.

The current task was to generate high-quality synthetic training data for an EAGLE-3 speculative decoding drafter. Rather than using a generic dataset, the team decided to capture Kimi-K2.5's actual reasoning outputs by feeding questions from the open-perfectblend dataset through the live vLLM inference server. The script 01b_generate_synthetic.py was designed to send each question as an independent chat completion request (at a concurrency of 200, with up to 8,192 completion tokens), capture both the reasoning field and the content from the model's response, and reconstruct the full token sequence with the correct special tokens (<|thinking|> and <|response|>) wrapping the reasoning content.

This approach—using the model's own outputs as training data—is a form of self-distillation or behavioral cloning. The hypothesis was that a drafter trained on the actual distribution of Kimi-K2.5's generations would perform better at speculative decoding than one trained on a generic corpus or on a different model's outputs.

The Timeout Crisis

The inference run had been proceeding for about 100 minutes when the user checked the logs and discovered a problem: a significant number of requests were timing out. The user's terminal output (shown in <msg id=2899>) revealed an 8% error rate—222 timeouts out of 2,700 completed requests. The user manually interrupted the process and suggested tuning the concurrency down to 128 and increasing the timeout.

The assistant immediately recognized the root cause: the OpenAI Python client's default timeout (60 seconds) was far too short for 8K-token generations at high concurrency. When 200 requests are all in-flight simultaneously, each individual request can take minutes to complete because the vLLM server is scheduling them through its continuous batching engine. A 60-second timeout guarantees failure for any request that isn't lucky enough to be scheduled promptly.

The assistant killed the inference process and began rewriting 01b_generate_synthetic.py to address four issues simultaneously: increasing the client timeout to 600 seconds, reducing concurrency to 128, adding retry logic for transient failures, and—crucially—adding the ability to resume from an existing output file so that already-completed samples would not need to be re-generated.

The Moment of Discovery

After implementing the fixes and copying the updated script to the remote machine, the assistant performed a routine check: how many good results had already been saved to disk?

ssh root@10.1.230.174 'wc -l /data/eagle3/synth_25k/prepared/raw_responses.jsonl 2>/dev/null || echo "0 lines"'

The answer: 0 lines.

This is the subject message. The assistant's response—"Zero"—is delivered with the flat affect of a factual report, but the implications are staggering. The old script had been designed to write the output file only at the very end, after all 25,000 samples had completed. The 2,700 samples that had been successfully processed by the vLLM server—representing roughly 3.4 million tokens of generation, or about 38 minutes of compute time at 1,500 tok/s—existed only in the Python process's memory. When the process was killed, they vanished.

The Mistake: Non-Streaming Output

The design flaw was subtle but consequential. The original 01b_generate_synthetic.py accumulated all results in a list and wrote them to disk in a single batch at the end. This pattern is common in prototyping—it's simpler to implement, avoids partial-file concerns, and works fine for small datasets. But for a 25,000-sample run that could take 5–10 hours, it was catastrophic. Any interruption—a timeout, a network blip, a manual kill, a server crash—would lose all progress.

The assistant's assumption was that the script would run to completion uninterrupted. This assumption was baked into the architecture: no intermediate checkpoints, no streaming writes, no resume capability. The timeout issue exposed this fragility, but the real lesson is that any long-running data generation pipeline must be designed for interruption from the start.

The Fix: Streaming and Resume

The new version of the script, as patched in <msgs id=2904-2909>, addressed this by:

  1. Streaming writes: Each completed sample is written to raw_responses.jsonl immediately via an asyncio-managed file handle, using orjson for fast serialization.
  2. Resume support: Before starting, the script reads the existing raw_responses.jsonl to build a set of already-completed sample IDs. It skips those samples, picking up exactly where it left off.
  3. Progress logging: A separate progress file is updated periodically so the user can monitor progress without parsing the JSONL file.
  4. Timeout handling: The httpx.AsyncClient timeout was increased from the default 60 seconds to 600 seconds, and individual request timeouts are caught and retried rather than causing a hard failure. The assistant's verification of httpx version 0.28.1 in the subject message is a small but telling detail. The new streaming approach relies on httpx for the underlying HTTP transport (the OpenAI client wraps httpx internally), and the assistant wanted to confirm it was available and at a modern version. This check reflects a methodical engineering mindset: before deploying a fix that depends on a library, verify that the library is present and functional.

Input Knowledge Required

To fully understand this message, one needs:

Output Knowledge Created

This message creates several pieces of actionable knowledge:

  1. Confirmation of data loss: 2,700 samples were irretrievably lost, establishing the need for the streaming fix.
  2. Validation of the streaming approach: The new script design is confirmed to be the correct response.
  3. Verification of httpx availability: Version 0.28.1 is confirmed present, unblocking the streaming implementation.
  4. A design lesson: Non-streaming output for long-running pipelines is a fragility risk.

The Thinking Process

The assistant's reasoning, visible in the progression from <msg id=2911> to <msg id=2912>, follows a clear chain:

  1. Check the damage: Before restarting, assess how much work was preserved. The command wc -l on the output file is the obvious first step.
  2. Interpret the result: "0 lines" means total loss. The assistant immediately identifies the root cause: the old script's non-streaming architecture.
  3. Quantify the loss: "2700 completed samples" — the assistant knows exactly how many were done from the last progress log (2,700/25,000 at the time of kill).
  4. Contrast old and new: "The old script only wrote the file at the end, not streaming. The new script streams to disk as each result comes in." This is a clear before/after comparison that explains both the failure and the fix.
  5. Verify dependencies: The httpx check is a prudent step before deploying the updated script. If httpx were missing or too old, the streaming approach would need adjustment.

Broader Implications

This episode illustrates a recurring tension in ML engineering pipelines: the tradeoff between simplicity and robustness. The original non-streaming design was simpler to implement and reason about, but it traded robustness for that simplicity. The fix—streaming writes with resume support—adds complexity (file handle management, deduplication logic, partial-file handling) but provides resilience against the inevitable interruptions that occur in real-world deployments.

The loss of 2,700 samples is not just a compute waste. Those samples represented real diversity from the open-perfectblend dataset, and their loss could affect the quality of the EAGLE-3 drafter. If the timeout errors were correlated with certain types of questions (e.g., longer reasoning chains), the surviving samples might be biased toward shorter, simpler responses. The streaming fix prevents further loss but cannot recover what was already discarded.

Conclusion

Message <msg id=2912> is a small moment with large implications. In two sentences and a verification command, it documents a failure mode, quantifies the cost, identifies the root cause, validates the solution, and checks a dependency. It is a textbook example of the kind of real-time debugging and recovery that characterizes production ML engineering—where the difference between a script that writes at the end and a script that writes as it goes can mean the difference between 2,700 usable samples and nothing at all.