The Batch Save Breakthrough: Eliminating Per-Sample I/O Overhead in GPU Extraction Pipelines

Message 7353: "The core change: instead of 1 safetensors file per sample, write 1 safetensors per batch containing all samples keyed by index. Fewer file ops, fewer S3 puts, much less syscall overhead."

In the course of building a high-throughput hidden state extraction pipeline for training a DFlash speculative decoding drafter, the assistant reached a critical inflection point. Message 7353 is deceptively brief — a single sentence describing a code change, followed by a confirmation that a file was written. But this message represents the culmination of a multi-step diagnostic process that identified the fundamental bottleneck throttling an entire GPU cluster, and the elegant fix that would unlock an order-of-magnitude throughput improvement.

The Problem: GPUs Starved by the Filesystem

To understand why message 7353 matters, we must reconstruct the chain of events that led to it. The assistant had built a custom offline hidden state extraction pipeline using HuggingFace Transformers, because the speculators library's online vLLM integration was incompatible with Qwen3.6-27B's GDN hybrid KV cache architecture. This pipeline loaded the 55GB Qwen3.6-27B model on each of four RTX PRO 6000 Blackwell GPUs (96GB each), then iterated through a 913,786-sample tokenized dataset, running the model forward pass to extract per-layer hidden states needed for drafter training.

The initial implementation wrote each sample's hidden states to its own safetensors file on disk, then uploaded each file to S3 via a subprocess-based async uploader. This per-sample approach was the natural first design — it was simple, allowed independent uploads, and made resume logic straightforward since each sample was a self-contained file.

But the performance was abysmal. In message 7351, the user reported a critical observation: "There's really high cpu use, in SYS and USR, SYS when GPUs active, USR when idle." This pattern was the diagnostic key. High SYS CPU (kernel time) during GPU activity meant the GPUs were constantly being interrupted by system calls. High USR CPU (user time) when GPUs were idle meant the Python process was busy serializing data while the GPU sat waiting for the next batch.

The Diagnosis: 545 Syscalls Per Batch

In message 7352, the assistant connected the dots. With a batch size of 545 samples (the first batch after sorting by sequence length), each batch required 545 individual save_file calls. Each call triggered a file create, a write to disk, and an fsync to ensure durability — all kernel operations that required context switches, consumed CPU cycles, and blocked the Python thread. Meanwhile, the GPU had finished its forward pass in milliseconds and was waiting for the CPU to finish I/O before it could receive the next batch.

The assistant's diagnosis was precise: "High SYS CPU when GPUs active = kernel overhead from the 545 individual save_file calls per batch (each one does a file create + write + fsync). High USR when idle = the safetensors serialization or dataset reads."

This was a textbook I/O bottleneck, but with an unusual signature: the I/O wasn't slow storage (the NVMe drives could handle far more throughput), but the sheer number of individual file operations was overwhelming the kernel's syscall handling capacity. Each 50KB safetensors file required the same syscall overhead as a 1GB file — the overhead was per-operation, not per-byte.

The Solution: One File Per Batch

Message 7353 implements the fix. Instead of writing one safetensors file per sample, the assistant rewrites the extraction script to accumulate all hidden states from a batch into a single safetensors file, with each sample keyed by its index within the batch. This collapses 545 file operations into 1, 545 S3 uploads into 1, and eliminates nearly all the syscall overhead.

The change is conceptually simple but architecturally significant. It required restructuring the data flow: instead of a tight loop of "extract → serialize → write → queue upload → repeat" for each sample, the new pipeline does "extract all samples in batch → concatenate tensors on GPU → single .cpu() transfer → single save_file → single S3 upload." The GPU→CPU transfer, previously done per-sample (545 transfers per batch), now happens once. The safetensors serialization, previously invoked 545 times, now runs once. The S3 upload queue, previously receiving 545 entries, now receives one.

Assumptions and Knowledge Required

Understanding this message requires knowledge of several layers of the system. First, the reader must know what safetensors files are — the SafeTensors format used by HuggingFace for storing tensor data, which provides fast zero-copy loading and metadata safety. Second, the reader must understand the concept of syscall overhead: each file operation (open, write, fsync, close) requires a transition from user space to kernel space, which has a non-trivial cost in CPU cycles and cache pollution. Third, the reader must grasp the GPU pipeline model: the GPU executes forward passes asynchronously, but the CPU must prepare batches, transfer data, and save results. If the CPU is busy with I/O, the GPU starves.

The assistant made several assumptions in this message. It assumed that batching the file writes would not cause memory pressure (accumulating tensors for 545 samples in memory before writing). It assumed that the S3 upload subprocess could handle larger files without increased latency (a single 27MB file vs 545 50KB files). It assumed that the resume logic could be adapted to skip individual samples within a batch rather than entire batches. And it assumed that the bottleneck was indeed syscall overhead rather than, say, network bandwidth to S3 or GPU kernel launch overhead.

Was the Diagnosis Correct?

The evidence strongly supports the assistant's diagnosis. The user had observed that SYS CPU spiked when GPUs were active — exactly the pattern you'd expect if the GPU was completing its work quickly and the CPU was thrashing on file operations. The subsequent performance improvement (documented in the chunk summary as a jump from 7-11 samples/s per GPU to 140-155 samples/s per GPU) confirms that the per-sample I/O was the primary bottleneck. The 14x throughput improvement is consistent with eliminating 545 syscalls per batch.

However, there was a subtle incorrect assumption in the earlier message 7345, where the assistant blamed the GIL (Global Interpreter Lock) from boto3.upload_file for the GPU starvation. The assistant initially moved S3 uploads to subprocesses (message 7346), which helped but didn't fully solve the problem. The real bottleneck was the per-sample file writes themselves, not the S3 uploads. Message 7353 corrects this by targeting the root cause: the number of file operations, not the thread contention.

Output Knowledge Created

Message 7353 creates a new version of extract_hidden_states.py that implements batched safetensors writes. This is not just a code change — it's a reusable architectural pattern for any GPU extraction pipeline that needs to save intermediate results. The key insight — that per-sample file I/O can be the hidden bottleneck in GPU pipelines — is transferable knowledge. Many ML engineers default to per-sample file outputs for simplicity, but this case demonstrates that the cost can be catastrophic at scale.

The message also implicitly documents a diagnostic methodology: when GPU utilization is low despite high CPU usage, look at the type of CPU usage. SYS time indicates kernel overhead from system calls; USR time indicates serialization or computation. This distinction, which the user's observation enabled and the assistant exploited, is a powerful debugging tool for performance-critical ML pipelines.

The Broader Context

This message sits within a larger narrative about the gap between research code and production infrastructure. The assistant had already pivoted from the speculators library's vLLM integration (which couldn't handle GDN hybrid models) to a custom Transformers-based pipeline. It had already moved S3 uploads to subprocesses to avoid GIL contention. It had already sorted the dataset by sequence length to maximize GPU utilization. Message 7353 is the final piece of the optimization puzzle — the one that transforms a pipeline that "works" into one that "works at scale."

The 14x throughput improvement from this single change is remarkable, but it's the diagnostic process that's truly instructive. The assistant didn't guess at the bottleneck; it used the user's observation about CPU time breakdown, correlated it with GPU utilization patterns, and traced the root cause to a specific system call pattern. Message 7353 is the artifact of that diagnosis — a small code change with outsized impact, born from careful observation and systems thinking.