When Optimization Breaks the Buffer: The Tmpfs Overflow in a High-Throughput Hidden State Extraction Pipeline

In the course of building a training dataset for a DFlash speculative decoding drafter, a hidden state extraction pipeline was optimized from a crawl to a sprint — only to discover that the sprint could not stop itself from crashing into a wall. This article examines a single user message ([msg 7413]) that reports a critical failure in that pipeline: the /dev/shm tmpfs buffer has completely filled up, causing serialization errors on two of the four GPU shards. The message is a brief, almost offhand report, but it encapsulates the tension between throughput optimization and resource management that defines production machine learning engineering.

The Message

The user reports:

Also: GPU2: 6%|▌ | 412/7364 [21:12<6:02:37, 3.13s/it] Err: Error while serializing: I/O error: No space left on device (os error 28)

>

GPU2: 6%|▌ | 413/7364 [21:16<6:23:11, 3.31s/it] Err: Error while serializing: I/O error: No space left on device (os error 28)

>

GPU2: 6%|▌ | 414/7364 [21:20<6:38:28, 3.44s/it]--- GPU 3 --- GPU3: 6%|▌ | 415/7352 [21:12<6:03:56, 3.15s/it] Err: Error while serializing: I/O error: No space left on device (os error 28)

>

GPU3: 6%|▌ | 416/7352 [21:16<6:23:40, 3.32s/it] Err: Error while serializing: I/O error: No space left on device (os error 28)

>

GPU3: 6%|▌ | 417/7352 [21:20<6:38:39, 3.45s/it] -- Filesystem Size Used Avail Use% Mounted on overlay 1.1T 66G 981G 7% / tmpfs 64M 0 64M 0% /dev shm 251G 251G 0 100% /dev/shm /dev/nvme0n1 14T 8.3T 5.7T 60% /etc/hosts tmpfs 504G 0 504G 0% /sys/fs/cgroup tmpfs 101G 5.8M 101G 1% /run/nvidia-persistenced/socket /dev/mapper/ubuntu--vg-ubuntu--lv 98G 24G 70G 26% /usr/bin/nvidia-smi tmpfs 4.0K 4.0K 0 100% /run/nvidia-ctk-hook Probably want some backpressure around 90%.. S3 uploads don't entirely keep up

The df output tells the story: /dev/shm is at 251 GB used out of 251 GB — 100% full, zero bytes available. The overlay filesystem has 981 GB free. The NVMe has 5.7 TB free. The bottleneck is not disk space; it is tmpfs space. The user's diagnosis is succinct: "Probably want some backpressure around 90%.. S3 uploads don't entirely keep up."

The Optimization Journey That Led Here

To understand why this happened, we must trace the optimization chain that preceded it. The hidden state extraction pipeline was processing a 913,786-sample dataset to capture intermediate hidden states from the Qwen3.6-27B model — states that would later be used to train a DFlash speculative decoding drafter. The pipeline ran on a 4× RTX PRO 6000 Blackwell node, with each GPU handling one shard of the dataset.

The initial extraction ran at 7–11 samples per second per GPU, with high CPU system overhead. The assistant diagnosed the problem in stages. First, it identified that writing safetensors files to the container's overlay filesystem was causing 50% system CPU time ([msg 7386]). The fix was to redirect writes to /dev/shm — a tmpfs mount that lives entirely in RAM, eliminating filesystem kernel overhead. This improved SYS CPU but left the rates at only 4.7–8.6 samples/s per GPU ([msg 7391]).

The assistant then identified a deeper bottleneck: the per-sample loop was performing 2,725 individual GPU-to-CPU tensor copies per batch (545 samples × 5 layer captures), each requiring a .cpu() call, a .to(torch.bfloat16) conversion, and a concatenation ([msg 7398]). The fix was to concatenate all tensors on GPU and perform a single bulk .cpu() transfer per batch. This was the breakthrough: rates jumped from ~35 samples/s aggregate to ~590 samples/s — a 17× improvement ([msg 7400]). GPU utilization hit 100% on all four cards, CPU dropped to near zero, and the estimated completion time collapsed from 8–10 hours to about 25 minutes.

But this optimization created a new problem. The pipeline wrote each batch of hidden states to /dev/shm as a temporary file, then queued it for asynchronous S3 upload. With the old throughput of ~35 samples/s, the S3 uploads could keep pace. At ~590 samples/s, each GPU was producing data faster than the S3 upload thread could drain the tmpfs buffer. The 251 GB tmpfs filled to capacity, and the serialization calls began failing with "No space left on device."## The Assumption That Broke

The assistant's optimization made a subtle assumption: that the S3 upload pipeline could drain the tmpfs buffer faster than the extraction pipeline could fill it. This assumption was reasonable given the previous throughput — at 35 samples/s, the data production rate was low enough that the upload thread had ample time to transfer files to S3 and remove them from tmpfs. But when throughput increased 17-fold, the data production rate exceeded the upload consumption rate, and the buffer filled irreversibly.

The assumption was not explicitly stated. It was embedded in the architecture: write to tmpfs (fast), queue for S3 (async), and trust that the async upload would drain the buffer before the next batch. There was no monitoring of tmpfs usage, no backpressure mechanism, and no rate-limiting on the extraction side. The system was designed for the old throughput regime and was not parameterized to handle a 17× speedup.

This is a classic engineering failure mode: optimizing one component of a pipeline without verifying that downstream components can handle the increased throughput. The S3 upload was never benchmarked independently; its capacity was simply "enough" until it wasn't.

The User's Diagnosis

The user's message is remarkable for its brevity and precision. Rather than describing the problem in abstract terms, the user pastes the actual error output from the extraction processes — the tqdm progress bars showing batches 412–417 on GPU2 and 415–417 on GPU3, each failing with "Error while serializing: I/O error: No space left on device." The ETA estimates in the progress bars are telling: they climb from 6 hours to 6 hours 38 minutes over the span of a few batches, reflecting the increasing latency as the tmpfs fills and serialization attempts begin to fail.

The user then appends the full df output, which reveals the precise state of every filesystem. The overlay filesystem has 981 GB free. The NVMe has 5.7 TB free. But /dev/shm — the tmpfs that was chosen specifically to avoid overlay filesystem overhead — is at 251 GB used, 0 available, 100% full. The diagnosis is delivered in a single line: "Probably want some backpressure around 90%.. S3 uploads don't entirely keep up."

This is a masterclass in concise debugging. The user provides the symptom (serialization errors), the context (progress bars with ETAs), the evidence (df output), and the root cause analysis (S3 uploads can't drain tmpfs fast enough) in a single message. The solution is implied: add backpressure to pause extraction when tmpfs usage exceeds a threshold (e.g., 90%), giving the S3 upload thread time to catch up.

The Thinking Process

The user's reasoning is visible in the structure of the message. The first line — "Also:" — signals that this is an additional observation, not the primary status update. The user was likely monitoring the extraction progress (perhaps via the Flask monitoring WebUI that was set up earlier, or via direct SSH) and noticed that GPU2 and GPU3 were stuck at 6% with climbing ETAs. The serialization errors confirmed that something was wrong with file writing.

The decision to include the full df output is strategic. The user could have simply said "tmpfs is full," but by providing the raw filesystem table, they enable the assistant to see the full picture: the overlay has plenty of space, the NVMe has plenty of space, but the tmpfs — the one filesystem that was specifically chosen for its speed — is completely exhausted. This eliminates any confusion about which storage layer is the bottleneck.

The final line — "Probably want some backpressure around 90%.. S3 uploads don't entirely keep up" — shows the user's mental model of the system. They understand that the extraction pipeline is a producer-consumer system with a bounded buffer (tmpfs). The producer (hidden state extraction) is running at ~590 samples/s, the consumer (S3 upload) is running at some lower rate, and the buffer (251 GB tmpfs) has finite capacity. The fix is to add a control loop that monitors buffer occupancy and throttles the producer when the buffer exceeds a threshold. This is textbook producer-consumer backpressure, applied to a machine learning pipeline.

Input Knowledge Required

To understand this message, one needs to know several things about the system architecture:

  1. The hidden state extraction pipeline writes intermediate hidden states to /dev/shm (tmpfs) before uploading them to S3. This was a deliberate optimization to avoid overlay filesystem overhead ([msg 7386]).
  2. The pipeline was recently optimized to perform GPU-side concatenation and a single bulk .cpu() transfer per batch, achieving a 17× throughput improvement from ~35 to ~590 samples/s aggregate (<msg id=7398-7400>).
  3. The S3 upload is asynchronous — files are written to tmpfs, queued, and uploaded by a background thread. The upload rate was never explicitly measured or bounded.
  4. The /dev/shm tmpfs is 251 GB — a large buffer, but finite. At the old throughput, it was more than sufficient. At the new throughput, it fills faster than the upload thread can drain it.
  5. The extraction runs in shards — GPU0, GPU1, GPU2, GPU3 each process a subset of the dataset independently. The errors appear on GPU2 and GPU3, suggesting that those shards hit the tmpfs capacity limit first (possibly because they were processing longer sequences that produced larger hidden state files).

Output Knowledge Created

This message creates actionable knowledge for the assistant:

  1. The tmpfs buffer is the new bottleneck. The S3 upload pipeline cannot keep pace with the optimized extraction rate.
  2. Backpressure is needed. The extraction must pause or slow down when tmpfs usage exceeds a threshold (e.g., 90%).
  3. The fix is in the extraction script. The extract_hidden_states.py script needs a monitoring loop that checks /dev/shm usage before writing each batch, and sleeps if the buffer is too full.
  4. The system is otherwise healthy. The GPUs are running at high utilization, the overlay filesystem has plenty of space, and the model inference is working correctly. The only issue is the tmpfs overflow.

Mistakes and Incorrect Assumptions

The primary mistake was the implicit assumption that the S3 upload could drain the tmpfs faster than the extraction could fill it. This assumption was never validated — it was simply "good enough" at the old throughput. When the throughput increased 17×, the assumption broke.

A secondary mistake was the lack of monitoring. The progress JSON files tracked processed samples and rates, but they did not track tmpfs usage or S3 upload queue depth. The first indication of trouble was the serialization error itself. Adding a tmpfs usage metric to the progress file would have allowed early detection.

A third issue is that the tmpfs size (251 GB) was not chosen with the new throughput in mind. At 590 samples/s, even with aggressive S3 uploads, the buffer fills in minutes. The buffer size should be proportional to the product of the extraction rate and the maximum expected S3 upload latency. With the old rate of 35 samples/s, a 251 GB buffer was enormous overkill. With the new rate of 590 samples/s, it is barely adequate.

The Broader Lesson

This message illustrates a fundamental principle of systems engineering: optimization exposes new bottlenecks. Every time you make one component of a pipeline faster, you stress-test the downstream components. The assistant's GPU-side concat optimization was brilliant — it eliminated 2,725 individual tensor copies per batch and achieved a 17× speedup. But it also exposed the S3 upload pipeline as the new bottleneck, causing a different kind of failure.

The solution — backpressure — is a universal pattern in distributed systems. It ensures that no component produces data faster than its downstream consumer can handle. In this case, the backpressure mechanism is simple: check tmpfs usage before writing, and sleep if it exceeds a threshold. This is not a fundamental redesign; it is a control loop added to an existing pipeline. But it is essential for making the system robust under varying conditions.

The user's message, though brief, captures the entire arc of this optimization journey: from the triumph of a 17× speedup to the discovery that speed without balance is just a faster way to break things.