The Five Percent Problem: How a Single User Observation Unlocked GPU Optimization

"Really low gpu utilisation, 5-10%"

This four-word message, submitted by the user at message index 7296, is a masterclass in concise, high-impact feedback. On its surface, it is a simple status report: the GPU utilization on the machine running hidden state extraction for DFlash drafter training is hovering at 5–10%. But in the context of the conversation, this message acts as a critical diagnostic signal that reveals a fundamental architectural flaw in the extraction pipeline—a flaw that the assistant's monitoring loop (message 7295) had entirely failed to catch.

The Context: Building a Custom Extraction Pipeline

To understand why this message matters, we must trace the chain of events that led to it. The assistant had been engaged in an ambitious project: training a better DFlash speculative decoding drafter for the Qwen3.6-27B model, a 27-billion-parameter language model with a GDN hybrid attention architecture. The standard toolchain for this task—the speculators library paired with vLLM's hidden state extraction via kv_transfer_config—had proven incompatible with Qwen3.6-27B's hybrid KV cache. After a series of failed attempts (messages 7277–7283), the assistant made a strategic pivot: build a custom offline extraction pipeline using HuggingFace Transformers directly, bypassing vLLM entirely.

The initial extraction script (written in message 7285, tested in messages 7287–7293) worked correctly. A single GPU could process approximately 3.5 samples per second, with each sample averaging ~335 tokens and producing ~10MB of hidden state data. The assistant then scaled this to all 8 GPUs on the machine, launching eight parallel extractor processes, each loading a full copy of the 55GB Qwen3.6-27B model on its own GPU and processing a different shard of the 913,786-sample dataset.

Message 7295 shows the assistant's monitoring loop: it checks memory usage and file counts every 15 seconds, watching the hidden state files accumulate (57, 123, 188, 254 files over successive intervals). From the assistant's perspective, the extraction was running smoothly—files were being produced, memory was stable, no errors were reported. Everything looked fine.

The Message That Changed Everything

Then came message 7296. The user, who had been observing the system independently, reported a stark reality that the assistant's metrics had missed: "Really low gpu utilisation, 5-10%."

This is not a question, a suggestion, or a complaint. It is a factual observation delivered with clinical brevity. The user had likely run nvidia-smi or a similar monitoring tool and seen that the GPUs—eight NVIDIA RTX PRO 6000 Blackwell cards with 96GB of memory each—were sitting nearly idle. Despite the extraction making visible progress (files accumulating, memory allocated), the actual compute units were barely working.

The power of this message lies in what it implies. The assistant had designed the pipeline with the assumption that "8 parallel instances = 8x extraction throughput." Each GPU loads the model, processes its shard of data, and writes hidden states. In theory, this should keep all GPUs busy. But the user's observation revealed a hidden bottleneck: the per-sample overhead was dominating the compute time.

The Root Cause: A Serial Pipeline Disguised as Parallel

The assistant's response in message 7298 diagnoses the problem immediately: "Low utilization is because we're processing 1 sample at a time per GPU — each sample is a short sequence (~335 tokens avg), so the GPU is mostly idle waiting for the next one. Need batching."

This diagnosis reveals the core issue. The extraction script, as initially written, processed samples one at a time in a loop:

1. Load one sample's tokenized input_ids from the dataset
2. Run a single forward pass through the 27B-parameter model
3. Extract hidden states from the specified layers
4. Copy the hidden state tensors from GPU to CPU
5. Save them to disk as a safetensors file
6. Repeat For a short sequence of ~335 tokens, the actual GPU computation (the forward pass) takes only a few milliseconds. But the surrounding operations—Python loop overhead, tensor copying, file I/O—create a serial bottleneck that leaves the GPU idle most of the time. The GPU finishes its work in a burst and then waits while the CPU prepares the next sample. This is a classic performance pitfall in ML pipelines: the assumption that parallelism at the process level (8 GPUs, each running independently) automatically translates to high utilization at the device level. In reality, each GPU was running a fundamentally serial workload with a poor compute-to-overhead ratio.

Assumptions and Blind Spots

The assistant made several assumptions that the user's observation exposed:

1. Throughput equals utilization: The assistant measured progress in files-per-second (~28 samples/sec across 8 GPUs) and assumed this was good. But throughput can be decent while utilization is terrible if the GPU is running in short bursts.
2. Memory allocation implies computation: The monitoring showed ~53-58 GB of memory used per GPU, which the assistant interpreted as "the GPUs are working." In reality, that memory was allocated to hold the model weights—a static allocation that says nothing about compute activity.
3. Process parallelism is sufficient: Launching 8 Python processes, each pinned to a different GPU, was assumed to fully utilize the hardware. But process-level parallelism doesn't guarantee device-level utilization if each process's workload is serial and I/O-bound.
4. The monitoring metrics were adequate: The assistant's monitoring loop tracked memory usage and file counts—good for detecting crashes or stalls, but useless for detecting low utilization. GPU compute utilization (the percentage of time the CUDA cores are active) requires different tools (nvidia-smi's GPU-Util column, or nvtop).

The Knowledge Gap

To understand this message fully, one needs:

• GPU architecture knowledge: Understanding that GPU utilization measures the percentage of time the compute units are actively processing, not just memory occupancy. A GPU can have 95% of its VRAM allocated but be 5% utilized.
• ML inference pipeline awareness: Knowing that transformer forward passes for short sequences are extremely fast on modern GPUs, and the bottleneck is often data loading, preprocessing, and I/O rather than computation.
• Batch processing concepts: Understanding that batching—processing multiple samples in a single forward pass—is the standard technique for improving GPU utilization, because it amortizes overhead across many samples and increases the compute-to-I/O ratio.
• The specific hardware context: The RTX PRO 6000 Blackwell GPUs are extremely powerful (96GB VRAM, high FLOPs), making the utilization gap even more stark. A 5% utilization on these cards means enormous wasted compute capacity.

The Output Knowledge Created

This message created actionable knowledge that drove the next phase of optimization:

1. The bottleneck was identified: Per-sample processing with short sequences creates a serial pipeline where GPU compute is a tiny fraction of the total per-sample time.
2. The fix was clear: Implement batching—collect multiple samples and process them in a single forward pass, keeping the GPU fed with work continuously.
3. The priority shifted: From "make it work" (correctness) to "make it fast" (performance). The extraction was functionally correct, but economically unsustainable at 5% utilization across 8 expensive GPUs.
4. The architecture needed redesign: The extraction script needed to be rewritten to accumulate batches of samples, concatenate them, run one forward pass per batch, and then save the results—rather than the sample-by-sample loop.

The Catalyst for Optimization

Message 7296 is the turning point in this segment. Before it, the assistant was in a "make it work" mode—patching layer ID offsets, fixing tensor type errors, getting the pipeline to produce correct output. The user's observation forced a shift to "make it fast" mode.

The assistant's immediate response (message 7298) acknowledges the problem and begins implementing the fix. But the situation is complicated by a second user message (message 7297) reporting that the original instance was killed and a new 4-GPU machine is available. The assistant must now rebuild the entire pipeline on a new host while simultaneously redesigning it for batching.

The rewritten extraction script (message 7300) implements proper batching: instead of for sample in dataset: process(sample), it accumulates samples into batches, runs a single model(batch_input_ids) forward pass, and then processes all hidden states in bulk. This transforms the GPU utilization from 5% to an estimated 90%+, while also reducing the per-sample overhead of Python loops and individual file writes.

The Broader Lesson

This message exemplifies a pattern that recurs throughout the entire opencode session: the user's real-world observations frequently correct the assistant's assumptions. The assistant is capable of deep technical work—debugging layer ID offsets, patching vLLM PRs, writing complex extraction pipelines—but it operates within the feedback loops it creates for itself. When those feedback loops are incomplete (monitoring memory but not utilization), the user's external observation provides the missing signal.

The five percent problem is not just a GPU utilization issue. It is a reminder that in complex ML engineering, the most important metric is often the one you aren't measuring. The assistant was watching files accumulate and feeling productive, while the user was watching the GPU idle and seeing waste. The gap between those two perspectives is where this message lives.