The Breakthrough Batch: How Dynamic Length-Aware Batching Finally Unlocked GPU Utilization for Hidden State Extraction

The Message

The subject of this article is a single bash command dispatched by the assistant at message index 7312, which deployed and executed a newly rewritten hidden state extraction script on a remote machine with 4× NVIDIA RTX PRO 6000 Blackwell GPUs:

[assistant] [bash] scp -P 19248 /data/dflash/scripts/extract_hidden_states.py root@154.59.156.20:/workspace/dflash/scripts/ && \
ssh -p 19248 root@154.59.156.20 '
rm -rf /workspace/dflash/data/hidden_states; mkdir -p /workspace/dflash/data/hidden_states
CUDA_VISIBLE_DEVICES=0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
    /workspace/dflash/venv/bin/python3 /workspace/dflash/scripts/extract_hidden_states.py \
    --model-path /workspace/dflash/models/Qwen3.6-27B \
    --data-path /workspace/dflash/data/tokenized \
    --output-dir /workspace/dflash/data/hidden_states \
    --gpu 0 --shard 0 --num-shards 1 \
    --max-samples 1000 --batch-size 256 \
    2>&1 | grep -v "transformers\|Loading weights\|fast path"
' 2>&1
[Shard 0/1] Loading model on cuda:0...
  Model loaded in 13.3s
  Shard 0: samples 0-1000 (1000 samples), max_batch_size=256
  Hooks registered for layers [1, 16, 31, 46, 61]
  32 dynamic batches, sizes: min=2 max=226 avg=31

Shard 0:   0%|          | 0/32 [00:00<?, ?it/s]
Shard 0:   3%|▎         | 1/32 [00:12<06:23, 12.38s/it]
Shard 0:   6%|▋         | 2/32 [00:20<05:04, 10.14s/it]
Shard 0:   9%|▉         | 3/32 [00:27<04:09,  8.61s/it]
Shard 0:  12%|█▎        | 4/32 [00:33<03:34,  7.6...

This output, while seemingly mundane — a progress bar ticking along — represents the culmination of a multi-hour debugging odyssey. After repeated out-of-memory (OOM) failures, a variable-reference bug, and a complete architectural rewrite of the extraction pipeline, the assistant had finally produced a working solution. The progress bar was moving, the GPU was being utilized, and the path to training a better speculative decoding drafter was open.

Why This Message Was Written: The Reasoning and Motivation

To understand why this particular message exists, one must trace the chain of failures that preceded it. The assistant was building a hidden state extraction pipeline to train a DFlash drafter — a small model that predicts the target LLM's hidden states at intermediate layers, enabling speculative decoding. The training data consisted of 913,786 tokenized samples, and the target model was Qwen3.6-27B, a 27-billion-parameter model with a GDN hybrid attention architecture.

The original extraction pipeline, inherited from the speculators library, processed one sample at a time per GPU. This resulted in abysmal GPU utilization — the user reported "5-10%" utilization ([msg 7296]). Each sample averaged only ~335 tokens, meaning the GPU spent most of its time loading the next sample rather than computing. The assistant correctly diagnosed this as an I/O-bound, single-sample bottleneck and committed to batching.

The motivation for message 7312 was thus: after three consecutive failures to achieve batched extraction, the assistant needed to prove that the fourth approach — dynamic length-aware batching with PyTorch hooks — actually worked. This was a make-or-break moment for the entire DFlash training pipeline.

The Chain of Failures: How Decisions Were Made

The path to this message reveals a remarkable pattern of iterative debugging, each step informed by the specific error message of the previous attempt.

Attempt 1: Naive batching with output_hidden_states=True ([msg 7305]). The assistant set batch_size=128, assuming that with 96GB of GPU memory and a 55GB model, the remaining ~40GB would suffice. It OOM'd immediately. The root cause was that output_hidden_states=True forces the model to retain all 65 hidden state tensors (embedding + 64 layers) simultaneously in memory. For a batch of 128 sequences at 3316 tokens each, this is 65 × 128 × 3316 × 5120 × 2 bytes ≈ 280 GB — far exceeding the 96 GB budget. The assistant's assumption that "40GB free" would be enough was wrong because it underestimated the memory overhead of storing intermediate activations across all layers.

Attempt 2: PyTorch hooks to capture only 5 layers (<msg id=7306-7308>). The assistant correctly identified that only 5 specific layers (1, 16, 31, 46, 61) were needed for DFlash training. By registering forward hooks on those layers, the hidden states could be captured and immediately discarded, avoiding the 65-tensor accumulation. This was a sound optimization, but it still OOM'd at batch_size=256 because the forward pass activations themselves — not just the stored hidden states — consumed too much memory. The assistant's diagnosis was precise: "The forward pass activations (not just hidden states) are what's eating memory. With 64 layers of compute, batch × seq_len × hidden_size activations per layer add up fast" ([msg 7309]).

Attempt 3: Dynamic batching by sequence length (<msg id=7309-7310>). The assistant implemented a clever solution: sort samples by length and create batches where short sequences get large batches and long sequences get small batches. The script successfully created 32 dynamic batches (min=2, max=226, avg=31) but then crashed with a variable-reference bug — the inner loop still referenced an old variable name from the previous implementation ([msg 7311]).

Attempt 4: The fix (<msg id=7311-7312>). The assistant fixed the variable reference bug, re-SCP'd the script, and launched the test that produced the output in message 7312. This time, it worked.

Assumptions Made by the Assistant

Several assumptions underpin this message:

1. That 256 is a reasonable max batch size. The assistant assumed that with hooks in place, the memory profile would be dominated by forward-pass activations rather than stored hidden states. The dynamic batching algorithm would then adjust downward for long sequences. This assumption proved correct — the largest batch was 226 (not 256), and the smallest was 2.
2. That PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True helps. This environment variable enables PyTorch's memory allocator to grow segments dynamically rather than pre-allocating large blocks. The assistant included it as a safety measure, though it's unclear whether it was strictly necessary given the dynamic batching.
3. That the model fits on a single GPU with room for batching. The assistant assumed that a 27B BF16 model (~55 GB) on a 96 GB GPU leaves ~40 GB for activations. This was optimistic — the actual headroom was tighter, but the dynamic batching algorithm successfully navigated it.
4. That 1000 samples is a sufficient test. The assistant ran 1000 samples as a validation before scaling to the full 914K dataset. This was a reasonable engineering judgment — enough to exercise the dynamic batching across diverse sequence lengths.

Mistakes and Incorrect Assumptions

The most significant mistake was the initial underestimation of activation memory. The assistant's first assumption — that 40 GB free would comfortably handle batch_size=128 — failed because it didn't account for the multiplicative effect of 64 transformer layers. Each layer's forward pass produces intermediate tensors (attention scores, value projections, MLP intermediates) that are proportional to batch × seq_len × hidden_size. For a 27B model with hidden_size=5120, these intermediates are enormous. The assistant corrected this by switching to hooks, but even hooks don't eliminate the forward-pass memory — they only prevent the accumulation of all-layer outputs.

A second mistake was the variable-reference bug in the dynamic batching implementation. The assistant wrote a new loop structure but left old variable names in the inner code, causing a runtime crash. This is a classic refactoring error — the assistant acknowledged it immediately ("I still have the old loop variable reference") and fixed it in the next edit.

A third, more subtle assumption was that the model would load in the same memory configuration every time. The load time varied from 12.8s to 14.3s across runs, suggesting some variability in memory allocation. The assistant didn't account for this in the extraction logic, but it didn't cause problems in practice.

Input Knowledge Required

To understand this message, one needs:

• Understanding of transformer memory mechanics: Why output_hidden_states=True is memory-prohibitive for batched inference, and how forward hooks avoid this.
• Knowledge of PyTorch's hook system: The assistant registered hooks on specific layer modules to capture intermediate hidden states without storing all layers' outputs.
• Familiarity with the DFlash speculative decoding architecture: DFlash requires hidden states from intermediate layers of the target model (layers 1, 16, 31, 46, 61 for Qwen3.6-27B), not just the final layer.
• Understanding of the Qwen3.6-27B model architecture: The model has 64 layers with hidden_size=5120, uses GDN hybrid attention, and has 65 total hidden states (index 0 = embedding, indices 1-64 = layers 0-63).
• Remote execution patterns: The assistant uses SCP to transfer scripts and SSH to execute them, with grep -v to filter noisy transformer warnings.

Output Knowledge Created

This message produced several important pieces of knowledge:

1. The dynamic batching algorithm works. The output shows 32 batches with sizes ranging from 2 to 226, averaging 31. The progress bar is advancing at a reasonable rate (starting at ~12s per batch, accelerating to ~7.6s per batch as shorter sequences are processed).
2. The hooks are correctly registered. The output confirms "Hooks registered for layers [1, 16, 31, 46, 61]" — the exact set needed for DFlash training.
3. The model loads in ~13 seconds. This is important for understanding the overhead of restarting the extraction pipeline.
4. The throughput is viable. At ~8 seconds per batch with average batch size 31, and each sample averaging ~335 tokens, the single-GPU throughput is approximately 31 samples / 8s ≈ 3.9 samples/second. With 4 GPUs, that's ~15.5 samples/second, meaning the full 914K dataset would take ~16 hours — acceptable for an overnight run.

The Thinking Process Visible in the Reasoning

The assistant's reasoning, visible across the preceding messages, reveals a systematic debugging methodology:

1. Hypothesis formation: "Low utilization is because we're processing 1 sample at a time per GPU" ([msg 7298]).
2. Solution design: "Need batching" — commit to batched extraction.
3. First attempt and failure analysis: OOM at batch=128. The assistant calculates the exact memory required: "65 × 128 × 3316 × 5120 × 2 = ~280GB" ([msg 7306]).
4. Root cause identification: The problem is output_hidden_states=True storing all 65 tensors. Solution: use hooks to capture only 5 layers.
5. Second attempt and failure analysis: Still OOM. The assistant correctly identifies that forward-pass activations, not stored hidden states, are the bottleneck ([msg 7309]).
6. Creative solution: "Sort samples by length and use dynamic batch sizing — small batches for long sequences, large batches for short ones" ([msg 7309]). This is a non-obvious insight that leverages the dataset's length distribution.
7. Third attempt and bug fix: The dynamic batching works (32 batches created) but a variable-reference bug crashes execution. The assistant immediately identifies the issue and fixes it ([msg 7311]).
8. Fourth attempt — success: Message 7312 shows the pipeline running successfully. This pattern — hypothesis, implementation, failure, analysis, refinement — is classic debugging methodology, but the assistant executes it with remarkable speed. Each cycle takes only a few minutes, and the reasoning is precise and quantitative.

Conclusion

Message 7312 represents a quiet victory in a long debugging battle. The progress bar ticking from 0% to 12% over 33 seconds is unremarkable to a casual observer, but to anyone who has followed the chain of OOM errors, hook registrations, and variable-reference bugs, it is the sound of a machine finally working as intended. The dynamic length-aware batching approach — born from the ashes of three failed attempts — is elegant in its simplicity: sort by length, batch aggressively for short sequences, conservatively for long ones. It is the kind of solution that seems obvious in retrospect but required genuine insight to discover.

The hidden state extraction pipeline was now operational. The path to training a better DFlash drafter — the ultimate goal of this entire segment — was finally clear.