The Packing Problem: How One Parameter Change Unlocked 4x GPU Utilization in EAGLE-3 Training

Introduction

In the middle of a marathon EAGLE-3 draft model training session for the Kimi-K2.5 large language model, a single assistant message ([msg 4266]) captures a pivotal moment of optimization. The message is deceptively simple — a bash command that launches a training script with one parameter changed from --max-seq-len 8192 to --max-seq-len 32768. But behind this single-line change lies a rich chain of reasoning spanning multiple rounds of diagnosis, measurement, and iteration. This article unpacks that reasoning, examining how the assistant diagnosed a GPU underutilization problem, traced it to sequence packing mechanics, and converged on a solution that would dramatically accelerate training.

The Context: Training an EAGLE-3 Drafter

To understand this message, one must first understand what is being trained. EAGLE-3 is a speculative decoding framework: a lightweight "draft" model that proposes multiple future tokens in a single forward pass, which a larger "verifier" model then validates in parallel. For a PCIe-connected multi-GPU system — where each allreduce across GPUs incurs significant communication overhead — deeper speculation (more tokens per round) directly translates to fewer PCIe roundtrips and higher throughput.

The assistant was training a 1.2B parameter EAGLE-3 draft model on 100,000 samples of hidden state data extracted from the Kimi-K2.5 verifier. The training used 4 GPUs via torchrun with Fully Sharded Data Parallelism (FSDP). The training script, 04_train.py, was built on the speculators library and used a technique called sequence packing: instead of processing one training sample at a time, the collation function concatenates multiple samples into a single long sequence, with block-diagonal attention masks ensuring each sample attends only to itself. This is the key mechanism for improving GPU utilization when individual samples are short.

The Diagnostic Chain: From 250W to 476W

The story begins at [msg 4251], where the user observed that GPUs were drawing only 250W out of a 600W thermal design power (TDP). Despite showing 100% GPU utilization in nvidia-smi, the power draw revealed that the GPU cores were not being fully saturated. The assistant correctly diagnosed this: 100% utilization in the NVIDIA driver means the GPU always has some work queued, but if each individual computation step is tiny, the GPU spends most of its time in lower-power states between kernel launches.

The root cause was the sequence length. With --max-seq-len 4096 and batch_size=1 (forced by the model architecture, which expects a batch dimension of 1), each training step processed exactly one packed sequence of 4096 tokens. Given that individual training samples averaged around 2,000-4,000 tokens, each step processed at most 1-2 samples. The compute per step was minuscule for a 96 GB GPU, resulting in the low 250W power draw.

The assistant's first fix ([msg 4261]) was to double --max-seq-len to 8192. After 120 seconds of training, the results were promising: power draw jumped to 412-476W across the four GPUs, and VRAM usage climbed to 39.8 GB. The GPUs were finally doing meaningful work.

But the user immediately spotted a problem ([msg 4263]): "Max seq 8k sounds like too small for many of train seqs btw." This was a crucial insight. The training data had sequences up to 8192 tokens (from an earlier merge truncation). With max_seq_len=8192, a sample of length 8000 would occupy almost the entire packed sequence, leaving only 192 tokens of space — not enough to fit another sample. The packing mechanism was effectively disabled for long samples. The assistant confirmed this: "each packed batch fits exactly ONE sample with no packing benefit."

The Decision in Message 4266

This brings us to the target message. The assistant evaluates the situation and makes a decision: increase --max-seq-len to 32768. The reasoning, visible across the preceding messages, is:

  1. VRAM headroom exists: At 8192 seq len, VRAM usage was 39.8 GB out of 96 GB available. There is 56 GB of unused capacity.
  2. Packing ratio scales with max_seq_len: With 32768 tokens per packed sequence, and average sample length of ~2000-4000 tokens, each step would pack 8-16 samples instead of 1-2. This would reduce the number of steps per epoch by a factor of 8-16, proportionally reducing training time.
  3. Attention cost is manageable: The model uses flex_attention with block-diagonal masking. The computational cost scales as O(sum of individual sequence lengths²), not O(total_seq_len²), because each sample's attention is confined to its own block. Longer packed sequences with more samples do not incur the quadratic penalty that a single long sequence would.
  4. The model architecture supports it: The underlying model has max_position_embeddings=131072, so 32768 is well within the supported range. The command itself is straightforward: it kills any existing training processes, clears the output directory, and relaunches with the same parameters except for --max-seq-len 32768. The training script, data paths, and all other hyperparameters remain unchanged.

Assumptions and Potential Risks

The assistant makes several assumptions in this decision:

That the collation function will efficiently pack 32768 tokens. The create_collate_fn in the speculators library greedily packs samples until the cumulative length exceeds max_len. With 32768 tokens, the packing should be efficient, but the distribution of sample lengths matters. If there are many very long samples (near 8192), the packing ratio could be lower than expected. The assistant assumes the average sample is short enough to benefit.

That VRAM at 32768 will not exceed capacity. At 8192 seq len, VRAM was 39.8 GB. Attention memory scales linearly with total sequence length (for the KV cache) and with the number of samples (for the block mask). A 4x increase in sequence length might not produce a 4x increase in VRAM because the attention is block-diagonal. But the assistant is implicitly assuming the total will stay under 96 GB. This is a reasonable assumption given 56 GB of headroom, but not guaranteed — especially with 4 GPUs doing FSDP which replicates optimizer states.

That the training dynamics remain valid. Increasing the packed sequence length changes the batch composition — each step now sees a different random set of samples concatenated together. The gradients are averaged over more samples per step, which changes the effective batch size. The optimizer (AdamW with cosine LR schedule) should handle this, but the convergence trajectory might differ from the shorter-sequence training.

That the data loader can keep up. With 4x the tokens per step, the data loader must read 4x more .pt files per batch. The NVMe drive was reading at 393 MB/s with 12% utilization at the 8192 setting. At 32768, the I/O demand would increase, potentially becoming a bottleneck if disk read speed is insufficient.

Input and Output Knowledge

Input knowledge required to understand this message includes:

The Thinking Process

What is most striking about this message is the iterative, measurement-driven thinking process it represents. The assistant does not jump directly from the 250W observation to 32768. Instead, it follows a careful chain:

  1. Observe symptom: 250W power draw (msg 4251)
  2. Diagnose cause: Small per-step compute due to batch_size=1 and short packed sequences (msg 4252)
  3. Apply first fix: Double max-seq-len to 8192 (msg 4261)
  4. Measure result: 412-476W, 39.8 GB VRAM (msg 4264)
  5. Receive user correction: 8192 is still too small for the data (msg 4263)
  6. Re-evaluate: With 8192, packing is ineffective for long samples; VRAM headroom is 56 GB (msg 4264-4265)
  7. Apply second fix: Quadruple to 32768 (msg 4266) This is not a single insight but a dialogue-driven optimization loop. The user's observation at step 5 is critical — it reveals a subtlety the assistant had missed. The assistant had assumed that doubling max-seq-len would proportionally increase packing, but the user recognized that the distribution of sample lengths matters. If most samples are near 8192, then 8192 max-seq-len gives no packing benefit at all. The fix to 32768 provides enough headroom that even long samples leave space for additional shorter ones.

Conclusion

Message [msg 4266] is a masterclass in practical ML engineering optimization. It demonstrates that the most impactful changes are often not algorithmic innovations but rather careful tuning of data pipeline parameters to match hardware characteristics. The shift from 8192 to 32768 sequence length would ultimately reduce training steps per epoch from 35,446 to approximately 4,000-8,000 (depending on actual packing efficiency), cutting training time from ~35 hours to perhaps 5-10 hours — all from changing a single integer parameter.

More broadly, this message illustrates the value of the assistant's access to real-time monitoring tools. By checking power draw, VRAM usage, and disk I/O, the assistant could make data-driven decisions rather than guessing. And the user's domain expertise — recognizing that 8192 was too small for the specific data distribution — provided the crucial insight that the assistant, lacking direct visibility into the data, could not have deduced on its own. The collaboration between human intuition and machine analysis is what made this optimization possible.