The Third Iteration: Tuning Sequence Packing for EAGLE-3 Training Throughput

In the high-stakes world of large language model training, GPU utilization is the difference between a project that finishes in days and one that drags on for weeks. Message [msg 4265] captures a pivotal moment in precisely such an optimization cycle — the third iteration of tuning the EAGLE-3 draft model training pipeline, where the assistant makes a data-driven decision to quadruple the sequence packing length from 8,192 to 32,768 tokens, aiming to slash training time by 4–8×.

The Context: An Optimization Chain Reaction

This message is the latest in a cascade of training restarts, each triggered by a new insight about GPU utilization. The chain began when the user asked about deeper speculative predictions ([msg 4246]), leading the assistant to restart training with --ttt-steps 5 instead of 3 ([msg 4250]). But almost immediately, the user noticed something subtle: the GPUs were drawing only 250W out of their 600W capacity ([msg 4251]). Despite showing 100% GPU-Util in nvidia-smi, the hardware was loafing — the "100% utilization" metric merely indicated the GPU always had some work, not that it was saturated.

This sparked a deep investigation. The assistant discovered that the EAGLE-3 model's forward method rigidly expects a batch dimension of 1 (shape: [1, total_seq_len, ...]), ruling out increasing batch_size in the DataLoader ([msg 4260]). However, the training pipeline uses sequence packing — multiple samples concatenated into a single long sequence with block-diagonal attention masking. The lever for throughput was --max-seq-len, which controls how many tokens fit into each packed batch. At max_seq_len=4096, the GPUs used only 28.5 GB of 96 GB VRAM and drew a paltry 250W.

The first fix was bumping to --max-seq-len 8192 ([msg 4261]). The assistant waited two minutes, then checked the results. Power draw jumped to 412–476W, and VRAM usage climbed to 39.8 GB. Progress.

But then the user delivered a crucial correction: "Max seq 8k sounds like too small for many of train seqs btw" ([msg 4263]). This was the key insight that drives message [msg 4265].

What the Message Actually Says

The subject message is deceptively short — just a status report and a kill command:

Already much better — 412-476W (up from 250W), 39.8 GB VRAM. But still 35,446 batches/epoch because each batch packs only ~1 sample at 8192 seq len. With 96GB available and only 40GB used, we have plenty of room.

>

Let me kill and restart with --max-seq-len 32768 — that'll pack ~4-8 samples per batch, cutting steps/epoch by ~4-8x: [bash] ssh ... kill ...

The message packs three layers of reasoning into a few lines: measurement, diagnosis, and action.

The Reasoning: Why 8,192 Tokens Isn't Enough

The assistant's diagnosis reveals a subtle interaction between sequence packing and data characteristics. The training data consists of sequences that were truncated at 8,192 tokens during the merge phase. When max_seq_len is set to 8,192, the collate function can fit at most one sample per packed batch — and often not even a full one, since many samples are close to the limit. The result: 35,446 batches per epoch, identical to what you'd get with no packing at all.

This is the moment where the assistant synthesizes multiple data points into a decision:

  1. Power draw (412–476W): The GPUs are working harder than at 250W, but still below the 600W thermal design power. Headroom exists.
  2. VRAM usage (39.8 GB): Less than half of the available 96 GB is utilized. Massive memory headroom.
  3. Batch count (35,446): The number of steps per epoch hasn't decreased from the unpacked baseline, confirming that packing isn't happening effectively.
  4. User's observation: The sequences are ~8,192 tokens, confirming that 8,192 is the worst possible setting — just large enough to fit one sample, too small to pack multiple. The conclusion is clear: jump to 32,768 tokens. At that length, the collate function can pack 4–8 samples per batch (assuming a mix of sequence lengths), cutting steps per epoch by the same factor. Training time drops from ~35 hours to potentially under 10.

Assumptions and Risks

The decision rests on several assumptions that could prove wrong:

The first is that the model's flex_attention with block-diagonal masking will handle 32,768-token sequences without running out of memory. Attention complexity is O(n²) per sequence, but with packing, the effective complexity is O(sum of individual sequence lengths²). If the packed batch contains eight 4,096-token sequences, the attention computation is eight independent O(4096²) operations, not one O(32768²) operation. This scaling property is what makes aggressive packing viable — but it depends on the actual distribution of sequence lengths in the data.

The second assumption is that the collate function will efficiently pack sequences up to 32,768 tokens. The packing algorithm greedily fills the max_len buffer with samples until the next one would exceed the limit. If the data contains many long sequences (say, 7,000+ tokens), packing efficiency could be lower than the estimated 4–8×.

The third assumption is about training dynamics. Increasing max_seq_len increases the effective batch size — each step processes more samples. The optimizer's learning rate schedule and batch normalization may behave differently. The assistant is using cosine scheduling with warmup, which should be robust to batch size changes, but it's not guaranteed.

The Thinking Process Visible in the Message

The message reveals a clear diagnostic loop characteristic of experienced ML engineers. The assistant doesn't just report numbers — it interprets them in context. The power draw of 412–476W is "much better" relative to 250W, but the batch count of 35,446 is flagged as a problem because it hasn't changed from the unpacked baseline. The VRAM figure of 39.8 GB is compared against the 96 GB ceiling to justify the next jump.

The structure is: measure → compare to expected → identify root cause → propose fix → execute. This pattern has repeated three times in this optimization cycle (4096→8192→32768), and each iteration has been faster because the assistant now knows what to look for.

Notably, the assistant doesn't wait for confirmation from the user before killing the training. The previous message ([msg 4264]) had already proposed 32,768 or 65,536, and the user's observation about 8k being too small was the green light. The assistant now acts decisively, killing the process and preparing for the next launch.

Input and Output Knowledge

To understand this message, a reader needs knowledge of: sequence packing in transformer training (how multiple samples are concatenated into a single sequence with attention masking), GPU utilization metrics (the difference between GPU-Util percentage and actual power draw), the EAGLE-3 model architecture (specifically its use of flex_attention with BlockMask and the constraint that batch dimension must be 1), the speculators library's data pipeline (the create_collate_fn packing logic), and the characteristics of the training data (sequences truncated at 8,192 tokens).

The message creates new knowledge: the 8,192 configuration is confirmed insufficient for effective packing, the GPUs have significant headroom in both power and memory, and the next target of 32,768 is expected to deliver 4–8× fewer steps per epoch. The kill command output (6944 ... done) confirms the training process was successfully terminated, clearing the GPUs for the next restart.

The Broader Significance

This message exemplifies the iterative nature of production ML training. Each optimization cycle reveals new constraints and opportunities. The initial assumption that 100% GPU-Util meant good utilization was wrong — it took the user's observation about power draw to reveal the truth. The assumption that 8,192 would provide packing benefit was wrong — it took the user's knowledge of the data characteristics to correct it. The assistant's willingness to kill and restart repeatedly, losing only minutes of work each time, reflects a pragmatic approach: better to iterate fast on configuration than to let a suboptimal training run to completion.

The message also highlights the importance of human-in-the-loop optimization. The assistant had the tools to measure and diagnose, but the user's domain knowledge about the data (sequences are ~8,192 tokens) was the critical insight that the automated analysis missed. The collaboration between human intuition and machine execution is what drives the rapid iteration cycle.

In the end, message [msg 4265] is a snapshot of an ML engineer at work: measuring, reasoning, deciding, and executing — all in a few lines of text and a single bash command.