Sizing Up the Work: Diagnosing GPU Underutilization in EAGLE-3 Training

Introduction

In the course of training an EAGLE-3 draft model for speculative decoding with the Kimi-K2.5 language model, a seemingly minor observation from the user — "Also was seeing only 250/600W GPU power use, can we bump batch or sth to get better util?" — triggered a deep diagnostic investigation by the AI assistant. Message [msg 4254] captures the moment when the assistant pivots from a running training job to fundamentally rethink the batch sizing strategy. This message is a turning point: it identifies the root cause of GPU underutilization, formulates a plan to fix it, and takes the first concrete step by reading the training script's argument parser. The message is brief in terms of tool calls (a single read), but dense with analytical reasoning that reveals how the assistant thinks about GPU compute efficiency, sequence packing, and the interplay between data loading and model execution.

The Context: A Training Run That Looks Right But Feels Wrong

The training of the EAGLE-3 draft model had been launched on 4 NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB of VRAM. The initial run used batch_size=1 (implicitly, via the speculators library's defaults) and max_seq_len=4096. From the GPU-Util metric, the GPUs appeared fully utilized at 100%. But the user noticed something the metrics didn't capture: power draw was only 250W per GPU, far below the 600W thermal design power (TDP). This discrepancy is a classic signal in GPU computing — a GPU can show 100% utilization while doing very lightweight work per unit time, because the utilization metric measures whether the GPU is idle or busy, not how hard it is working.

The assistant had previously killed the first training run (TTT=3) and restarted with TTT=5 after the user agreed to the tradeoff. Now, with the user's power observation, the assistant had to kill the training again and diagnose the real issue.

The Core Insight: Unpacking the Packing

The message opens with a crisp diagnosis:

The collate function packs multiple samples into a single max_len sequence. With batch_size=1 in the DataLoader, each GPU step processes one packed sequence of 4096 tokens. That's tiny.

This is the key insight. The speculators library uses a packing collate function: instead of processing individual sequences of varying lengths, it concatenates multiple training samples into a single long sequence up to max_seq_len tokens. This is a common technique to avoid wasted padding and improve throughput. However, when batch_size=1 is combined with max_seq_len=4096, each GPU forward pass only processes one packed sequence of 4096 tokens. For a model with only ~1.2 billion trainable parameters (the EAGLE-3 drafter), this is a trivially small computation. The GPU finishes the work almost instantly and then waits for the next batch, resulting in low sustained power draw despite showing 100% utilization.

The assistant correctly identifies that the utilization metric is misleading: "The 100% util just means the GPU always has some work, but the work per step is small." This is a nuanced understanding of GPU performance monitoring that many practitioners miss.

The Proposed Solution: Saturation Through Scale

With 96 GB of VRAM per GPU and only 28.5 GB used, the assistant identifies massive headroom. The proposed solution is twofold:

  1. Increase batch_size: Process multiple packed sequences per step instead of just one.
  2. Increase max_seq_len: Pack more tokens into each sequence, increasing the computational work per step. Both changes increase the amount of work per GPU step, which should drive up power draw and true compute utilization. The assistant doesn't guess at the new values — it first reads the training script to understand the argument parser and determine what parameters can be modified.

The Read Tool Call: A Methodical Approach

The message includes a single tool call: reading the training script at /home/theuser/glm-kimi-sm120-rtx6000bw/eagle3-train/04_train.py. This is a deliberate, methodical action. Rather than guessing what arguments the script accepts or blindly modifying the speculators library internals, the assistant goes to the source of truth — the training script's argument parser.

The read returns lines 215-220 of the script, showing the beginning of the main() function and the argument parser setup. The assistant can see that the script uses argparse and can determine where to add a --batch-size argument. This is a small but important step: it shows the assistant's commitment to understanding the code before modifying it.

Assumptions and Their Validity

The message rests on several assumptions:

Assumption 1: The low power draw is caused by insufficient batch size, not by other factors. This is a reasonable assumption. The GPUs are at 100% utilization but low power, which typically means the kernel launch overhead dominates and the actual compute work per kernel is tiny. Increasing batch size increases the matrix dimensions, making each kernel do more work. However, there could be other bottlenecks — data loading, CPU-side preprocessing, or communication overhead. The assistant implicitly assumes these are not the primary constraint, which is supported by earlier observations (the NVMe was only 12% utilized, CPU was 93% idle).

Assumption 2: The speculators library's collate function packs samples correctly and the packing is the limiting factor. This is verified by the assistant's earlier analysis of the collate function code (from a previous message). The packing is confirmed to work as described.

Assumption 3: Increasing batch_size and max_seq_len will not cause out-of-memory errors. With 96 GB VRAM and only 28.5 GB used, there is ~67.5 GB of headroom. Even increasing the workload 3-4x would still leave plenty of margin. However, the assistant doesn't yet know the memory scaling behavior — doubling batch_size might not double memory usage linearly due to activation memory, optimizer states, and FSDP buffers.

Assumption 4: The training script can be easily modified to accept a --batch-size argument. This is almost certainly true for a well-structured training script using argparse, but the assistant hasn't yet verified that the script doesn't hardcode batch_size elsewhere.

What Knowledge Was Required

To understand and write this message, the assistant needed:

  1. GPU architecture knowledge: Understanding that GPU-Util (%) measures whether the GPU is busy, not how hard it's working, and that power draw is a better proxy for actual compute intensity.
  2. Sequence packing mechanics: Understanding how the speculators library's collate function works — that it concatenates multiple training samples into a single packed sequence up to max_seq_len.
  3. EAGLE-3 model architecture: Knowing that the draft model is only ~1.2B trainable parameters (plus frozen embeddings and verifier LM head), which is tiny relative to the GPU's compute capacity.
  4. VRAM budgeting: Knowing that 28.5 GB used out of 96 GB available leaves massive headroom for larger batches.
  5. The training script's structure: Knowing that the script uses argparse and can be extended with new arguments.
  6. The conversation history: Knowing that the user asked about power utilization and that the previous training run was killed.

What Knowledge Was Created

This message creates several important outputs:

  1. A clear diagnosis: The root cause of low GPU power draw is identified — tiny per-step workload due to batch_size=1 and max_seq_len=4096.
  2. A concrete action plan: Increase both batch_size and max_seq_len to saturate the GPUs.
  3. A code reading: The assistant reads the training script's argument parser, establishing the baseline for modification.
  4. A precedent for optimization: This message establishes that GPU utilization optimization is an ongoing concern, not a one-time setup. The assistant is willing to kill and restart training jobs to improve efficiency.

The Thinking Process

The assistant's reasoning in this message follows a clear diagnostic chain:

  1. Observe symptom: GPU power is only 250W/600W despite 100% utilization.
  2. Form hypothesis: The per-step workload is too small.
  3. Verify hypothesis: The collate function packs samples into a single 4096-token sequence at batch_size=1. This is indeed tiny work.
  4. Quantify headroom: 28.5 GB used vs 96 GB available → 67.5 GB free.
  5. Formulate solution: Increase batch_size and max_seq_len.
  6. Take action: Read the training script to understand the argument parser. The chain is logical and data-driven. The assistant doesn't jump to conclusions — it verifies each step with evidence from the code and system metrics.

Broader Significance

This message represents a common but often overlooked aspect of large-scale ML training: the difference between utilization metrics and actual compute efficiency. A GPU showing 100% utilization can be doing trivial work, especially when the model is small relative to the GPU's capacity. The assistant's willingness to dig into the collate function, understand the packing mechanics, and propose a targeted fix demonstrates the kind of systems-level thinking required for efficient training at scale.

The message also highlights the value of user observations. The user noticed the low power draw and asked about it, which triggered this investigation. In many ML workflows, such observations are dismissed or ignored, but here they led to a meaningful optimization that could cut training time significantly.

Conclusion

Message [msg 4254] is a compact but dense piece of technical reasoning. It diagnoses a subtle GPU utilization problem, proposes a concrete fix, and takes the first step toward implementation. The message exemplifies the kind of systems-level debugging that separates efficient training from merely functional training. By understanding the collate function, the packing mechanics, and the GPU utilization metrics, the assistant turns a user's casual observation into a targeted optimization that could save tens of hours of training time.