The 250W Paradox: Why 100% GPU Utilization Doesn't Mean Saturated Performance

Introduction

In the high-stakes world of large language model training, few metrics are as closely watched as GPU utilization. When a cluster of NVIDIA RTX PRO 6000 Blackwell GPUs—each rated for 600 watts of thermal design power—shows 100% utilization but draws only 250 watts, something is fundamentally wrong. This discrepancy is the central puzzle addressed in message 4252 of an extended EAGLE-3 training session for the Kimi-K2.5 language model. The assistant's response to a sharp-eyed user observation reveals deep truths about GPU workload characterization, the economics of batch sizing, and the subtle ways that monitoring tools can mislead even experienced practitioners.

The message sits at a critical juncture in the training pipeline. The team had just restarted a 100K-sample EAGLE-3 drafter training run with --ttt-steps 5 (up from 3) to enable deeper speculative decoding, after the user correctly identified that a PCIe-connected multi-GPU system would benefit from 10–16 token speculation depths to amortize communication overhead. Now, barely minutes into the new run, the user notices something alarming: the GPUs are drawing only 250 watts out of a possible 600. The assistant's response—the subject of this article—diagnoses the root cause and takes decisive corrective action.

The Message: A Diagnostic in Miniature

Let us examine the message in full:

Good eye — 250W/600W means the GPUs are underutilized despite showing 100% GPU-Util. The 100% util just means the GPU always has some work, but the work per step is small (tiny batch). The drafter model is only 1.2B trainable params and batch_size=1 with sequence packing to 4096 tokens — that's a very small workload per step.

>

Let me kill this and restart with higher batch size and longer packed sequences to actually saturate the GPUs: [bash] ssh -o ConnectTimeout=10 root@[REDACTED_IP] 'pct exec 129 -- bash -c "ps aux | grep python3 | grep -v grep | awk \"{print \\\$2}\" | xargs -r kill -9; sleep 2; fuser -k /dev/nvidia* 2>/dev/null; echo done"' done

This is a remarkably dense piece of technical communication. In two paragraphs and one shell command, the assistant:

  1. Validates the user's observation
  2. Explains the root cause of the discrepancy
  3. Connects the symptom to the architectural details of the training configuration
  4. Makes a decision to abort the current run
  5. Executes that decision

The GPU Utilization Paradox

The core insight in this message is the distinction between GPU utilization percentage and GPU power draw—a distinction that many practitioners conflate. The assistant explains that "100% util just means the GPU always has some work, but the work per step is small." This is a critical point: modern GPU utilization metrics (as reported by nvidia-smi and similar tools) measure what fraction of the time the GPU has at least one kernel running, not how many of its cores are active or how hard they are working.

A GPU running a tiny matrix multiplication of size 1×4096 every millisecond will show 100% utilization because it's never idle—it always has a kernel queued. But the actual computational throughput (FLOPs delivered per second) might be a tiny fraction of the GPU's peak capability. The power draw tells the real story: a Blackwell RTX PRO 6000 GPU drawing 250W while rated for 600W is operating at roughly 42% of its thermal design capacity, indicating that the majority of its compute units are idle or clock-gated during each kernel execution.

This phenomenon is particularly acute for small models. The EAGLE-3 drafter being trained has only 1.2 billion trainable parameters (out of 2.6B total, with embeddings and verifier head frozen). For context, the verifier model (Kimi-K2.5) is likely an order of magnitude larger. A 1.2B parameter model is, by modern LLM standards, quite small—it fits comfortably in the VRAM of a single GPU with room to spare. When distributed across 4 GPUs via FSDP (Fully Sharded Data Parallelism), each GPU holds only ~300M parameters worth of trainable weights. The compute per forward-backward pass is correspondingly modest.

The Batch Size Bottleneck

The assistant identifies the proximate cause: batch_size=1 with sequence packing to 4096 tokens. In the speculators library used for EAGLE-3 training, "batch_size=1" means a single sequence per GPU per step. The sequence packing mechanism concatenates multiple shorter training examples into a single 4096-token sequence, which improves data efficiency but doesn't increase the total token count processed per step beyond 4096 tokens per GPU.

With 4 GPUs each processing 4096 tokens per step, the global batch is 16,384 tokens. For a 1.2B parameter model, this is a very small workload. The forward pass involves a handful of matrix multiplications: an embedding lookup, a single Llama decoder layer (attention + MLP), a projection layer, and an LM head. The backward pass doubles this compute. Modern GPUs can process this amount of work in a few milliseconds, but the synchronization overhead of FSDP (allreducing gradients across 4 GPUs) and the Python-level training loop overhead (data loading, loss computation, logging) mean that each step takes on the order of 0.7 seconds (as seen in the earlier rate of ~1.4 steps/s across 4 GPUs). The GPU spends most of that 0.7 seconds waiting—either for the allreduce to complete, for the CPU to prepare the next batch, or for the small kernel launches to propagate through the driver stack.

The power draw of 250W reflects this "bursty" compute pattern: the GPU ramps up to high power for a few milliseconds of actual computation, then idles at low power while waiting for the next step to begin. The utilization metric, however, sees only that the GPU was never completely idle and reports 100%.

The Decision: Kill and Restart

The assistant's response is decisive: kill the current training and restart with higher batch size and longer packed sequences. This is a non-trivial decision because it sacrifices the training progress already made (approximately 4 minutes of training, or roughly 0.1% of the estimated 35-hour run). However, the cost of not restarting would be far greater: 35 hours of training at 42% GPU utilization means effectively wasting 58% of the available compute capacity. The wall-clock time would be the same (the GPU-limited step rate would remain unchanged), but the effective throughput in terms of tokens processed per dollar of GPU time would be dramatically suboptimal.

The assistant does not specify the new batch size in this message—that comes in subsequent messages. But the reasoning is clear: larger batches mean more compute per step, which means the GPU spends more time in high-power, high-throughput computation and less time idling. The goal is to push the power draw from 250W toward 600W, indicating that the GPU's compute units are fully occupied.

Assumptions and Domain Knowledge

This message makes several implicit assumptions that are worth examining:

Assumption 1: The bottleneck is batch size, not data loading. The assistant assumes that increasing batch size will increase GPU compute load without hitting other bottlenecks. This is a reasonable assumption given the earlier observation that disk I/O was only 12% utilized and CPU was 93% idle (from message 4244). However, increasing batch size also increases memory pressure—the 28.5 GB VRAM usage observed in message 4240 would grow, potentially hitting the 96 GB ceiling of the RTX PRO 6000. The assistant implicitly assumes there is sufficient headroom.

Assumption 2: Sequence packing can be scaled up. The current packing to 4096 tokens per sequence can likely be increased to 8192 or 16384 tokens. This requires that the individual training examples can be concatenated without exceeding the model's maximum sequence length. The EAGLE-3 architecture likely supports longer sequences, but the Triton shared-memory OOM issue encountered earlier (mentioned in the chunk summary) at seq_len=16384 suggests there are practical limits.

Assumption 3: The user wants maximum throughput, not minimum time-to-first-result. By restarting the training, the assistant prioritizes long-term efficiency over short-term progress. This aligns with the user's earlier concern about PCIe roundtrip amortization and deep speculation—the user is clearly optimizing for production inference throughput, not rapid prototyping.

Input knowledge required to understand this message includes:

Was the Decision Correct?

The assistant's decision to kill the training and restart with higher batch size is almost certainly correct, but it's worth examining potential counterarguments.

One could argue that the training should have been allowed to continue for a few more steps to gather more data on the convergence behavior before making a configuration change. The loss was dropping well (from 11.0 to 7.2 in the first few hundred steps), and changing batch size changes the training dynamics—the effective learning rate scales with batch size, and the optimizer's behavior may differ. However, the assistant had already demonstrated that the training was converging correctly with the smaller batch size, and the restart with larger batches would simply be a scaling of the same configuration. The speculators library's handling of batch size (via sequence packing) means that the effective training dynamics should be similar.

Another consideration: the assistant could have attempted to increase batch size without killing the run, by modifying the training script's parameters on the fly. However, the speculators library likely initializes the data loader and optimizer at startup based on the command-line arguments, making runtime changes impractical. A clean restart is the safer approach.

The only real cost is the lost training time (~4 minutes out of ~35 hours, or 0.2%). This is negligible. The assistant's decision is sound.

The Broader Context

This message exemplifies a pattern that recurs throughout the EAGLE-3 training saga: the tension between what monitoring tools report and what is actually happening in the hardware. Earlier in the session, the assistant had to debug a Triton shared-memory OOM error, fix weight key name mismatches, and correct a hidden state concatenation bug. Each of these required looking past surface-level symptoms to understand the underlying hardware or software dynamics.

The GPU power utilization issue is particularly instructive because it involves a metric (GPU-Util) that is widely trusted but easily misinterpreted. The assistant's explanation—that 100% util means "the GPU always has some work, but the work per step is small"—is a lesson that applies to any practitioner working with GPU-accelerated training. It's a reminder that utilization percentages are not throughput numbers, and that power draw is often a better proxy for actual compute utilization.

Conclusion

Message 4252 is a masterclass in diagnostic reasoning under time pressure. In just two paragraphs, the assistant identifies a subtle performance issue, explains its root cause in accessible terms, connects it to the specific architectural details of the training configuration, and takes decisive corrective action. The message demonstrates that effective GPU utilization optimization requires understanding not just the high-level metrics but the underlying hardware behavior—the relationship between kernel launch frequency, compute intensity, and power draw.

The decision to kill and restart the training, sacrificing minimal progress for dramatically better hardware utilization, reflects a pragmatic engineering mindset. In the world of large-scale ML training, where a single run can consume thousands of GPU-hours, the ability to recognize and correct suboptimal configurations early is a force multiplier. This message captures that moment of recognition and the action that follows.