The TP Question: A Pivotal Diagnostic Moment in Hidden State Extraction

The Message

"@2026-05-09-201129_2336x1300_scrot.png still piss-poor gpu utilisation and big cpu use. Are we doing TP and maybe shouldnt?"

This short message from the user, accompanied by a screenshot of GPU utilization metrics, arrived at a critical juncture in an intensive optimization session. After hours of iterating on a hidden state extraction pipeline for training a DFlash speculative decoding drafter, the user cut through the technical noise with a single, incisive question: "Are we doing TP and maybe shouldnt?"

The message is deceptively simple. On its surface, it reports continued poor performance despite multiple rounds of optimization. But the real weight lies in the question about tensor parallelism (TP) — a hypothesis that challenged a fundamental architectural decision in the extraction pipeline and would force a re-examination of assumptions about how to efficiently scale data generation across multiple GPUs.

The Context: A Pipeline Under Optimization

To understand why this message matters, we must trace the optimization journey that preceded it. The assistant had built a custom offline hidden state extraction pipeline using HuggingFace Transformers, designed to process a 913,786-sample dataset for DFlash drafter training. The pipeline loaded the full Qwen3.6-27B model (55GB in BF16) on each of 4 GPUs, processed batches of samples, extracted hidden states from specific layers, and uploaded the results to S3.

The initial performance was abysmal: approximately 7–11 samples per second per GPU, with high CPU system (SYS) usage during GPU activity and high user (USR) usage during idle periods. The assistant diagnosed the bottleneck as per-sample safetensors writes — each of the ~545 samples in a batch triggered an individual file create, write, and fsync call, generating enormous kernel overhead. The fix was to batch the saves: accumulate all hidden states in GPU memory and write a single safetensors file per batch.

This optimization worked spectacularly. The rate jumped from ~3.5 samples/s to ~11.3 samples/s per GPU, with aggregate throughput reaching 34.5 samples/s across all 4 GPUs. GPU utilization spiked to 97–100% on some cards. The ETA dropped to approximately 7.3 hours. The assistant declared the pipeline "safe to leave overnight."

The User's Challenge: A Second Look

But the user was not satisfied. The screenshot told a different story from the summary statistics. While the assistant saw "GPU 1: 100.0% util, GPU 2: 97.0% util" and declared victory, the user observed a more nuanced reality: GPU utilization was still "piss-poor" in aggregate, and CPU usage remained high. The screenshot likely showed the utilization pattern over time — brief bursts of activity followed by idle periods, a pattern the assistant had observed earlier but believed was resolved.

This is where the user's question becomes brilliant. Instead of asking "what else can we optimize?" — which would lead down a path of incremental tweaks — the user asked a structural question: "Are we doing TP and maybe shouldnt?"

Deconstructing the TP Question

"TP" refers to tensor parallelism, a technique for sharding a model's weights across multiple GPUs so that each GPU holds only a fraction of the parameters. In standard TP, GPUs communicate during every forward pass to share intermediate results. But the extraction pipeline was not using TP in the traditional sense — each GPU loaded a complete, independent copy of the model and processed disjoint subsets of the data.

However, the user's intuition was that the architecture behaved like TP in a pathological sense: multiple processes competing for shared resources. With 4 independent processes each loading the full 55GB model, the system faced:

  1. Memory bandwidth contention: Each process independently loads model weights from CPU memory to GPU memory. With 4 processes doing this simultaneously, they compete for PCIe bandwidth and CPU memory bandwidth.
  2. Dataset I/O contention: All 4 processes read from the same dataset files (Arrow/Parquet format). The operating system's page cache and disk scheduler must serve 4 competing readers, potentially causing thrashing.
  3. CPU core contention: Each process runs its own Python interpreter, with its own GIL, its own memory allocator, and its own dataset deserialization pipeline. With 4 processes, the CPU cores become a bottleneck for data loading and preprocessing.
  4. Python GIL interactions: While each process has its own GIL (since they're separate processes, not threads), the subprocess-based S3 uploader and the main process still compete for CPU time. The user had previously noted "high cpu use, in SYS and USR, SYS when GPUs active, USR when idle" — a pattern suggesting the CPU was maxed out on data loading (USR) when GPUs were idle, and on kernel operations (SYS) when GPUs were active. The user's question implicitly challenged the assumption that 4 independent processes would scale linearly. Perhaps the system was hitting a shared bottleneck — most likely CPU-side data loading — that made additional GPUs counterproductive beyond a certain point.

Assumptions Embedded in the Architecture

The assistant had made several assumptions in designing the 4-GPU extraction pipeline:

Assumption 1: The dataset loading is trivially parallelizable. The pipeline sorted samples by length and assigned shards to GPUs, assuming each GPU could independently iterate through its portion of the dataset without interference. In practice, all 4 processes were likely reading from the same memory-mapped Arrow files, causing the OS page cache to oscillate between serving different processes' access patterns.

Assumption 2: GPU memory is the primary constraint. With 96GB per GPU and a 55GB model, there was plenty of headroom. But the bottleneck wasn't GPU memory — it was the CPU's ability to prepare data fast enough to keep the GPUs fed.

Assumption 3: More GPUs means more throughput. The pipeline used 4 GPUs to process 4 shards in parallel, expecting 4x throughput. But if the shared bottleneck (dataset I/O, CPU memory bandwidth) was already saturated at 2 GPUs, adding more would yield diminishing returns while consuming additional resources.

Assumption 4: The subprocess-based S3 uploader fully decouples I/O. The assistant had moved S3 uploads to a subprocess to avoid GIL blocking. But the subprocess still competes for CPU cores and memory bandwidth, and the main process still needs to serialize tensors (CPU work) before handing them off.

The Thinking Process: What the User's Question Reveals

The user's message reveals a systems-level thinking process. Rather than looking at individual metrics (GPU util %, CPU util %, samples/s), the user was synthesizing across signals:

  1. The screenshot showed a temporal pattern — GPU utilization was not consistently high but spiky. This pattern is characteristic of a producer-consumer pipeline where the producer (CPU data loading) cannot keep up with the consumer (GPU computation).
  2. The CPU usage remained high despite the batched-save optimization. This suggested the bottleneck had shifted from file I/O to data loading/processing, but hadn't been eliminated.
  3. The question about TP was a hypothesis about shared resource contention. The user recognized that the architecture — multiple independent model instances — might be creating a new class of bottleneck that didn't exist with a single process. This is a classic diagnostic pattern in systems optimization: when incremental improvements to one bottleneck reveal another bottleneck at a different layer of the system. The user was looking at the system holistically and questioning whether the entire parallelization strategy was sound.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the extraction pipeline architecture: That each GPU loads an independent copy of Qwen3.6-27B and processes disjoint data shards.
  2. Knowledge of tensor parallelism (TP): What it is, how it differs from data parallelism, and why it might cause contention.
  3. Knowledge of the optimization history: The previous bottlenecks (per-sample safetensors writes, S3 upload GIL blocking) and the fixes applied.
  4. Knowledge of system resource contention patterns: How multiple processes competing for shared resources (disk I/O, CPU memory bandwidth, page cache) can create bottlenecks that don't exist with single-process execution.
  5. Knowledge of the dataset format: That the 913K samples are stored in Arrow/Parquet files that may be memory-mapped, creating shared access patterns.

Output Knowledge Created

This message created several important insights:

  1. The hypothesis that 4 independent model instances may cause resource contention that negates the benefits of parallelism. This is a testable hypothesis — one could run the extraction with 1, 2, or 4 GPUs and measure per-GPU throughput to find the saturation point.
  2. The recognition that CPU-side data loading is the likely bottleneck, not GPU computation or file I/O. This shifts the optimization target from GPU utilization to data pipeline throughput.
  3. The insight that "looks good in summary stats" ≠ "looks good in practice." The assistant's summary showed 97-100% GPU utilization on some cards, but the user's screenshot revealed a spikier pattern that the averages masked.
  4. A potential architectural pivot: Instead of running 4 independent processes, perhaps the pipeline should use fewer processes with larger batch sizes, or use multiprocessing with shared memory for dataset access, or pre-load the entire dataset into memory.

Mistakes and Incorrect Assumptions

The assistant's primary mistake was declaring victory too early based on aggregate statistics. The summary showed "GPU 1: 100.0% util, GPU 2: 97.0% util" and an ETA of 7.3 hours, which looked like success. But the user's deeper inspection revealed the utilization was not sustained — it was bursty, with periods of high utilization followed by idle waiting.

The assistant also assumed that the batched-save optimization had resolved the bottleneck, when in fact it had merely shifted it. The bottleneck moved from "file I/O per sample" to "dataset loading and CPU-side tensor processing," but the pipeline was still not running at maximum efficiency.

The question "Are we doing TP?" also reveals a potential misunderstanding that the assistant could have clarified: the pipeline was not using tensor parallelism in the traditional sense. Each GPU had its own complete model copy — this was data parallelism, not model parallelism. But the user's intuition about resource contention was correct regardless of the terminology.

Broader Implications

This message exemplifies a crucial dynamic in AI infrastructure work: the tension between summary statistics and actual performance patterns. The assistant, focused on optimizing the code, looked at aggregate rates and declared success. The user, focused on the operational reality, looked at utilization patterns over time and saw a system that was still not running efficiently.

The question "Are we doing TP and maybe shouldnt?" is a model of concise, high-impact diagnostic communication. In 12 words, the user: