The 570-Second Wait: A Readiness Polling Loop That Reveals the Cost of Tensor Parallelism

Introduction

In the middle of an intense benchmarking session comparing parallelism strategies for the Kimi K2.6 model on 8× RTX PRO 6000 Blackwell GPUs, there is a message that at first glance appears trivial: a simple bash loop that polls a remote service until it becomes ready. Message [msg 11539] is exactly that—a 15-line shell script that checks every 15 seconds whether the SGLang inference server has finished loading. The output shows a progression of "loading..." timestamps at 90, 180, 270, 360, 450, and 540 seconds, finally culminating in "READY" at 570 seconds. Nine and a half minutes of waiting.

But this message is far from trivial. It is a quiet witness to one of the most important findings in the entire parallelism optimization campaign: the stark cost of tensor parallelism (TP8) on a PCIe-interconnected GPU cluster. The 570-second wait—nearly four times longer than any previous service start in this session—is not an anomaly. It is a direct consequence of the architectural choice being tested, and it foreshadows the benchmark results that will confirm TP8's fundamental weakness on this hardware.

The Message in Full

The assistant issued a single bash command:

for i in $(seq 1 60); do
    sleep 15
    health=$(curl -s --max-time 5 "http://10.1.2.200:30001/v1/models" 2>/dev/null)
    if echo "$health" | grep -q '"id"' 2>/dev/null; then echo "[$((i*15))s] READY"; break; fi
    st=$(ssh -o ConnectTimeout=5 root@10.1.2.200 "systemctl is-active sglang-k26-tp8-tuned.service" 2>&1)
    if [ "$st" = "failed" ]; then
        echo "[$((i*15))s] FAILED"
        ssh -o ConnectTimeout=5 root@10.1.2.200 "journalctl -u sglang-k26-tp8-tuned.service --no-pager -n 10 | grep -E 'Error|FAILED' | tail -3" 2>&1
        break
    fi
    if [ $((i % 6)) -eq 0 ]; then echo "[$((i*15))s] loading..."; fi
done

The output:

[90s] loading...
[180s] loading...
[270s] loading...
[360s] loading...
[450s] loading...
[540s] loading...
[570s] READY

The script is a well-structured readiness probe with three branches: (1) if the HTTP health endpoint responds with a JSON body containing "id", the service is ready and the loop exits; (2) if systemctl is-active reports "failed", the service has crashed and the loop exits with diagnostic information; (3) otherwise, it prints a "loading..." message every sixth iteration (every 90 seconds) to provide progress feedback without flooding the output.

Why This Message Was Written: The Context

To understand why this message exists, we must look at what immediately preceded it. In [msg 11538], the assistant had just completed a series of parallelism benchmarks. The team had tested EP8 (expert parallelism across 8 GPUs), achieving ~1538 tok/s peak throughput. They had tested EP4 (expert parallelism with TP2 groups), achieving ~1531 tok/s. Both configurations loaded in 90–150 seconds. The assistant then wrote:

"EP4 is slightly better across the board. Now TP8 with CUDA 13 and cuda graphs (the fair TP8 baseline we never had)."

This line is crucial. The assistant recognized a gap in the experimental design: all previous comparisons had been between EP configurations, but the true baseline—pure tensor parallelism without any expert parallelism—had never been properly benchmarked with the tuned settings (CUDA graphs, continuous decode, high max-running-requests). The assistant stopped the EP4 service and deployed a new systemd unit called sglang-k26-tp8-tuned.service, configured with --tp-size 8 and no --ep-size flag, meaning all 8 GPUs would collaborate on every single layer via tensor parallelism.

This message ([msg 11539]) is the readiness check for that deployment. It exists because the assistant needed to know when the service was ready to accept benchmark requests, and because the assistant had learned from experience that services can fail silently or crash during startup—the polling loop provides early failure detection.

The 570-Second Anomaly

Every previous service start in this session had completed within 90–150 seconds. The EP8 and EP4 services both loaded in roughly 120–150 seconds. The TP8 service took 570 seconds—nearly four times longer.

This is not a bug. It is a direct consequence of how tensor parallelism loads model weights. With TP8, every GPU must hold a full copy of all model parameters, sharded across the hidden dimension. For a 590 GB model like Kimi K2.6, this means each GPU loads and processes approximately 74 GB of weights. But more importantly, TP8 requires all GPUs to synchronize at every layer during both loading and inference—every AllReduce operation must complete before the next layer can proceed. On a PCIe-interconnected system (the RTX PRO 6000 GPUs are connected via PCIe, not NVLink), this creates a massive communication bottleneck.

Expert parallelism (EP8), by contrast, shards the MoE (Mixture-of-Experts) layers by expert, meaning each GPU only loads the experts assigned to it. The attention layers still use TP, but the MoE layers—which constitute the bulk of the model's parameters—are distributed without requiring AllReduce across all 8 GPUs. This dramatically reduces both loading time and per-step communication overhead.

The 570-second wait is the PCIe bottleneck made visible in the startup phase. It is the canary in the coal mine for the benchmark results that follow in [msg 11540], where TP8 achieves 97.9 tok/s at C=1 (respectable, thanks to CUDA graphs) but plateaus at ~1291 tok/s peak throughput—significantly lower than EP8's ~1538 tok/s and EP4's ~1531 tok/s. The same PCIe AllReduce overhead that slowed loading by 4× also caps throughput by 15–20%.

Assumptions and Knowledge Required

To fully understand this message, the reader needs several pieces of background knowledge:

1. The hardware topology: The 8× RTX PRO 6000 GPUs are connected via PCIe, not NVLink. This means inter-GPU communication must traverse the PCIe bus, which has limited bandwidth compared to NVLink's high-speed direct GPU-to-GPU interconnects. This assumption is validated by the stark contrast with the later B300 SXM6 NVLink machine ([chunk 64.2]), where TP8 becomes the clear winner.
2. The model architecture: Kimi K2.6 is a Mixture-of-Experts model, meaning it has both dense attention layers and sparse MoE feed-forward layers. The MoE layers can be sharded by expert (EP) without requiring AllReduce, while the attention layers require tensor parallelism (TP) for the full hidden dimension. This hybrid architecture makes parallelism strategy a non-trivial optimization problem.
3. SGLang's parallelism model: The --tp-size and --ep-size flags control how the model is distributed across GPUs. TP shards every operation across all specified GPUs, requiring AllReduce at every step. EP shards only the MoE experts, with AllReduce only needed for the dense layers. The assistant's deep understanding of this distinction drives the entire experimental design.
4. The service lifecycle: SGLang's model loading involves downloading weights (if not cached), mapping them into GPU memory, initializing CUDA graphs (if enabled), and warming up Triton kernels. The health endpoint (/v1/models) only responds once the model is fully loaded and ready to serve requests.
5. The polling pattern: The script uses a 15-second polling interval with a 5-second timeout on the health check, and a separate SSH-based systemctl check for failure detection. The "loading..." message every 6 iterations provides progress feedback without excessive output. This pattern reflects lessons learned from earlier failures where services crashed silently.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

1. TP8 loading time baseline: 570 seconds for Kimi K2.6 on 8× PCIe RTX PRO 6000. This is a data point for future deployment planning—if startup time matters, EP configurations are 4× faster to load.
2. Service stability confirmation: The TP8 service did not crash. It eventually loaded and became ready. This confirms that TP8 is functional on this hardware, even if slow.
3. Failure detection path not triggered: The fact that the script reached READY rather than FAILED means the service never entered a failed state. This is useful information—the long loading time is not due to errors or OOM, but simply the time required for PCIe-bound weight distribution.
4. A reference point for the benchmark: The 570-second wait sets expectations for the benchmark results. When the assistant runs the TP8 benchmark in [msg 11540] and sees 97.9 tok/s at C=1 with a peak of ~1291 tok/s, the long startup time has already hinted at the PCIe bottleneck that will cap throughput.

The Thinking Process Visible in the Message

While this message contains no explicit "reasoning" section, the assistant's thinking is embedded in the script's structure:

• Why 60 iterations? The loop allows up to 60 × 15 = 900 seconds (15 minutes) of waiting. The assistant knows that model loading can take a long time, especially for a 590 GB model on PCIe. The 60-iteration limit provides a generous upper bound while still eventually giving up if something is fundamentally broken.
• Why check health first, then systemctl? The health check (/v1/models) is the definitive signal—it tells us the service is actually serving requests. The systemctl check is a fallback for early failure detection. By checking health first, the script minimizes latency in the happy path. By checking systemctl only when health fails, it avoids unnecessary SSH calls.
• Why report every 6 iterations (90 seconds)? Reporting every iteration would produce 38 lines of "loading..." output before READY. Every 6 iterations provides a reasonable progress cadence—about once per minute and a half—without overwhelming the output.
• Why include failure diagnostics? The journalctl command in the failure branch retrieves the last 10 log lines matching Error or FAILED. This shows the assistant has learned from previous debugging sessions that log context is essential for diagnosing startup failures.

Broader Significance

This message sits at a pivot point in the benchmarking campaign. The EP8 and EP4 results have shown that expert parallelism is the winning strategy on PCIe Blackwell. But the assistant needs the TP8 baseline to complete the picture—to quantify exactly how much better EP is, and to understand whether the advantage comes from reduced communication or some other factor.

The 570-second wait is the first data point in that comparison. It tells us, before any tokens are generated, that TP8 is struggling with the PCIe interconnect. The benchmark results that follow will confirm this diagnosis: TP8's peak throughput of ~1291 tok/s is 16% lower than EP8's ~1538 tok/s, and its single-request throughput of 97.9 tok/s, while higher than EP8's 65 tok/s (thanks to CUDA graphs), cannot compensate at high concurrency.

The message also demonstrates a methodological virtue: the assistant does not assume the service will start quickly or correctly. It builds a robust polling loop that handles success, failure, and the long middle ground of "still loading." This attention to operational detail is what makes the benchmarking campaign reliable—every result is built on a confirmed stable service state.

In the end, this message is about patience. Nine and a half minutes of waiting for a service to start, just to run a benchmark that will confirm what the wait already suggested. The assistant could have skipped the TP8 baseline, declared EP the winner, and moved on. But the scientific method demands the counterfactual. The 570-second wait is the price of that rigor.