The 135-Second Wait: How a Simple Polling Loop Marked the Turning Point for Expert Parallelism on PCIe Blackwell

The Message

for i in $(seq 1 90); do
    sleep 15
    st=$(ssh -o ConnectTimeout=5 root@10.1.2.200 "systemctl is-active sglang-k26-ep8.service" 2>&1)
    if [ "$st" = "failed" ]; then
        echo "[$((i*15))s] FAILED"
        ssh -o ConnectTimeout=5 root@10.1.2.200 "journalctl -u sglang-k26-ep8.service --no-pager -n 20 | grep -E 'Error|error|FAILED|assert|Traceback|ValueError' | tail -8" 2>&1
        break
    fi
    health=$(curl -s --max-time 5 "http://10.1.2.200:30001/v1/models" 2>/dev/null)
    if echo "$health" | grep -q '"id"' 2>/dev/null; then
        echo "[$((i*15))s] K2.6 EP8 READY!"
        break
    fi
    if [ $((i % 6)) -eq 0 ]; then echo "[$((i*15))s] loading..."; fi
done
[90s] loading...
[135s] K2.6 EP8 READY!

At first glance, message [msg 11511] appears to be nothing more than a routine service readiness check — a bash loop that polls every fifteen seconds, waiting for an SGLang inference server to finish loading a model. But this seemingly mundane polling loop sits at a critical inflection point in a much larger investigation. It marks the moment when the assistant pivoted from a disappointing pipeline-parallel configuration to a promising new parallelism strategy — Expert Parallelism (EP) — for deploying the massive Kimi K2.6 mixture-of-experts model across eight NVIDIA RTX PRO 6000 Blackwell GPUs connected over PCIe. The 135 seconds this loop waited were the calm before a storm of benchmark results that would validate a key architectural hypothesis and reshape the entire project's direction.

The Context: A Parallelism Puzzle on PCIe

To understand why this message matters, we must reconstruct the reasoning that led to it. The conversation preceding [msg 11511] was a deep investigation into the optimal parallelism strategy for running K2.6 — a 671B-parameter MoE model with 384 experts — on a system with eight RTX PRO 6000 GPUs connected exclusively via PCIe (no NVLink). This is a challenging hardware environment because PCIe bandwidth (~64 GB/s per direction per card) is dramatically slower than NVLink (~900 GB/s), making inter-GPU communication the primary bottleneck.

The assistant had already benchmarked two strategies. Tensor Parallelism (TP8) distributed every layer's computation across all eight GPUs, requiring an AllReduce operation at every layer. At single-request concurrency, TP8 achieved only ~26 tok/s — the AllReduce overhead on PCIe was crippling. Pipeline Parallelism (PP8) split the model's layers across GPUs, with each GPU owning a contiguous slice of the model. PP8 improved single-request throughput to ~36-37 tok/s by eliminating per-layer AllReduce, but suffered from pipeline bubbles at high concurrency: aggregate throughput peaked at ~396 tok/s, well below TP8's ~808 tok/s. Neither strategy was satisfactory — PP8 had poor utilization, TP8 had terrible latency.

The user's instruction was succinct: "Restart in EP8" ([msg 11507]). This was not a casual suggestion. Expert Parallelism distributes the model's experts across GPUs rather than splitting layers or replicating the full model. For an MoE model like K2.6, where each token activates only a small subset of experts (typically 8 out of 384), EP promises to replace expensive AllReduce operations with cheaper All-to-All communication — sending only the tokens that need non-local experts to the appropriate GPU, rather than broadcasting every activation across the entire cluster.

The Reasoning Behind the Message

Message [msg 11511] was written because the assistant had just deployed the EP8 service in the previous round ([msg 11510]) and needed to confirm it was operational before running benchmarks. The service startup involved:

  1. Stopping the previous PP8 service
  2. Writing a new systemd unit file with --tp-size 8 --ep-size 8 (combining tensor parallelism for the attention layers with expert parallelism for the MoE layers)
  3. Configuring environment variables for CUDA 13.0, NCCL settings optimized for PCIe (NCCL_P2P_LEVEL=5, NCCL_ALGO=Ring), and thread control (OMP_NUM_THREADS=8)
  4. Running systemctl daemon-reload and systemctl start The assistant could not simply assume the service started correctly. K2.6 is a 590 GB model (as noted elsewhere in the segment). Loading it across eight GPUs with a new parallelism configuration could fail in many ways: CUDA out-of-memory errors, NCCL initialization failures, incompatible attention backends, or assertion failures in SGLang's EP implementation. The polling loop was designed to detect all of these failure modes. The loop's design reveals careful thinking. It checks two independent signals of health: the systemd service status (which catches crashes and assertion failures) and the HTTP /v1/models endpoint (which confirms the model is fully loaded and accepting requests). The 15-second polling interval is a pragmatic choice — short enough to detect failures quickly, long enough to avoid hammering the loading process. The 90-iteration maximum (22.5 minutes total) provides a generous upper bound for loading a 590 GB model over PCIe. The --max-time 5 on the curl command prevents a hung request from stalling the loop indefinitely. And the periodic "loading..." messages every 6 iterations (90 seconds) give the human observer visibility into progress.

Assumptions and Their Risks

Several assumptions underpinned this message, and some were fragile. The assistant assumed that SGLang's EP8 implementation would work correctly with K2.6 — specifically, that the combination --tp-size 8 --ep-size 8 was valid. The help output from [msg 11509] confirmed that --ep-size was a recognized parameter, but it did not confirm that TP+EP co-existed correctly for this model architecture. The assistant also assumed that the NCCL configuration (NCCL_P2P_LEVEL=5, NCCL_IB_DISABLE=1) was appropriate for EP's All-to-All communication pattern — this was carried over from the TP8 and PP8 configurations, but EP has a fundamentally different communication topology that might benefit from different NCCL settings.

The most significant assumption was that EP8 would actually outperform the alternatives. At this point, the assistant had invested significant effort in TP8 and PP8, both of which had clear weaknesses. EP8 was the unexplored third option, and the user's intuition that "expert parallelism avoids PCIe AllReduce bottlenecks" was a hypothesis, not a proven fact. The polling loop in [msg 11511] was the gateway to testing that hypothesis — if the service failed to start, the hypothesis could not be evaluated.

The Input Knowledge Required

To fully understand this message, one must grasp several layers of technical context. First, the Mixture-of-Experts architecture: K2.6 has 384 experts, of which only 8 are activated per token. This sparsity is what makes EP attractive — each GPU holds 48 experts, and most token-expert routing stays local. Second, the difference between AllReduce (used in TP) and All-to-All (used in EP): AllReduce requires every GPU to sum its partial results with every other GPU, creating a bandwidth bottleneck proportional to the hidden dimension times the number of GPUs. All-to-All sends only the token representations that need non-local experts, which scales with the number of tokens that are "routed" to remote GPUs — a much smaller volume for sparse MoE. Third, the PCIe constraint: without NVLink, inter-GPU communication must traverse the PCIe switch fabric, which has limited bandwidth and higher latency than NVLink's direct GPU-to-GPU connections.

One must also understand the SGLang deployment architecture: the model is served via a systemd service on a remote machine (CT200, IP 10.1.2.200), the service is configured through environment variables and command-line arguments, and readiness is verified through both systemd and HTTP health checks. The polling script itself runs on the local machine (the assistant's environment), not on CT200, using SSH and curl for remote monitoring.

The Output Knowledge Created

This message produced one critical piece of knowledge: the EP8 service started successfully and was ready for benchmarking. The output [135s] K2.6 EP8 READY! confirmed that the model loaded in approximately 2 minutes and 15 seconds — a reasonable time for a 590 GB model distributed across eight GPUs. The absence of any failure messages meant that SGLang's EP implementation, the NCCL configuration, the CUDA 13.0 toolkit, and the attention backend all worked together correctly for this model.

But the true output knowledge was created in the very next message ([msg 11512]), where the assistant ran the first benchmarks. The results were dramatic:

=== K2.6 EP8 - C=1 (max=512) ===
  65.1 tok/s | 512 tok in 7.9s
  65.3 tok/s | 512 tok in 7.8s
  63.0 tok/s | 47 tok in 0.7s
  65.7 tok/s | 512 tok in 7.8s
  66.2 tok/s | 512 tok in 7.7s

65 tok/s at single-request concurrency. This was a 2.5× improvement over TP8's 26 tok/s and a 1.8× improvement over PP8's 36 tok/s. The hypothesis was validated: by eliminating AllReduce on the MoE layers, EP8 unlocked the PCIe-bound system's true potential. The aggregate throughput at high concurrency (which would be explored in subsequent messages) also showed dramatic improvements.

The Thinking Process Visible in the Message

While the message itself is a simple bash script, the thinking process is visible in its structure. The assistant anticipated multiple failure modes and built a robust polling loop that could handle each one:

Why This Message Matters

In the grand narrative of this coding session, [msg 11511] is the moment of transition between diagnosis and validation. The preceding twenty messages were a deep diagnostic dive into why PP8 was underperforming — investigating CUDA graph capture, micro-batch sizing, pipeline bubbles, and the pp_max_micro_batch_size auto-computation bug. The user's pivot to EP8 was a strategic bet that a fundamentally different parallelism strategy would solve the PCIe bottleneck. This polling loop was the gateway to testing that bet.

The 135-second wait was brief, but it represented the culmination of hours of work: installing CUDA toolkits, debugging FlashInfer compatibility, patching SGLang's attention backend, and benchmarking two failed parallelism strategies. When the loop printed "K2.6 EP8 READY!", it didn't just confirm that a service was running — it confirmed that the entire infrastructure stack, from the NVIDIA drivers to the CUDA libraries to the SGLang server to the NCCL configuration, was finally aligned to support the parallelism strategy that would deliver the best results.

The lesson for the reader is that even the most mundane operations in a complex engineering workflow — a bash loop that polls every fifteen seconds — carry the weight of the reasoning that led to them. Every design choice in that loop, from the polling interval to the error detection patterns to the heartbeat messages, reflects the assistant's understanding of the system's failure modes and its priorities for recovery. The message is simple, but the thinking behind it is not.