The 135-Second Wait: A Case Study in ML Service Readiness Polling

Introduction

In the middle of an intense optimization session for deploying the Kimi K2.6 model with speculative decoding on 8× RTX PRO 6000 Blackwell GPUs, there is a message that appears, at first glance, to be nothing more than a simple bash loop. The assistant writes:

for i in $(seq 1 60); do
    sleep 15
    st=$(ssh -o ConnectTimeout=5 root@10.1.2.200 "systemctl is-active sglang-k26-ep8-tuned.service" 2>&1)
    if [ "$st" = "failed" ]; then
        echo "[$((i*15))s] FAILED"
        ssh -o ConnectTimeout=5 root@10.1.2.200 "journalctl -u sglang-k26-ep8-tuned.service --no-pager -n 15 | grep -E 'Error|error|FAILED|OOM' | tail -5" 2>&1
        break
    fi
    health=$(curl -s --max-time 5 "http://10.1.2.200:30001/v1/models" 2>/dev/null)
    if echo "$health" | grep -q '"id"' 2>/dev/null; then
        echo "[$((i*15))s] READY"
        break
    fi
    if [ $((i % 6)) -eq 0 ]; then echo "[$((i*15))s] loading..."; fi
done
[90s] loading...
[135s] READY

This message, <msg id=11526>, is the 26th message in a segment spanning 64 segments of a sprawling coding session. It is a bridge — a narrow, functional passage between two much more glamorous activities: deploying a tuned inference service and benchmarking its performance. Yet within this seemingly mundane polling loop lies a wealth of engineering judgment, implicit assumptions, and practical wisdom about deploying large language models in production environments. This article examines this single message in depth: why it was written, the decisions embedded in its design, the assumptions it makes, and what it reveals about the broader challenge of ML service deployment.

The Context: Why This Message Exists

To understand why this polling loop was necessary, we must trace the conversation that produced it. The assistant and user had been engaged in an intensive optimization campaign for Kimi K2.6, a 590-billion-parameter Mixture-of-Experts model, on a machine with 8× RTX PRO 6000 Blackwell GPUs connected via PCIe. The central challenge was finding the optimal parallelism strategy for this hardware configuration.

Earlier in the session, the assistant had systematically benchmarked four parallelism strategies: TP8 (pure tensor parallelism), PP8 (pipeline parallelism), EP8 (expert parallelism), and EP4 (expert parallelism with TP2 groups). EP8 emerged as the clear winner, achieving 65 tok/s at single-request concurrency — 2.5× faster than TP8's 26 tok/s — and scaling to approximately 961 tok/s at peak concurrency. The reason was architectural: EP8 eliminated the expensive AllReduce communication across PCIe for the MoE layers that dominate the model's computation, replacing it with more efficient All-to-All token dispatch.

The user then asked the assistant to theorize what would be the best setup for maximum throughput ([msg 11522]). The assistant produced a detailed analysis of the tradeoffs ([msg 11523]), concluding that EP8 was already near-optimal but could potentially be improved with higher max_running_requests and num_continuous_decode_steps — parameters that batch multiple decode steps together, reducing Python interpreter overhead and amortizing scheduling costs. The user responded with "Try all those things" ([msg 11524]).

In the immediately preceding message ([msg 11525]), the assistant acted on this directive. It stopped the existing EP8 service, created a new systemd unit file sglang-k26-ep8-tuned.service with --max-running-requests 256 and --num-continuous-decode-steps 8, and started the service. The service started successfully — systemctl start returned immediately — but this only meant the process had been launched, not that the model had finished loading. And loading a 590-billion-parameter model from disk into GPU memory across 8 devices is not instantaneous.

This is the precise moment that produces the subject message. The assistant cannot proceed to benchmarking until the service is actually ready to accept requests. It needs a robust way to wait.

Anatomy of a Polling Loop

The script the assistant writes is a classic service readiness poller, but its design reveals several deliberate engineering choices.

The polling interval of 15 seconds represents a reasonable compromise. Model loading for a 590 GB model on PCIe-connected GPUs takes on the order of minutes, not seconds. Polling every second would add unnecessary noise and log spam. Polling every 30 seconds might miss a failure for too long. Fifteen seconds keeps the loop responsive enough to detect readiness within one polling window while being gentle on the system.

The dual failure-detection strategy is particularly well-considered. The script checks two independent signals: the systemd service state and the HTTP health endpoint. Checking systemctl is-active first allows early detection of catastrophic failures — crashes, OOM errors, assertion failures — without waiting for a network timeout. If the service has failed, the script immediately fetches relevant log lines via journalctl, filtering for Error|error|FAILED|OOM. This diagnostic step is crucial: when a 590 GB model OOMs on GPU, you want to know immediately, not after 15 minutes of fruitless waiting.

The health check uses the OpenAI-compatible /v1/models endpoint, which SGLang exposes. The check looks for the presence of "id" in the JSON response — a heuristic that confirms the endpoint is returning a valid model listing rather than an error page or empty response. The curl --max-time 5 timeout prevents a hung request from blocking the loop.

The progress indicator — printing "loading..." every 6 iterations (90 seconds) — is a small but important touch. Without it, a long wait would produce only silence, making it impossible to distinguish between "still loading" and "the script hung." This is the kind of detail that separates robust automation from fragile scripts.

The overall timeout of 60 iterations × 15 seconds = 900 seconds (15 minutes) is generous. If the service hasn't loaded in 15 minutes, something is likely wrong — either the model is too large for the available memory, there's a driver issue, or the process is stuck in an infinite loop. The timeout prevents the benchmark pipeline from hanging indefinitely.

Assumptions Embedded in the Design

Every engineering artifact encodes assumptions about the world it operates in. This polling loop is no exception.

Assumption 1: The service will either become ready or fail within 15 minutes. This is reasonable for a model of this size on modern hardware, but it's not guaranteed. Network filesystem latency, driver compilation (Triton JIT), or CUDA graph capture could all extend loading time. If the service takes 16 minutes, the script would exit without declaring READY, and the benchmark would never run.

Assumption 2: systemctl is-active returning "failed" is a reliable failure signal. Systemd has multiple states: active, reloading, inactive, failed, activating, deactivating. The script only checks for "failed." If the service enters "activating" and stays there (e.g., because a pre-exec hook hangs), the script would fall through to the health check, which would fail, and it would continue polling indefinitely until the 15-minute timeout. This is a potential blind spot.

Assumption 3: The /v1/models endpoint with "id" in the response is a reliable readiness signal. This is generally true for SGLang, but there's a subtle race condition. The previous service (sglang-k26-ep8.service) was stopped moments before the new one started. Both used port 30001. If the old process's socket linger time overlaps with the new process's startup, the health check might briefly succeed against the dying old process, causing the script to declare READY prematurely. The assistant mitigates this by checking the new service's systemd state first, but the race window is not entirely closed.

Assumption 4: The model will load successfully with the new parameters. The assistant is implicitly assuming that --max-running-requests 256 and --num-continuous-decode-steps 8 won't cause memory issues during loading. This is a reasonable assumption — these parameters affect the scheduler and decode loop, not model weight loading — but it's not tested until this moment.

What the Result Reveals

The output tells us something important: [90s] loading... followed by [135s] READY. The service took approximately 2 minutes and 15 seconds to load the Kimi K2.6 model across 8 GPUs. This is a valuable data point for the deployment pipeline. Future iterations can use this as a baseline: if loading takes significantly longer, something may be wrong.

The fact that the service didn't fail is also significant. The tuned parameters didn't cause any loading-time errors. The EP8 configuration with max_running_requests=256 and continuous_decode_steps=8 is viable. This unblocks the next phase: benchmarking.

The Bridge to Benchmarking

The message that follows this one ([msg 11527]) is the benchmark itself — a comprehensive Python script that tests the tuned service across concurrency levels from 1 to 512, measuring throughput and latency. The benchmark would have failed entirely if the service weren't ready. The polling loop in <msg id=11526> is the gatekeeper that makes the benchmark possible.

This pattern — deploy, poll for readiness, benchmark — is fundamental to ML infrastructure work. It appears dozens of times across this coding session, each time with slight variations. Sometimes the service fails and the assistant must debug. Sometimes it loads faster than expected. Sometimes the health check reveals a subtle issue like the wrong model being loaded. The polling loop is the connective tissue between deployment actions and evaluation actions.

Broader Lessons

The subject message, for all its apparent simplicity, embodies several principles of robust ML deployment automation:

  1. Always check readiness before proceeding. A service that has started is not necessarily a service that is ready. Model loading, JIT compilation, and weight initialization all take time and can fail independently of the initial launch.
  2. Check multiple signals. Systemd state and HTTP health are independent indicators. Using both provides redundancy and faster failure detection.
  3. Include diagnostics in the failure path. When the service fails, the script automatically fetches relevant log lines. This saves the engineer from having to manually inspect logs after a failed deployment.
  4. Provide progress feedback during long waits. The periodic "loading..." messages transform an opaque silence into a visible heartbeat, confirming the script is still running and making progress.
  5. Set generous but bounded timeouts. Fifteen minutes is long enough for legitimate loading but short enough to prevent runaway hangs. In the broader narrative of this coding session, <msg id=11526> is a minor character — a utility player rather than a star. But it performs its role flawlessly. The service becomes ready at 135 seconds, the benchmark proceeds, and the assistant gathers the data needed to evaluate whether the tuning was effective. Without this humble polling loop, the entire optimization pipeline would stall at the deployment step. The message is a reminder that in complex engineering systems, the boring parts matter just as much as the exciting ones. A polling loop doesn't generate breakthrough performance numbers or implement novel algorithms, but it creates the conditions under which those things can happen reliably. It is the infrastructure beneath the infrastructure, the wait that enables the work.