The Art of the Readiness Check: A Single Bash Loop in an ML Benchmarking Workflow

Introduction

In the sprawling narrative of deploying and benchmarking large language models across multi-GPU infrastructure, most of the drama unfolds in the headline moments: the breakthrough benchmark result, the discovery of a critical bug, the architectural insight that changes everything. But the day-to-day reality of ML engineering is built on a foundation of small, unglamorous automations—scripts that wait, check, retry, and report. Message [msg 11536] in this opencode session is a perfect specimen of that genre: a 15-line bash loop that polls a remote service until it becomes ready. On its surface, it is trivial. But examined closely, it reveals the operational discipline, the failure-awareness, and the methodological rigor that distinguishes systematic experimentation from ad-hoc tinkering.

This article unpacks that single message—its motivation, its design decisions, its assumptions, its blind spots, and the knowledge it both consumes and produces—to show how even the simplest automation carries the fingerprints of the larger scientific process it serves.

The Message Itself

The subject message is an assistant message containing a bash command executed against a remote host at 10.1.2.200 (an internal CT200 machine). It reads:

for i in $(seq 1 60); do
    sleep 15
    health=$(curl -s --max-time 5 "http://10.1.2.200:30001/v1/models" 2>/dev/null)
    if echo "$health" | grep -q '"id"' 2>/dev/null; then echo "[$((i*15))s] READY"; break; fi
    st=$(ssh -o ConnectTimeout=5 root@10.1.2.200 "systemctl is-active sglang-k26-ep4.service" 2>&1)
    if [ "$st" = "failed" ]; then
        echo "[$((i*15))s] FAILED"
        ssh -o ConnectTimeout=5 root@10.1.2.200 "journalctl -u sglang-k26-ep4.service --no-pager -n 10 | grep -E 'Error|FAILED' | tail -3" 2>&1
        break
    fi
    if [ $((i % 6)) -eq 0 ]; then echo "[$((i*15))s] loading..."; fi
done
[90s] loading...
[120s] READY

The output confirms that the EP4 service started successfully and was ready to serve requests after 120 seconds—consistent with the loading times observed for previous configurations.

Why This Message Was Written: The Experimental Loop

To understand the motivation behind this message, we must zoom out to the experimental context. The session had been systematically benchmarking parallelism strategies for the Kimi K2.6 model across 8× RTX PRO 6000 Blackwell GPUs connected via PCIe. The assistant had already tested:

• TP8 (pure tensor parallelism across all 8 GPUs): achieved ~98 tok/s at C=1 but plateaued at ~1291 tok/s aggregate.
• PP8 (pipeline parallelism): disappointing due to pipeline bubble overhead.
• EP8 (expert parallelism across 8 GPUs): the winner so far, with ~65 tok/s at C=1 and ~1538 tok/s peak aggregate throughput.
• EP8 with EPLB (expert load balancing): a failed experiment that cratered performance to 7.7 tok/s at C=1 and dropped peak throughput to 862 tok/s. The previous message ([msg 11535]) had just deployed the EP4 configuration—expert parallelism with 4 expert groups, each group internally using TP2 (tensor parallelism of 2). The assistant had stopped the EP8 service, written a new systemd unit file for EP4, reloaded systemd, and started the service. Now it needed to wait for the model weights to load and the HTTP server to become available before running the benchmark suite. This readiness check is thus the bridge between two experiments. It is the operational prerequisite for generating the next data point in the parallelism comparison matrix. Without it, the assistant cannot proceed to the benchmark script that will measure EP4's throughput, latency, and scaling characteristics. The message exists because the experimental method demands it: each configuration must be tested under identical conditions, and testing cannot begin until the service is fully initialized.

How Decisions Were Made: The Design of the Readiness Check

The script embodies several design decisions, each reflecting tradeoffs between speed, reliability, and information content.

Polling interval of 15 seconds. This is a pragmatic choice. Model loading for a 590 GB parameter model (the Kimi K2.6) takes on the order of 2–6 minutes depending on disk I/O and GPU memory initialization. Polling every 15 seconds means the script will detect readiness within 15 seconds of it becoming available—a resolution more than adequate for this context. Faster polling (e.g., every 1 second) would add unnecessary network traffic and log noise without providing meaningful benefit. Slower polling (e.g., every 60 seconds) would introduce noticeable delay in the experimental loop. The 15-second cadence is a Goldilocks choice.

Two-tier failure detection. The script checks two independent signals: the HTTP health endpoint (/v1/models) and the systemd service status. This dual check is important because each signal can fail independently. The HTTP endpoint might return an error even though systemd thinks the service is "active" (e.g., if the Python process is running but the model hasn't finished loading). Conversely, systemd might report "failed" while a stale HTTP process lingers. By checking both, the script can distinguish between "still loading" (HTTP not ready, systemd active) and "crashed" (systemd failed). This design reflects an understanding of the failure modes of distributed ML services.

Maximum 60 iterations (15 minutes). The upper bound of 60 iterations gives a maximum wait time of 900 seconds (15 minutes). This is generous—the model typically loads in 2–3 minutes—but provides headroom for edge cases like slow disk reads, CUDA kernel compilation, or first-time Triton autotuning. If the service hasn't started in 15 minutes, something is fundamentally wrong and the script will exit the loop without further action (the loop simply terminates without printing anything after the last iteration). This is a minor weakness: the script does not explicitly report timeout.

Progress reporting every 6 iterations (90 seconds). The if [ $((i % 6)) -eq 0 ] clause prints a "loading..." message every 90 seconds. This is a user-experience consideration: for a human watching the output, silence for 15 minutes would be unnerving. The periodic progress indicator confirms that the script is still running and the service is still loading (not crashed). It also provides a rough sense of how long loading is taking.

Error detail capture on failure. When systemd reports "failed", the script immediately SSHes into the host to grab the last 10 lines of the journal, filtered for error-related terms. This is a crucial design choice: it captures the failure reason at the moment of failure, before the journal is overwritten by subsequent restart attempts. The filtered grep (Error|FAILED) and tail limit (-n 10) ensure the output is concise and actionable—the assistant can immediately see whether the failure is an OOM, a CUDA error, a missing model file, or something else.

Assumptions Embedded in the Script

Every automation encodes assumptions about the world it operates in. This script is no exception.

The /v1/models endpoint is a reliable health indicator. The script assumes that if the OpenAI-compatible API returns a valid JSON response containing an "id" field, the service is fully ready to accept generation requests. This is generally true for SGLang, but there is a subtle race condition: the HTTP server may begin accepting connections and returning model metadata before the model weights are fully loaded and the first forward pass is possible. A request submitted immediately after the health check passes might still encounter a "model not ready" error. The script does not perform a generation-level health check (e.g., sending a short completion request), which would be more robust but also more expensive and complex.

The service name is correct and unique. The script references sglang-k26-ep4.service—a name chosen in the previous message. If the systemd unit file had a typo or the service name didn't match, the systemctl is-active check would return "inactive" or "not-found" rather than "failed", and the script would not detect the error. This is a fragile coupling: the readiness check is tightly bound to the naming convention established in the deployment step.

SSH connectivity is stable. The script uses ssh -o ConnectTimeout=5 to check systemd status and fetch journal logs. It assumes the remote host is reachable and SSH credentials are available (via key-based authentication). If the network is flaky or the SSH connection drops, the systemctl is-active call might fail with a connection timeout, producing an empty or error string that does not match "failed", causing the script to silently continue polling. The error output from the SSH command is redirected to stderr (2>&1), but the script does not check the exit code of the SSH command itself.

The model loads within 15 minutes. The 60-iteration limit assumes that any successful load will complete within 15 minutes. If the model takes longer (e.g., due to first-time Triton kernel compilation for a new GPU architecture), the script will silently exit the loop without reporting success or failure. The assistant would then see no output and might incorrectly assume the service is still loading or has hung.

Potential Blind Spots and the Race Condition

The most significant blind spot in this readiness check is the race condition between the old and new service processes. When systemd restarts a service, it sends a SIGTERM to the old process, waits for it to exit, and then starts the new process. However, the old HTTP server might continue to accept connections briefly during the shutdown sequence. If the curl health check hits the old process before it fully dies, it gets a successful response from the old service—not the new one. The script then reports "READY" prematurely, and the subsequent benchmark requests might fail or hit a stale model configuration.

This race condition was actually discovered later in the session (documented in chunk 1 of segment 64), where the assistant found that "the old process answered /v1/models briefly before being killed, while the new process was still loading weights for ~6 minutes." The readiness check in this message does not guard against this scenario—it has no mechanism to verify that the responding process is the new one. A more robust approach would be to check the process PID or the service start timestamp, but that would add complexity far beyond what a simple polling loop requires.

Input Knowledge Required to Understand This Message

To fully grasp what this message is doing and why, a reader needs:

1. Knowledge of systemd service management. The script uses systemctl is-active and journalctl, which are Linux systemd commands. Understanding that is-active returns "active", "inactive", "failed", etc., and that journalctl retrieves system logs, is essential.
2. Knowledge of SGLang's HTTP API. The health check targets /v1/models, which is an OpenAI-compatible endpoint that SGLang exposes. Knowing that this endpoint returns a JSON list of loaded models with an "id" field explains why the script greps for '"id"'.
3. Knowledge of the experimental context. The service name sglang-k26-ep4.service encodes the model (Kimi K2.6) and the parallelism strategy (EP4). Without knowing that the assistant is benchmarking parallelism strategies, the service name is opaque.
4. Knowledge of the remote infrastructure. The IP 10.1.2.200 and the model path /root/models/Kimi-K2.6 reference a specific remote machine (CT200) and model location. The script assumes the reader (or the assistant) knows what these refer to.
5. Understanding of bash scripting idioms. The for loop, seq, grep -q, $((arithmetic)), and 2>/dev/null redirections are standard bash patterns that a reader must parse to understand the script's logic.

Output Knowledge Created by This Message

This message produces two kinds of knowledge: operational and experimental.

Operational knowledge: The output confirms that the EP4 service started successfully and was ready in 120 seconds. This is a data point about deployment behavior: the model loading time is consistent across EP8 and EP4 configurations (~2 minutes), suggesting that loading time depends primarily on model size and disk I/O, not on the parallelism topology. The "loading..." message at 90 seconds provides a checkpoint: the service had been loading for 1.5 minutes without failure.

Experimental knowledge: The successful readiness check enables the next step in the experimental sequence. The assistant can now run the benchmark suite against the EP4 service, producing throughput and latency measurements that will be compared against EP8, TP8, and PP8 results. This message is thus a prerequisite for the knowledge that will be generated in the subsequent messages—the EP4 benchmark numbers that will complete the parallelism comparison matrix.

Methodological knowledge: The script itself, though not novel, reinforces a pattern for automated experimentation: deploy → wait for readiness → benchmark → record → compare. This pattern is reusable across future experiments with different models, configurations, and hardware setups.

The Thinking Process Visible in the Reasoning

While this particular message does not contain explicit reasoning traces (it is a straightforward bash command), the reasoning behind it is visible in the surrounding conversation. The assistant had just deployed EP4 after a series of experiments:

1. EP8 baseline: confirmed at ~1500 tok/s peak ([msg 11534])
2. EPLB experiment: failed, performance cratered ([msg 11531])
3. EP4 deployment: the next variable to test ([msg 11535]) The readiness check is the natural consequence of this experimental loop. The assistant does not need to think about whether to wait for the service—that is an operational necessity. The thinking is in the design of the wait: the 15-second interval, the dual failure detection, the progress reporting, the error capture. These choices reflect experience with remote service deployment and an understanding of what can go wrong. The assistant also shows awareness of time cost. The 120-second wait is not dead time—it is an investment in experimental validity. Rushing the benchmark against a partially loaded service would produce meaningless numbers. The readiness check ensures that when the benchmark runs, it measures the EP4 configuration in a steady state, comparable to the EP8 measurements taken under identical conditions.

Conclusion

Message [msg 11536] is, on its face, a simple bash loop that polls a service until it becomes ready. But in the context of the broader experimental workflow, it is a critical piece of operational infrastructure—a bridge between deployment and measurement, between configuration and validation. Its design reflects tradeoffs between speed and reliability, between simplicity and robustness. Its assumptions encode beliefs about how the service behaves during startup. Its blind spots—particularly the race condition between old and new processes—are the seeds of future debugging sessions.

In the end, this message is a testament to the fact that rigorous ML engineering is not just about the clever algorithms and the breakthrough results. It is also about the boring, necessary work of making sure the service is actually running before you point your benchmark harness at it. The art of the readiness check may not be glamorous, but it is indispensable.