The Long Wait: What a Polling Loop Reveals About the Gap Between "Started" and "Ready" in Large-Model Serving

The Message

[assistant] ## Agent Reasoning
The service just started and the model loading with JIT compilation and CUDA graph capture will take around 8-10 minutes, so I need to poll the health endpoint periodically until it's ready to serve requests.
Service is `active` and loading the 1T model (cold start ~8–10 min: 548 GB load + JIT + graph capture). Let me poll until it's serving:
[bash] for i in $(seq 1 4); do sleep 28; r=$(timeout 12 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'curl -s --max-time 8 http://127.0.0.1:30001/v1/completions -H "Content-Type: application/json" -d "{\"model\":\"/root/models/Kimi-K2.6\",\"prompt\":\"hi\",\"max_tokens\":3,\"temperature\":0}" 2>/dev/null' 2>/dev/null); if echo "$r" | grep -q choices; then echo "READY"; break; else echo "  loading (~$((i*28))s)..."; fi; done
  loading (~28s)...
  loading (~56s)...
  loading (~84s)...
  loading (~112s)...

At first glance, message [msg 12093] appears to be a mundane operational step: start a service, wait for it to become ready, report the result. But this message sits at a critical inflection point in a much larger narrative — the pivot between building a custom inference engine and keeping production running on an existing stack. The polling loop, the cold-start estimate, and the eventual timeout all encode deep knowledge about the realities of large-scale ML serving that are invisible to the casual reader.

Context: The Fork in the Road

To understand why this message exists, we need to trace back through the conversation. The user and assistant had been building a native C/C++/CUDA DDTree inference engine for Kimi K2.6 — a 1-trillion-parameter Mixture-of-Experts model — across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The native engine, called kdtree-engine, had reached an impressive milestone: it could run tiny models with FP32 and INT4 W4A16 group-quantized weights, executing the complete DDTree speculative decode loop with greedy-exact correctness validated against numpy golden references. The novel kernels — tree builder, tree-verify MLA-absorb attention, and tree accept — were all working and bit-exact.

But the native engine was not yet a server. It couldn't load the actual 548 GB of K2.6 weights, it had no tensor parallelism across GPUs, and it used naive GEMM kernels rather than the optimized Marlin INT4 kernel. It was a validated prototype, not a production inference system.

When the user asked at [msg 12088] whether they could move off SGLang and onto their own engine, the assistant gave a brutally honest assessment: the native engine was weeks away from serving real K2.6, with eight remaining phases of integration work. The user's response at [msg 12091] was pragmatic: "restart sglang for now." This single sentence represents a strategic decision — defer the custom engine ambition and restore the production service that actually works today.

The Reasoning: Why Poll at All?

The assistant's reasoning in [msg 12093] reveals a sophisticated understanding of the cold-start process. Starting a systemd unit (systemctl start sglang-k26-ddtree) does not mean the service is ready to accept requests. Between the systemd unit becoming active and the /v1/completions endpoint responding, there is a multi-stage initialization pipeline:

  1. Weight loading: 548 GB of INT4-quantized model weights must be read from disk and distributed across eight GPUs. At PCIe Gen5 speeds (~32 GB/s per lane), this alone takes tens of seconds.
  2. JIT compilation: SGLang compiles CUDA kernels at load time, optimized for the specific model architecture and hardware. This includes flash attention kernels, MoE routing kernels, and the DDTree-specific kernels.
  3. CUDA graph capture: For maximum throughput, SGLang captures CUDA graphs — pre-compiled sequences of GPU operations that eliminate kernel launch overhead. Graph capture requires a dry run of the model's forward pass, which itself requires the model to be loaded. The assistant estimates 8–10 minutes for this process, which is consistent with the experience of loading a 1T-parameter model on high-end hardware. The polling strategy — four iterations of 28-second sleeps, with a 12-second SSH timeout and 8-second curl timeout per attempt — is designed to be robust against transient failures while providing regular status updates to the user.

Assumptions Embedded in the Polling Loop

The message makes several assumptions that are worth examining:

Assumption 1: The service will become ready within ~2 minutes of polling. The loop only runs 4 iterations (112 seconds of wall time plus overhead). This is actually inconsistent with the assistant's own estimate of 8–10 minutes. If the cold start truly takes 8–10 minutes, the polling loop would need 17–21 iterations to cover the full window. The assistant appears to be running a quick check to see if the service happens to be ready sooner, rather than committing to a full wait. This is a practical compromise — the assistant can report progress and return control to the user rather than blocking indefinitely.

Assumption 2: The health endpoint is /v1/completions with a trivial prompt. This is a functional test, not just a health check. By sending an actual completion request ({"model":"/root/models/Kimi-K2.6","prompt":"hi","max_tokens":3,"temperature":0}), the assistant verifies that the full inference pipeline is operational — not just that the HTTP server is listening, but that the model is loaded, kernels are compiled, and the forward pass works. This is more robust than a simple /health ping, which might return 200 before the model is ready.

Assumption 3: SSH and network connectivity remain stable. The polling is done remotely via SSH to the CT200 host (10.1.230.171). Each iteration involves an SSH connection, a curl command inside the host, and parsing the result. The timeout 12 on SSH and --max-time 8 on curl provide defense against network hiccups, but the overall approach assumes the remote host remains reachable throughout the loading process.

Assumption 4: The model path and configuration are correct. The curl payload references /root/models/Kimi-K2.6 as the model path. This assumes the model files are present, correctly formatted, and compatible with the SGLang version running. Any mismatch — wrong path, corrupted weights, version incompatibility — would cause the service to fail silently or return errors that the simple grep -q choices check wouldn't catch.

What Actually Happened

The output shows four loading messages at 28-second intervals, ending at ~112 seconds with no "READY" signal. The service was still loading. This is entirely expected given the 8–10 minute estimate, but it means the assistant's polling loop terminated before the service was ready. The message ends ambiguously — the loop completed without success, but the assistant doesn't explicitly state that the service is still loading or that further polling is needed.

This creates an interesting narrative gap. The user, reading this output, might conclude that something went wrong (the service never became ready) or that the assistant gave up. In reality, the assistant simply ran a bounded number of polls as a quick check and would need to continue polling or report the status differently in a follow-up message.

Input Knowledge Required

To fully understand this message, a reader needs:

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Confirmation of cold-start duration: After ~112 seconds, the service is still loading. This validates the 8–10 minute estimate and sets expectations for how long the user must wait before the service is usable.
  2. Functional validation of the service startup: The fact that the curl request doesn't return an error (connection refused, timeout) suggests the HTTP server is at least listening on port 30001, even if the model isn't loaded yet.
  3. A template for future health checks: The polling pattern — SSH + curl + grep for "choices" — is reusable for any future service verification.
  4. Documentation of the model configuration: The exact model path, API endpoint, and request format are captured, which is useful for debugging and for any client code that needs to interact with the service.

The Thinking Process: What the Reasoning Reveals

The assistant's reasoning section is remarkably concise but packs significant operational wisdom:

"The service just started and the model loading with JIT compilation and CUDA graph capture will take around 8-10 minutes, so I need to poll the health endpoint periodically until it's ready to serve requests."

This single sentence reveals that the assistant understands the multi-phase nature of model initialization. It distinguishes between "the service started" (systemd says active) and "the service is ready to serve requests" (the model is loaded and kernels are compiled). This is a distinction that less experienced practitioners often miss, leading to confusing failures when they try to send requests to a service that appears to be running but isn't actually ready.

The assistant also correctly identifies the two main components of startup latency beyond weight loading: JIT compilation and CUDA graph capture. These are not obvious to someone unfamiliar with SGLang's architecture. JIT compilation means that CUDA kernels are compiled at runtime rather than being pre-compiled, which allows for hardware-specific optimizations but adds startup time. CUDA graph capture involves running the model's forward pass once to record the sequence of GPU operations, which can then be replayed with minimal overhead — but the capture itself requires the model to be loaded first.

The choice of polling interval (28 seconds) is also interesting. It's long enough to avoid hammering the service during initialization (which might slow down the loading process), but short enough to provide reasonably frequent status updates. The 12-second SSH timeout and 8-second curl max-time provide layered timeout protection, ensuring that a single hung request doesn't stall the entire polling loop.

Mistakes and Incorrect Assumptions

The most notable issue is the mismatch between the polling duration (~112 seconds) and the estimated cold-start time (8–10 minutes, or 480–600 seconds). The assistant correctly estimates the total time but then runs a polling loop that covers only about 20% of that window. This is not necessarily a mistake — the assistant may be doing a quick initial check to catch an early readiness signal, with the intention of continuing to poll in a subsequent message. But the message as presented ends without a clear plan for what happens next.

A more subtle issue is the use of a full completion request as a health check. While this provides the most thorough verification, it also means that a failure could be caused by many different things: the model not being loaded, a kernel compilation error, a CUDA graph capture failure, or even a problem with the specific prompt. A simpler health endpoint (if SGLang provides one) might give clearer diagnostic information.

The assistant also assumes that the model path /root/models/Kimi-K2.6 is correct and that the model files are in the expected format. If the model was moved, renamed, or corrupted since the last time the service was started, the service would fail to load, and the polling loop would never see a successful response. The assistant's check would detect this (no "choices" in the response), but wouldn't provide any diagnostic information about why the service isn't ready.

The Broader Significance

This message, for all its apparent simplicity, captures a fundamental tension in applied ML engineering: the gap between building new capabilities and keeping existing systems running. The native engine development had been progressing rapidly — the assistant had just validated INT4 W4A16 group-quantized MoE on the PRO 6000 hardware, a significant milestone. But the user's practical need was for a working inference service, not a prototype. The decision to "restart sglang for now" represents a prioritization of reliability over novelty.

The polling loop is the operational manifestation of that decision. It's unglamorous work — waiting for a service to start, checking repeatedly, reporting progress — but it's the kind of work that keeps production systems running. The assistant's willingness to do this work, and to do it thoughtfully (with appropriate timeouts, meaningful health checks, and clear status reporting), is what makes the difference between a research prototype and a production service.

In the end, the service would eventually become ready, and the user would have their inference endpoint back. But the path to that readiness — the 8–10 minutes of loading, the JIT compilation, the graph capture, the patient polling — is invisible to anyone who just sees the final "READY" message. Message [msg 12093] pulls back the curtain on that invisible work, showing the operational reality that underlies every production ML service.