The 80-Second Health Check: A Case Study in Production PD Disaggregation Restart Discipline

Introduction

In the complex world of production ML inference systems, few operations are as deceptively simple—and as critically important—as restarting a service. Message [msg 13622] captures a moment that, on its surface, appears mundane: an AI assistant waits for two services to become healthy before restarting a third. But beneath this routine polling loop lies a deep understanding of Prefill-Decode (PD) disaggregation architecture, the hard-won lessons of production debugging, and the disciplined operational procedures that separate stable deployments from cascading failures.

This message is the culmination of a multi-hour debugging session that traced a persistent request-wedging bug through CUDA-graph capture races, NIXL bootstrap state degradation, and finally to a race condition in the KV transfer status-update logic. The assistant had just deployed a fix—committed as 534f5bf18—that made transfer status updates terminal-sticky, preventing racing writes from resurrecting failed or successful states. Now, the critical moment had arrived: restart the services, verify the fix, and confirm that the production system could sustain clean operation under load.

The Message in Full

The assistant's reasoning and action are captured in a single message containing a bash command that polls the health endpoints of two SGLang services:

Agent Reasoning: I'll wait for both services to finish loading, then continuously check the health endpoint until it returns a 200 status before restarting the router.

Command: A bash loop polling http://127.0.0.1:30000/health (prefill) and http://127.0.0.1:30002/health (decode) every 8 seconds, up to 30 iterations, waiting for both to return HTTP 200.

The results reveal a clear sequence: for the first 56 seconds, both services return 000—meaning they are not even accepting TCP connections. At t+64s, prefill begins responding with 503 (Service Unavailable, still loading). At t+72s, prefill reaches 200 (ready), while decode still returns 503. Finally, at t+80s, both services report 200, and the assistant proceeds to restart the router.

Why This Message Matters

To understand why this seemingly trivial health-check loop is worth deep analysis, one must appreciate the context. The production system consists of three independent services: a prefill engine that processes prompt prefixes and generates KV caches, a decode engine that generates tokens using those caches, and a router that dispatches requests between them. This PD disaggregation architecture enables efficient scaling but introduces a critical dependency: the decode engine connects to the prefill engine at startup, and the router must discover both before accepting traffic.

The bug being fixed was a "PD transfer wedge"—a condition where requests would get stuck in the prefill queue with zero GPU activity, silently consuming resources while producing no output. The root cause was a race condition in the NIXL (NVIDIA Interconnect Library) transfer path: when client cancellations (AbortReq messages) arrived during active KV chunk transfers, the status-update logic could resurrect terminal states (Failed or Success) back to Transferring, causing the prefill queue to believe a request was still in flight when it was actually abandoned. The fix made status updates terminal-sticky, ensuring that once a request reached a terminal state, no racing write could undo it.

The co-restart procedure—restarting prefill first, then decode, then the router—was itself a discovery from earlier debugging. A decode-only restart against a long-running prefill would leave NIXL in a degraded bootstrap state, causing new requests to wedge immediately. The correct procedure, which the assistant follows here, is to always co-restart the PD pair.## The Reasoning Process: A Study in Operational Discipline

The assistant's reasoning reveals a sophisticated mental model of service startup dynamics. The key insight is not simply "wait for both services to be healthy"—it's understanding that the health check must be continuous and patient, with appropriate timeouts and retry logic. The assistant explicitly states: "I'll wait for both services to finish loading, then continuously check the health endpoint until it returns a 200 status before restarting the router."

This reasoning encodes several critical assumptions:

Startup is not instantaneous. The assistant budgets 30 iterations at 8-second intervals, allowing up to 240 seconds (4 minutes) for both services to become healthy. This is generous but prudent—model loading on multi-GPU systems with large language models can take significant time, especially when CUDA kernels need to be compiled and optimized.
Health must be verified independently for each service. The assistant polls both endpoints separately, not assuming that if one is healthy the other must be. This reflects the reality of distributed systems: components can fail independently, and a partial startup is worse than no startup.
The router must not be restarted until both upstream services are ready. Restarting the router prematurely would cause it to discover only partially-initialized services, potentially caching stale endpoints or entering an inconsistent routing state. The assistant's discipline here prevents a class of subtle bugs that would be difficult to diagnose later.
The health check must use HTTP status codes, not just TCP connectivity. The initial 000 responses indicate that the TCP port is not even open—the service process hasn't started listening yet. The 503 responses indicate the service is accepting connections but not yet ready to serve requests. The 200 response is the only valid signal that the service is fully initialized.

The Results: An 80-Second Startup Timeline

The actual startup sequence, captured in the command output, tells a story of gradual initialization:

t+0s to t+56s: Both services return 000. The processes are loading the model, allocating GPU memory, compiling CUDA kernels, and initializing the NIXL transport layer. No HTTP listener is active yet. This phase lasts nearly a full minute—a reminder that large-model inference services are not lightweight microservices that start in milliseconds.
t+64s: Prefill reaches 503. The HTTP listener is up, but the service is not ready. This could mean model weights are still being loaded, KV cache memory is still being allocated, or the engine is still warming up. The fact that prefill becomes ready before decode is expected—prefill has a simpler initialization path (no need to connect to another service), while decode must establish its NIXL connection to prefill.
t+72s: Prefill reaches 200 (fully ready), decode still at 503. This 8-second gap is the decode engine establishing its bootstrap connection to the prefill engine, negotiating RDMA transfer parameters, and initializing its own model state. The assistant correctly waits rather than proceeding.
t+80s: Both services report 200. The PD pair is fully initialized and the assistant can safely restart the router. This timeline is a concrete demonstration of why the co-restart procedure matters. If the assistant had restarted decode alone (as had been done in production earlier, causing the wedge), decode would have attempted to reconnect to a prefill that was still in its 000 or 503 phase, potentially entering the degraded NIXL bootstrap state that caused the original production incident.

Assumptions and Their Validity

The assistant's approach rests on several assumptions that deserve examination:

Assumption 1: Sequential health checks are sufficient. The assistant polls prefill and decode in the same loop iteration, using two separate curl commands. This assumes that a single TCP connection to each service is representative of overall health. In practice, this is reasonable—if a service can respond to a health check, it is likely capable of handling requests. However, it does not verify that the NIXL bootstrap between prefill and decode is complete. A more thorough check might involve querying the decode engine's peer status or checking that KV transfer endpoints are registered. The assistant implicitly trusts that a 200 response implies full initialization, which is a reasonable but unverified assumption.

Assumption 2: The router restart is safe after both services are healthy. Restarting the router while the PD pair is healthy assumes that the router will correctly discover and register both services. This is true if the router uses a service-discovery mechanism (e.g., connecting to prefill and decode at known addresses) rather than relying on state that was cached before the restart. The assistant has verified this earlier in the debugging session—the router reconnects to both services on startup.

Assumption 3: The fix is effective without additional validation. The assistant is deploying the fix and immediately proceeding to test it with an abort-storm. There is no intermediate step to verify that the fix was loaded correctly (e.g., checking that the Python bytecode matches expectations). This is a minor risk, but the assistant has already compiled the code remotely and confirmed the diff, so the assumption is well-supported.

Input and Output Knowledge

To fully understand this message, one needs:

Knowledge of PD disaggregation architecture: Understanding that prefill and decode are separate services with independent lifecycles, and that the router mediates between them.
Understanding of NIXL and RDMA: The KV transfer mechanism uses NVIDIA's NIXL library for GPU-to-GPU transfers, which requires a bootstrap handshake between services.
Familiarity with HTTP health-check patterns: The use of 000 (connection refused), 503 (service unavailable), and 200 (OK) as health indicators.
Context from the debugging session: The earlier discovery that decode-only restarts cause a degraded NIXL bootstrap state, leading to the co-restart procedure. The message creates new knowledge:
Empirical startup timing for this specific model and hardware: The assistant now knows that the GLM-5-NVFP4 model on 8 RTX PRO 6000 Blackwell GPUs takes approximately 64-80 seconds for both PD services to fully initialize.
Validation of the co-restart procedure: The successful dual-healthy startup confirms that the co-restart sequence works correctly.
A baseline for future restarts: Any deviation from this ~80-second startup time would be a diagnostic signal for future issues.

Broader Significance

This message exemplifies a principle that extends far beyond ML inference: in production systems, the restart procedure is as important as the code fix itself. A correct fix deployed incorrectly will not fix the problem—and may introduce new ones. The assistant's disciplined approach—waiting for independent health verification of each component, following the established co-restart sequence, and documenting the startup timeline—is the difference between a reliable fix and a rollback incident.

The 80-second health check is also a testament to the value of observability. Without health endpoints, the assistant would be guessing when services are ready. Without metrics, the earlier debugging session could not have identified the inflight-pin pattern. The entire debugging arc—from discovering the wedge, to root-causing the race condition, to deploying the fix, to verifying the restart—is built on a foundation of observable signals.

In the broader narrative of this coding session, message [msg 13622] represents the transition from debugging to validation. The fix is committed, the services are restarted, and the assistant is about to run the abort-storm test that will confirm the fix works. The health check is the gate that must be passed before validation can begin—and the assistant passes it with careful, methodical precision.