The 900-Second Vigil: When a Health Check Hides a Crash Loop

Introduction

In the course of optimizing speculative decoding for a large language model deployment, an engineer launched a server with a freshly patched feature and then waited. For 900 seconds — fifteen full minutes — a bash loop polled a health endpoint every ten seconds, printing periodic snapshots from the server log. The message at index 5505 captures this vigil in its entirety: a deceptively simple shell script that reveals far more about the state of the system than its author initially realized.

This message, dispatched by the assistant in an opencode coding session, is a health-check polling loop for an SGLang inference server running the GLM-5-NVFP4 model (via the Kimi K2.5 architecture) with EAGLE-3 speculative decoding. But what makes this message fascinating is not what it shows — it's what it doesn't show to its author. The server log excerpts embedded in the polling output tell a story of a process crash-looping silently, while the assistant remains unaware, waiting for a health check that will never return the expected text.

Context: The Dynamic Speculation Disable Feature

To understand this message, we must first understand what led to it. The session had been a marathon of ML infrastructure optimization spanning dozens of messages. The team had deployed a Kimi K2.5 model (a 32B-parameter architecture) quantized to INT4 across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using tensor parallelism. They had painstakingly tuned EAGLE-3 speculative decoding — a technique where a small "draft" model generates candidate tokens that a larger "target" model verifies in parallel — achieving a net-positive 96.1 tok/s after upgrading to CUDA 13 and enabling FlashInfer allreduce fusion.

But benchmarks revealed a troubling finding: while EAGLE-3 improved single-request latency at low concurrency, it hurt total throughput under load. The baseline server (no speculation) saturated at ~773 tok/s, while EAGLE-3 peaked at ~354 tok/s — a >2x gap. The speculative decoding overhead (running both draft and target models, managing the verify step) became a net negative when multiple requests competed for GPU memory and compute.

The natural solution: dynamically disable speculation when the server is busy, re-enabling it when load drops. The assistant wrote a patch adding a --speculative-disable-batch-threshold parameter to SGLang's EAGLEWorkerV2 class and server_args.py. After fixing a syntax error in the first patch attempt (the CLI argument was inserted inside another argument's definition, breaking the parser), the assistant verified both modules imported cleanly and launched the server.

The Message Itself

The subject message executes a bash loop that polls the server's health endpoint every 10 seconds for up to 900 seconds (90 iterations × 10 seconds):

for i in $(seq 1 90); do
  result=$(ssh root@10.1.230.174 "curl -s http://localhost:30000/health 2>/dev/null" 2>/dev/null)
  if echo "$result" | grep -q "ok\|healthy"; then
    echo "Server ready after ~$((i*10))s"
    exit 0
  fi
  if [ $((i % 15)) -eq 0 ]; then
    last_line=$(ssh root@10.1.230.174 "tail -3 /data/eagle3/synth_100k/logs/dynamic_spec_t5.log 2>/dev/null" 2>/dev/null)
    echo "[$((i*10))s] $last_line"
  fi
  sleep 10
done
echo "FAILED after 900s"

The loop has two responsibilities: check the health endpoint for a success response, and every 150 seconds (every 15th iteration), tail the last 3 lines of the server log for diagnostics. The output shows three diagnostic snapshots at 150s, 300s, and 450s — and each one shows the same pattern:

[150s] Loading safetensors checkpoint shards:  98% Completed | 63/64 [00:13<00:00, 19.74it/s]
Loading safetensors checkpoint shards: 100% Completed | 64/64 [00:14<00:00,  4.57it/s]
[300s] Loading safetensors checkpoint shards:  98% Completed | 63/64 [00:13<00:00, 19.74it/s]
Loading safetensors checkpoint shards: 100% Completed | 64/64 [00:14<00:00,  4.57it/s]
[450s] Loading safetensors checkpoint shards:  98% Completed | 63/64 [00:13<00:00, 19.74it/s]
Loading safetensors checkpoint shards:...

The Hidden Crash Loop

The log output is identical at each checkpoint — down to the timing (00:13&lt;00:00, 19.74it/s). This is the unmistakable signature of a crash loop: the server process starts, loads the model (64 safetensors shards across 8 GPUs), completes loading at ~14 seconds, then crashes immediately after initialization. The process manager (likely nohup in this case, or the shell's background process handling) restarts it, and the cycle repeats.

The crash happens after model loading but before the HTTP server starts listening on port 30000. This is why the health check never succeeds — the server never reaches the point where it can respond to requests. The health endpoint returns HTTP 200 with an empty body (as the assistant discovers in the next message, msg 5506), but the grep for "ok" or "healthy" in the body fails because the body is empty.

There is a profound irony here: the server is running at the moments the assistant checks the log, but it's running in a transient state between restart and crash. The health endpoint doesn't exist yet. The assistant's check is simultaneously too strict (looking for text in an empty body) and too lenient (not checking for HTTP status code alone, which would have shown 200).

Assumptions and Blind Spots

The assistant made several assumptions that shaped this message:

Assumption 1: The server would either start cleanly or fail permanently. The health check loop was designed for a binary outcome — ready or not ready. It didn't account for a crash-loop scenario where the server appears to be making progress (loading checkpoints) but never stabilizes. The diagnostic output at 150s intervals was meant to show progress, but instead showed repetition — a pattern the assistant didn't recognize.

Assumption 2: The health endpoint would return recognizable text. The grep for "ok\|healthy" was a reasonable heuristic, but it failed because the SGLang health endpoint at that version returns an empty 200 response. A more robust check would have tested for HTTP 200 status or checked that the response body was non-empty.

Assumption 3: The patch was applied to the correct file. The most consequential assumption was that EAGLEWorkerV2 (in eagle_worker_v2.py) was the active worker class. In reality, the server was using the non-overlap path with EAGLEWorker (v1, in eagle_worker.py), because the overlap scheduler was disabled (disable_overlap_schedule=True). The dynamic speculation disable code was patched into a file that was never imported. The assistant wouldn't discover this until msg 5510, five messages later.

Assumption 4: The crash was a startup issue rather than a runtime initialization failure. The log showed checkpoint loading completing successfully, but the crash happened afterward — likely during model initialization, CUDA graph compilation, or speculative worker setup. The assistant interpreted the repeating log as the server "still loading" rather than "loading, crashing, and reloading."

The Thinking Process

The message reveals a methodical, patient approach to server deployment. The assistant chose a 10-second polling interval — long enough to avoid hammering the server during startup, short enough to detect readiness within a reasonable time. The 900-second timeout (15 minutes) reflects the known startup time for an 8-GPU model with 64 checkpoint shards: even with fast NVMe storage, loading 32B parameters across 8 GPUs takes time.

The diagnostic snapshot every 150 seconds (15 iterations) is a deliberate choice: frequent enough to spot problems, infrequent enough to avoid log spam. The assistant is thinking about observability — ensuring that if the server fails to start, there will be evidence in the output.

But the assistant is also thinking optimistically. The log output shows "100% Completed" — a positive signal. The assistant likely reads this as "progress is being made, just give it more time." The identical timestamps across snapshots are subtle: 00:13&lt;00:00, 19.74it/s appears verbatim at 150s, 300s, and 450s. A human watching the output scroll by might miss this repetition, especially when each snapshot is separated by 2.5 minutes of silence.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. SGLang server architecture: Knowledge that SGLang loads model checkpoints in shards (64 shards for this 8-GPU deployment), that the health endpoint is only available after full initialization, and that the server uses a speculative decoding pipeline with draft and target models.
  2. The dynamic speculation disable patch: Understanding that the assistant had just modified server_args.py and eagle_worker_v2.py to add a --speculative-disable-batch-threshold parameter, and that this patch was the likely cause of the crash.
  3. Crash-loop behavior: Familiarity with the pattern where a process crashes immediately after a visible milestone (like "100% Completed"), causing the process manager to restart it, creating an infinite loop.
  4. The CUDA/GPU context: Awareness that loading 64 safetensors shards across 8 GPUs with tensor parallelism involves significant CUDA initialization, and that errors in this phase (e.g., out-of-memory, CUDA graph compilation failures) can cause silent crashes.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Evidence of a crash loop: The repeating checkpoint loading pattern at 150s, 300s, and 450s is definitive proof that the server process is restarting. This is the most important output — though the assistant doesn't immediately recognize it.
  2. Confirmation that model loading works: The checkpoint loading completes successfully (100% at ~14 seconds), ruling out corrupted model files, disk I/O issues, or insufficient GPU memory for the base model.
  3. Timing baseline: The model loads in ~14 seconds across 8 GPUs, which is useful diagnostic information for future startups.
  4. Health endpoint behavior: The empty 200 response from the health endpoint (discovered in the next message) is a finding that will inform future health check scripts.

The Aftermath

In the messages immediately following, the assistant discovers the server is actually returning HTTP 200 (msg 5506), realizes the health check was too strict (msg 5506-5507), and then makes the critical discovery: the dynamic speculation disable patch was applied to the wrong file (msg 5510). The server was using EAGLEWorker (v1, in eagle_worker.py), not EAGLEWorkerV2 (in eagle_worker_v2.py). The crash was likely caused by the speculative_disable_batch_threshold parameter being passed to a worker class that didn't recognize it, or by some other incompatibility introduced during the patching process.

This cascade of discoveries — the crash loop, the empty health response, the wrong file — all stem from the single message at index 5505. The 900-second vigil was not wasted; it produced the diagnostic data needed to understand what went wrong. But it also reveals a deeper truth about debugging distributed systems: the most important information is often hidden in the pattern of what repeats, not in what changes.

Conclusion

Message 5505 is a study in the gap between what a script measures and what a system does. The health check loop was designed to answer one question — "Is the server ready?" — but it inadvertently answered a more important one: "Is the server crash-looping?" The identical checkpoint loading timestamps at 150s, 300s, and 450s told a story that the assistant hadn't yet learned to read. In the end, the server was running — just not long enough to matter. The patch was applied to the wrong file, the health check was looking for the wrong text, and the crash loop was hiding in plain sight, repeating every 150 seconds like a stuck record.