The Waiting Game: A Server Readiness Poll and the Assumptions That Unraveled It

ssh root@10.1.230.174 'for i in $(seq 1 90); do if curl -s http://localhost:30000/v1/models 2>/dev/null | grep -q qwen; then echo "SERVER READY after ${i}0s"; exit 0; fi; echo "Waiting... ${i}0s"; sleep 10; done; echo "TIMEOUT"; journalctl -u sglang-qwen --no-pager -n 80'

At first glance, this is one of the most mundane lines in any infrastructure engineer's playbook: a polling loop that waits for a server to become ready. It curls a health endpoint, greps for a known string, and either reports success or times out after 15 minutes. Yet in the context of this opencode session — a high-stakes deployment of a 125-billion-parameter language model across four NVIDIA Blackwell GPUs — this single command becomes a dramatic fulcrum. The server never comes up. The user asks "crashed?" in the very next message ([msg 6120]). And what follows is a cascade of diagnostics, error log analysis, and configuration fixes that reveal deep assumptions about how SGLang's speculative decoding interacts with hybrid architectures.

The Context: A Model Swap Under Pressure

To understand why this message was written, we need to trace the events of the preceding hour. The assistant had just completed a major reconfiguration of the Proxmox GPU topology, splitting 8× RTX PRO 6000 Blackwell GPUs so that 4 were bound to the nvidia driver for an LXC container running SGLang, while the other 4 were moved to vfio-pci for SEV-SNP VM passthrough ([msg 6084]). The previous model — Qwen3.5-397B-A17B-NVFP4 — had been deemed "very low quality" by the user and was summarily deleted from /data along with its CUDA installer artifacts ([msg 6095]). The user then directed the assistant to deploy Qwen3.5-122B-A10B in FP16 with TP=4, tool calling, thinking, and MTP (Multi-Token Prediction) — the same speculative decoding setup used previously ([msg 6100]).

The assistant researched the model architecture via Hugging Face ([msg 6102]), discovering it was a 125B-parameter hybrid GDN (Gated Dense Network) with 48 layers, 256 experts (8 routed + 1 shared), and a single MTP hidden layer. The download completed rapidly — 234 GB in about 5 minutes ([msg 6114]). The assistant then crafted a new systemd service file, adapting the old 397B NVFP4 configuration by removing FP4-specific flags (--quantization modelopt_fp4, --moe-runner-backend flashinfer_cutlass, --fp4-gemm-backend flashinfer_cudnn) and adding the MTP flags: --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 ([msg 6112]). The old server was killed, the new service file was copied into place, and systemctl start sglang-qwen was issued ([msg 6118]).

The Polling Loop: What It Assumes

The command in message 6119 embodies several assumptions, each of which turned out to be fragile:

Assumption 1: The server will start within 15 minutes. The loop iterates 90 times with 10-second sleeps, giving the server a 15-minute window to become healthy. For a 125B model being loaded across 4 GPUs with CUDA kernel compilation, model weight loading, and distributed initialization, this is actually a generous window. The 397B NVFP4 model had loaded successfully in similar timeframes. This assumption was reasonable on its face.

Assumption 2: The /v1/models endpoint is the correct health check. The OpenAI-compatible models endpoint is the standard SGLang readiness indicator. If the server has loaded the model and is accepting requests, this endpoint returns a JSON payload containing the model name. Grepping for "qwen" is a lightweight way to confirm the correct model is loaded.

Assumption 3: The service configuration is correct. The assistant had already adapted the service file for the new model, removing FP4-specific flags and adding MTP flags. The assumption was that the same SGLang build that served the 397B NVFP4 model would serve the 122B BF16 model with only these parameter changes.

Assumption 4: The model will load without errors. The 122B BF16 model is ~250 GB, fitting comfortably in 4× 96 GB GPUs (384 GB total). The KV cache dtype was set to BF16 (via --kv-cache-dtype bf16), matching the model's native precision. No obvious resource constraints should have prevented loading.

What Actually Happened: The Server Never Responds

The polling loop runs. It prints "Waiting... 10s", "Waiting... 20s", and so on, all the way to "Waiting... 900s". Then it prints "TIMEOUT" and dumps the last 80 lines of the systemd journal. The user, watching the deployment, sees no server response and asks the natural question: "crashed?" ([msg 6120]).

The assistant then investigates and discovers the server did crash — repeatedly. The journal shows a TypeError during argument parsing:

Mar 09 11:58:19 llm-two sglang-qwen[2670]:   File "/root/sglang-main/python/sglang/srt/server_args.py", line 5458, in from_cli_args
Mar 09 11:58:19 llm-two sglang-qwen[2670]:     return cls(**{attr: getattr(args, attr) for attr in attrs})
Mar 09 11:58:19 llm-two sglang-qwen[2670]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...

The root cause, diagnosed in the following message ([msg 6123]), is that MTP with the hybrid GDN model requires two additional configuration parameters: --mamba-scheduler-strategy extra_buffer and the environment variable SGLANG_ENABLE_SPEC_V2=1. These were present in the old Kimi-K2.5 service configuration but were omitted when the assistant adapted the 397B NVFP4 service file for the 122B model.

The Mistake: A Missing Configuration Detail

The assistant's error was one of incomplete knowledge transfer. The 397B NVFP4 model did not use MTP — it was a pure FP4-quantized model served without speculative decoding. The service file for that model had no MTP flags at all. When the assistant added MTP flags for the 122B model, it correctly included --speculative-algo NEXTN and related parameters, but it did not include the infrastructure flags that SGLang's speculative decoding v2 path requires for hybrid architectures.

This is a subtle but critical distinction. The --speculative-algo NEXTN flag tells SGLang which speculative decoding algorithm to use. But the --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1 flags tell SGLang how to implement it — specifically, that the model uses a hybrid architecture (linear attention layers interspersed with full attention layers) and needs the extra buffer scheduling strategy to handle the Mamba-like state passing. Without these, the server fails during argument validation, never even attempting to load the model.

The assistant's assumption that "the same SGLang build that served the 397B will serve the 122B with just parameter changes" was wrong because the parameter changes were incomplete. The 397B service was a poor template for the 122B service because the two models used fundamentally different serving configurations — one was a quantized dense model without speculation, the other was a native-precision hybrid MoE model with MTP.

Input Knowledge Required

To fully understand this message, a reader needs:

  1. SGLang deployment conventions: Knowing that /v1/models is the standard OpenAI-compatible health endpoint, and that SGLang servers typically take 5–15 minutes to load large models.
  2. The model swap context: The 122B BF16 model replaced the 397B NVFP4 model, and the service file was adapted from the old one. The MTP flags were added but the spec v2 infrastructure flags were not.
  3. The GPU topology split: The server is running on 4 of 8 GPUs (NUMA 0), with the other 4 reserved for VM passthrough. TP=4 means tensor parallelism across these 4 GPUs.
  4. The hybrid GDN architecture: Qwen3.5 models use a mix of linear attention (Mamba-like) and full attention layers. This hybrid design requires special handling in SGLang's scheduler, which is what the missing flags provide.
  5. Systemd journal access: The journalctl -u sglang-qwen --no-pager -n 80 fallback in the timeout case is a deliberate design choice — it dumps the service's logs so the assistant can diagnose why the server failed to start.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. A definitive readiness verdict: Either "SERVER READY after Xs" or "TIMEOUT" with diagnostic logs. The timeout case is what actually occurs, triggering the diagnostic chain.
  2. A timing baseline: The loop structure (90 iterations × 10 seconds = 15 minutes) implicitly documents the expected startup time for this model on this hardware. When the server doesn't start within this window, something is wrong.
  3. A diagnostic trigger for the user: The user's "crashed?" query ([msg 6120]) confirms they were watching the deployment and expected a faster response. This social signal — a user asking if something crashed — is itself valuable context that shapes the assistant's next actions.

The Thinking Process Visible in the Command

The command's structure reveals the assistant's reasoning:

The Broader Lesson

This message, in isolation, is a simple polling loop. But in the context of the full session, it represents a moment of transition — from the known (the 397B NVFP4 model that was already working) to the unknown (the 122B BF16 model with MTP). The assistant's assumptions about configuration compatibility were reasonable but incomplete. The polling loop faithfully reports the failure, and the diagnostic chain that follows — examining the journal, identifying the missing flags, fixing the service file, and restarting — transforms this "failure" into a learning moment about SGLang's speculative decoding v2 requirements for hybrid architectures.

The command itself is not the story. The story is what it reveals about the gap between "this should work" and "this actually works" in complex ML infrastructure deployments. Every assumption embedded in that polling loop — about startup time, about configuration correctness, about model compatibility — was tested and found incomplete. And that is precisely why infrastructure engineers write polling loops with timeouts and diagnostic fallbacks: not because they expect failure, but because they know that when failure happens, the evidence is everything.