The 90-Second Wait: How a Simple Health Check Confirmed a Breakthrough in Expert Parallelism

[assistant] [bash] ssh root@10.1.230.174 "for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do resp=\$(curl -s -o /dev/null -w '%{http_code}' http://localhost:8000/health 2>/dev/null); if [ \"\$resp\" = '200' ]; then echo \"READY after ~\$((i*15))s\"; exit 0; fi; alive=\$(ps -p 233186 -o pid= 2>/dev/null); if [ -z \"\$alive\" ]; then echo 'PROCESS DIED'; grep -E 'RuntimeError|ValueError|OOM|out of memory' /tmp/vllm_minimax_tp8ep.log | grep -v 'repo_utils\|gpt_oss_triton' | tail -5; exit 1; fi; echo \"[\$(date +%H:%M:%S)] waiting...\"; sleep 15; done; echo 'TIMEOUT'"
[00:58:14] waiting...
[00:58:29] waiting...
[00:58:44] waiting...
[00:58:59] waiting...
[00:59:14] waiting...
READY after ~90s

On its surface, message 2328 is unremarkable: a shell script polling an HTTP endpoint in a loop, five "waiting..." lines, and then "READY after ~90s." It is the kind of log line that scrolls past unnoticed in a busy terminal. But in the context of the broader coding session — a multi-day odyssey of deploying trillion-parameter language models across eight Blackwell GPUs — this 90-second window represents a pivotal breakthrough. This message is the moment a carefully reasoned hypothesis about expert parallelism was tested and confirmed, unlocking nearly 4,000 tokens per second of inference throughput. The wait was short; the implications were enormous.

The Context: A Cascade of Failures and Discoveries

To understand why this simple health check matters, we must trace the reasoning that led to it. The session had been wrestling with the MiniMax-M2.5 model, a 230-billion-parameter FP8 mixture-of-experts (MoE) architecture. Earlier attempts had run into a wall: tensor parallelism (TP) of 8 failed because the model's FP8 quantization used a block size of [128, 128], and when the expert intermediate dimension (1536) was divided by TP=8, the resulting shard size of 192 was not divisible by 128 ([msg 2303]). This triggered a hard error in vLLM's quantization validation code.

The user then asked about alternative configurations: "can we do tp/ep or tp6?" ([msg 2303]). This question set off an intensive investigation. The assistant systematically checked whether TP=6 was viable, discovering that the model's 8 key-value heads were not divisible by 6 ([msg 2309]). It explored the vLLM codebase for expert parallelism support, finding the --enable-expert-parallel flag and tracing through the fused MoE layer implementation to understand how TP and EP interact ([msg 2308], [msg 2318]).

A critical discovery emerged from reading vLLM's source code documentation in config.py: when expert parallelism is enabled, the MoE layers use EP for expert distribution and TP becomes 1 for the expert weights — they are no longer tensor-sharded ([msg 2324]). This meant the FP8 block alignment constraint that killed TP=8 without EP would simply not apply to the expert weights when EP was active. The expert intermediate dimension would be handled whole per EP rank, not sharded.

But an initial attempt with TP=2 and EP enabled failed with an out-of-memory error ([msg 2311]). The assistant discovered that in vLLM, when EP is enabled without a separate --expert-parallel-size flag, the EP group equals the TP group — so TP=2 + EP used only 2 GPUs, and the 230GB model simply could not fit into 2 × 96GB ([msg 2312]). The model needed all 8 GPUs.

This led to the key insight: TP=8 with EP enabled would use all 8 GPUs, with attention layers tensor-sharded across 8 (which worked because the attention dimensions were divisible by 8), and expert weights distributed whole across 8 GPUs (which sidestepped the FP8 alignment issue entirely). The assistant launched this configuration in message 2327, and message 2328 is the wait loop that followed.

What the Message Actually Does

The command is a polling loop with a 5-minute timeout (20 iterations × 15-second sleeps). Each iteration, it:

  1. Makes an HTTP GET request to http://localhost:8000/health, discarding the response body and capturing only the HTTP status code.
  2. If the status code is 200, it prints "READY" with an estimated elapsed time and exits successfully.
  3. If the process (PID 233186) is no longer alive, it scans the log file for error patterns (RuntimeError|ValueError|OOM|out of memory) and exits with failure.
  4. Otherwise, it prints a timestamped "waiting..." message and sleeps 15 seconds. The five "waiting..." messages at 15-second intervals, followed by "READY after ~90s," tell us the server took approximately 90 seconds from launch to become healthy. This is notably fast for a 230B model — earlier attempts with the NVFP4 Kimi-K2.5 had taken many minutes to load ([msg 2310]). The rapid load time itself was a positive signal: the model was fitting comfortably into the 8-GPU memory pool with the EP configuration.

The Reasoning and Assumptions Embedded in This Message

This message encodes several layers of reasoning and assumptions:

Assumption 1: The configuration would not crash. The assistant had reasoned that TP=8 + EP would avoid the FP8 alignment error, but this was a hypothesis based on reading vLLM source code, not a tested configuration. The wait loop was designed to detect both success (HTTP 200) and failure (process death with error log inspection). The assistant was prepared for either outcome.

Assumption 2: 90 seconds was a reasonable expectation. The loop's 5-minute timeout suggests the assistant expected loading to complete within minutes. This was informed by earlier experience: the MiniMax model had loaded in 75 seconds with TP=4 ([msg 2310]). TP=8 + EP would involve more complex distributed initialization (NCCL allgather for expert routing tables, etc.), so 90 seconds was plausible.

Assumption 3: The health endpoint would become available only after full model loading. vLLM's health endpoint returns 200 only after the model is fully loaded and the server is ready to accept requests. The assistant correctly used this as a proxy for successful initialization.

Assumption 4: The error patterns in the grep would capture any failure. The assistant filtered for RuntimeError|ValueError|OOM|out of memory, excluding noise from repo_utils and gpt_oss_triton warnings. This was a pragmatic choice — these are the standard error types vLLM raises for configuration and memory issues.

Input Knowledge Required

To understand this message fully, one needs:

  1. Knowledge of the hardware topology: 8× NVIDIA RTX PRO 6000 Blackwell GPUs (96 GB each) connected via PCIe, with NUMA node partitioning (4+4). This explains why TP=8 is feasible and why PCIe allreduce is a bottleneck.
  2. Knowledge of the MiniMax-M2.5 model architecture: 230B parameters, FP8 quantization with [128,128] block size, MoE with 256 experts and intermediate_size=1536, GQA with 48 attention heads and 8 KV heads. These numbers determine the divisibility constraints that drove the TP/EP exploration.
  3. Knowledge of vLLM's parallelism model: How TP shards attention and dense layers, how EP distributes experts, and crucially, that EP overrides TP for MoE layers (TP becomes 1 for expert weights when EP is active). This was the key insight that made TP=8+EP viable.
  4. Knowledge of the earlier failure modes: The FP8 alignment error at TP=8 without EP (192 % 128 ≠ 0), the OOM at TP=2+EP (model too large for 2 GPUs), and the KV head divisibility issue at TP=6.
  5. Familiarity with the tooling: The curl health check, the process monitoring via ps, the log scanning pattern, and the SSH remote execution context.

Output Knowledge Created

This message produced several valuable outputs:

  1. Empirical confirmation that TP=8+EP works for MiniMax-M2.5 on 8× Blackwell GPUs. This was not guaranteed — the assistant had only theoretical reasoning from reading source code. The successful load validated the hypothesis.
  2. A load time benchmark of ~90 seconds. This is a useful data point for deployment planning and for comparison with other configurations (TP=4 loaded in 75 seconds).
  3. Confirmation that the model fits in 8× 96GB with TP=8+EP. With 230GB of weights and 8 GPUs, each GPU holds approximately 28.75GB of weights (plus KV cache overhead). The successful load confirmed memory adequacy.
  4. A validated launch command pattern that could be reused for the production systemd service later in the session.

The Broader Significance

This message sits at the inflection point of the entire chunk. Before it, the session was stuck — the NVFP4 Kimi-K2.5 was bottlenecked by PCIe allreduce for its 61-layer MLA architecture, and the MiniMax model couldn't use all 8 GPUs due to the FP8 alignment constraint. After this message, the assistant would benchmark the TP=8+EP configuration and discover it achieved nearly 4,000 tokens per second at high concurrency ([chunk 18.0]), far exceeding any previous result.

The 90-second wait was the bridge between a theoretical insight about expert parallelism and a practical breakthrough in throughput. It demonstrates a pattern that recurs throughout the session: careful source code analysis to understand constraints, hypothesis formation about alternative configurations, and then empirical validation through deployment. The health check loop is the moment of truth — the point where reasoning meets reality.

Reflections on the Thinking Process

The assistant's thinking in the messages leading up to this one ([msg 2303] through [msg 2327]) reveals a systematic, methodical approach. It begins by checking divisibility constraints (TP=6 fails on KV heads). It then explores the vLLM codebase, tracing through parallel_state.py, fused_moe/layer.py, fused_moe/config.py, and quantization/utils/fp8_utils.py to understand how EP and TP interact. It discovers the documentation comment explaining that EP overrides TP for MoE layers. It tests TP=2+EP and hits OOM, learning that EP group = TP group in vLLM's current implementation. It synthesizes these findings into the hypothesis that TP=8+EP will work because: (a) attention dimensions are divisible by 8, (b) expert weights won't be TP-sharded so the FP8 alignment constraint won't fire, and (c) 8 GPUs provide enough aggregate memory.

The wait loop in message 2328 is the culmination of this reasoning chain. It is not a random test — it is a deliberate experiment designed to validate a specific hypothesis. The "READY after ~90s" output is the experimental result, and it confirms the hypothesis. This is the scientific method applied to systems engineering: observe, hypothesize, predict, test, confirm.

Conclusion

A 90-second wait for an HTTP 200 response is, in isolation, a trivial event. But in the context of deploying trillion-parameter language models on cutting-edge hardware, it represents the successful resolution of a complex constraint satisfaction problem. The assistant had to navigate divisibility rules, quantization block sizes, memory budgets, and vLLM's internal parallelism model to arrive at a configuration that worked. Message 2328 is the moment that configuration was proven viable — and it unlocked throughput numbers that would define the rest of the session.