The 70-Second Recovery: Fixing a DeepSeek V4 HiCache Crash-Loop with One Configuration Change

Introduction

In the high-stakes world of production LLM serving, a single misconfigured flag can cascade a healthy cluster into a crash-looping nightmare. This article examines message [msg 13081] from an opencode coding session, where an AI assistant deploys a critical fix to a DeepSeek-V4-Flash inference cluster running on Blackwell GPUs with prefill-decode (PD) disaggregation. The message captures a moment of recovery: a prefill server that had been crash-looping due to an incompatible HiCache parameter is brought back online, and the assistant verifies that hierarchical caching is now functioning correctly. At just over 70 seconds from restart to health-check readiness, this message represents a successful surgical intervention in a production system.

The Context: A Production Incident and a Configuration Mistake

To understand why message [msg 13081] was written, we must trace the chain of events that preceded it. The cluster had suffered a production incident where the prefill server became unresponsive under load. The assistant diagnosed the root cause in [msg 13074] as an unbounded request queue: under a load burst of approximately 15 concurrent large prompts, the single prefill server's queue had piled up to 20+ requests and ~220,000 pending tokens. Time-to-first-token ballooned to minutes, clients aborted, and in-flight KV transfers failed with KVTransferError messages.

The assistant proposed two fixes: admission control via --max-queued-requests to prevent unbounded queue growth, and HiCache (hierarchical caching) to provide prefix reuse and VRAM relief. In [msg 13077], the assistant deployed both changes simultaneously — adding --max-queued-requests 32 to both decode and prefill servers, and --hicache-size 150 to the prefill server to allocate 150 GB of host memory for hierarchical caching.

This is where the mistake occurred. As the user reported in [msg 13078], the prefill server began crash-looping immediately. The assistant investigated in [msg 13079] and discovered the error in the prefill logs:

ValueError: DeepSeek V4 HiCache currently does not support --hicache-size; use --hicache-ratio instead.

DeepSeek V4's HiCache implementation uses a different configuration paradigm than other models: instead of an absolute size in gigabytes, it requires a ratio — a multiplier relative to the device KV cache size. The assistant had used --hicache-size 150, which the DeepSeek V4 HiCache stack explicitly rejects. This is a design choice in the SGLang codebase: for models with complex, multi-pool KV cache layouts like DeepSeek V4 (which has separate pools for the compressed KV, the indexer state, and sliding-window attention), the host cache size is derived proportionally from the device pools rather than specified as an absolute value.

In [msg 13080], the assistant edited the prefill script to replace --hicache-size 150 with --hicache-ratio 2.0, using the default ratio of 2.0 (meaning the host cache would be twice the size of the device radix cache). This edit set the stage for the deployment we see in the subject message.

The Message: Deploying the Fix

Message [msg 13081] is the assistant's deployment and verification of the corrected configuration. The message consists of a single bash command executed over SSH on the remote server, followed by its output. Let us examine each component.

The command begins with an scp to copy the updated script to the server:

scp -q /tmp/opencode/serve_dsv4_prefill.sh root@10.1.230.171:/root/serve_dsv4_prefill.sh

This is followed by an SSH session that executes a multi-step script on the remote host. The first step is a syntax check:

bash -n /root/serve_dsv4_prefill.sh && echo "script OK"

The bash -n flag performs a syntax check without executing the script. This is a safety measure: before restarting a production service, the assistant verifies that the edited script contains no syntax errors. The output confirms "script OK".

Next comes the actual service restart:

systemctl restart sglang-dsv4-prefill
echo "restarting prefill with --hicache-ratio 2.0..."

The assistant then enters a polling loop that checks the health endpoint every 5 seconds, up to 150 iterations (750 seconds maximum wait):

for i in $(seq 1 150); do
  curl -s -m 3 http://127.0.0.1:30000/health >/dev/null 2>&1 && { echo "PREFILL READY ~$((i*5))s"; break; }
  sleep 5
done

This is a robust pattern: rather than assuming the service will start quickly, the assistant polls with a generous timeout. The health endpoint at port 30000 is the prefill server's HTTP interface. The -m 3 flag limits each curl attempt to 3 seconds, preventing a hung request from delaying the loop. The output shows "PREFILL READY ~70s", indicating the service came online after 14 polling iterations (14 × 5 = 70 seconds).

Finally, the assistant inspects the prefill logs to confirm HiCache is functioning correctly:

journalctl -u sglang-dsv4-prefill --no-pager -n 500 2>/dev/null | sed "s/.*bash\[[0-9]*\]: //" | grep -iE "hierarchical|host memory|host cache|HiCache|host.*pool|host.*GB|indexer hierarchical|fired up|ValueError|Not enough" | tail -12

The grep pattern is carefully constructed to catch both success indicators ("host memory", "HiCache", "fired up") and failure indicators ("ValueError", "Not enough"). The output shows successful allocations:

[2026-06-18 20:27:39 TP2] Allocating 0.57 GB host memory for V4 state pool 'deepseek_v4_c4_indexer_state' (layers=21, pages=1670, state_page_bytes=16384, layout=page_first).
[2026-06-18 20:27:39 TP1] Allocating 0.57 GB host memory for V4 state pool 'deepseek_v4_c4_indexer_state' (layers=21, pages=1670, state_page_bytes=16384, layout=page_first).

These log lines confirm that HiCache is alive: the DeepSeek V4 indexer state pool is being allocated in host memory, with 0.57 GB per tensor-parallel rank. The allocation details — 21 layers, 1670 pages, 16384 bytes per state page — reveal the internal structure of the DSA (Dense Sparse Attention) indexer's KV cache.

Decisions Made

Several decisions are embedded in this message, some explicit and some implicit.

The choice of --hicache-ratio 2.0: The assistant used the default ratio of 2.0 rather than a higher value. This was a conservative choice, prioritizing service recovery over cache capacity. The reasoning, visible in [msg 13080], acknowledges that the ratio is relative to the device radix cache pool, which is small on the PD prefill server (since prefill computes KV and transfers it to decode rather than accumulating it locally). A ratio of 2.0 would yield a modest host cache, but it would at least get the server running. The assistant planned to tune this upward later.

The polling timeout of 750 seconds: The assistant chose 150 iterations at 5-second intervals, allowing up to 12.5 minutes for the prefill server to start. This is generous but prudent: model loading, especially for a large model like DeepSeek V4 with 8 GPUs and HiCache initialization, can take several minutes. The actual startup time of ~70 seconds fell well within this window.

The decision to restart only the prefill server: Unlike the previous deployment in [msg 13077] which restarted all three services (prefill, decode, router), this message restarts only the prefill. The decode and router were already running with the admission control fix and did not need to be disturbed. This minimizes service disruption.

The log inspection strategy: The assistant greps for both success and failure indicators, demonstrating awareness that the fix might not work. The inclusion of "ValueError" and "Not enough" in the grep pattern shows defensive monitoring: if the fix had failed, the assistant would see the error immediately rather than discovering it later through user reports.

Assumptions and Mistakes

The most significant mistake visible in this chain is the original use of --hicache-size for DeepSeek V4. The assistant assumed that HiCache configuration was uniform across model architectures, when in fact DeepSeek V4 has a specialized HiCache stack that requires --hicache-ratio. This assumption is understandable: SGLang's server arguments documentation lists both --hicache-size and --hicache-ratio as general options, and the DeepSeek V4-specific restriction is enforced only at initialization time in hybrid_pool_assembler.py. The assistant could not have discovered this incompatibility without either reading the source code or encountering the runtime error.

A subtler assumption is that HiCache would provide meaningful benefit on the prefill server in a PD-disaggregated setup. The prefill server computes KV caches and transfers them to the decode server; it does not accumulate long-running generation contexts. The host cache on the prefill side primarily serves prefix caching — reusing KV computations for repeated prompt prefixes (e.g., system prompts in agentic workloads). The assistant's reasoning in [msg 13074] acknowledged this limitation but proceeded with HiCache because the user explicitly requested it.

The assistant also assumed that a ratio of 2.0 would yield approximately 300 GB of host cache (as the user requested), but the actual allocation was only ~0.57 GB per rank. This discrepancy arose because the ratio is applied to the device radix cache pool, which on the prefill server is sized for prefix caching rather than full generation contexts. The device pool is small, so 2× a small number is still small.

Input Knowledge Required

To understand this message, a reader needs knowledge of several domains:

PD Disaggregation: The prefill-decode architecture separates prompt processing (prefill) from token generation (decode) onto different GPU sets. The prefill server computes the initial KV cache and transfers it to the decode server, which then generates tokens autoregressively. This architecture is visible in the script names (serve_dsv4_prefill.sh, serve_dsv4_decode.sh) and in the assistant's reasoning about which server should host HiCache.

HiCache (Hierarchical Caching): SGLang's hierarchical caching mechanism spills KV cache pages from GPU memory to host memory (DRAM), allowing the system to retain more cached prefixes than would fit in VRAM alone. The --hicache-ratio parameter controls the size of this host cache as a proportion of the device cache.

DeepSeek V4's KV Cache Architecture: DeepSeek V4 uses Multi-Head Latent Attention (MLA) with a compressed KV representation, plus DSA (Dense Sparse Attention) with an indexer. This creates multiple KV pools — the main compressed KV pool, the indexer state pool, and the sliding-window attention pool — each with different page sizes and layouts. The HiCache log lines show these pools being allocated in host memory.

Systemd Service Management: The assistant uses systemctl restart and checks service status with systemctl is-active. The restart counter and service states provide diagnostic information about the crash-loop.

SGLang Server Arguments: The assistant must understand the difference between --hicache-size (absolute GB, unsupported for DeepSeek V4) and --hicache-ratio (proportional multiplier, required for DeepSeek V4).

Output Knowledge Created

This message produces several concrete pieces of knowledge:

HiCache is functional with --hicache-ratio 2.0: The prefill server starts successfully and allocates host memory pools for the DeepSeek V4 indexer state. The absence of ValueError or other errors confirms the configuration is valid.

Startup time is ~70 seconds: This provides a baseline for future restarts and helps set expectations for deployment rollouts.

Per-rank allocation sizes are revealed: Each tensor-parallel rank receives 0.57 GB for the indexer state pool and 2.30 GB for the main compressed KV state pool. These numbers help the assistant (and the user) understand the memory footprint of HiCache and calibrate the ratio for future tuning.

The crash-loop is definitively resolved: The restart counter (visible in [msg 13079]) showed 2 restarts during the crash-loop. After this fix, the service stays active, confirming that the --hicache-size → --hicache-ratio change was the correct intervention.

The Thinking Process

While message [msg 13081] itself contains no explicit reasoning block (it is a direct tool call with output), the thinking that produced it is visible in the preceding messages. The diagnostic chain is instructive:

1. [msg 13074]: The assistant identifies the production incident root cause (queue saturation) and proposes HiCache as a secondary fix. It checks feasibility by examining host RAM availability and grepping the SGLang codebase for HiCache compatibility with DeepSeek V4.
2. [msg 13075]: The assistant considers NUMA constraints (each server is pinned to a 240 GB NUMA node) and plans a staged rollout: admission control first, then HiCache.
3. [msg 13077]: The assistant deploys both changes simultaneously, including the problematic --hicache-size 150.
4. [msg 13079]: Upon receiving the user's crash report, the assistant immediately investigates by checking service states and journalctl logs. The grep pattern targets error-related keywords, efficiently extracting the ValueError.
5. [msg 13080]: The assistant reads the error message, understands the constraint (DeepSeek V4 HiCache requires --hicache-ratio), and edits the script. The reasoning shows careful consideration of the ratio value, including calculations of device pool size and NUMA limits. This chain demonstrates a mature debugging methodology: form a hypothesis, test it, observe the failure, read the error message carefully, understand the constraint, and apply the minimal fix. The assistant does not panic or revert all changes — it isolates the problematic flag and corrects it while preserving the admission control fix that was working correctly.

Conclusion

Message [msg 13081] captures a moment of recovery in a complex production deployment. A single misconfigured flag — --hicache-size instead of --hicache-ratio — had caused a crash-loop that could have been destabilizing. The assistant's response demonstrates several virtues of systematic operations: rapid diagnosis through log inspection, precise surgical correction rather than wholesale reversion, robust deployment with syntax checking and health polling, and defensive verification through targeted log grepping.

The message also reveals an important lesson about configuration assumptions in complex systems: parameters that work for one model architecture may fail for another, and the only way to discover these constraints is through either exhaustive code review or (more practically) careful observation of runtime errors. The assistant's willingness to read the error message — truly read it, understand it, and act on it — is the core skill on display here.

For the broader deployment, this fix restores HiCache functionality on the prefill server, enabling prefix caching for repeated prompt prefixes. The modest allocation sizes (~0.57 GB per rank for the indexer) suggest that the ratio may need tuning upward to achieve the user's 300 GB target, but that is a tuning exercise for another day. The cluster is healthy, the crash-loop is broken, and the foundation is laid for the next round of optimization.