A Pivotal Restart: Disabling Radix Cache for Correct Hidden State Extraction in EAGLE-3 Training

In the course of building an EAGLE-3 speculative decoding pipeline for the Kimi-K2.5 language model, the assistant arrived at a critical infrastructure moment: restarting the SGLang inference server with a specific set of flags to enable correct hidden state extraction. Message <msg id=3385> contains a single bash command — a nohup-wrapped invocation of sglang.launch_server — that at first glance appears routine. But this command represents the culmination of a debugging process that uncovered a subtle but devastating interaction between SGLang's radix cache and the assistant's custom hidden state capture patch. Understanding why this particular configuration was chosen reveals deep insights about the engineering challenges of training speculative decoders at scale.

The Discovery That Forced a Restart

The preceding messages in the conversation reveal a careful empirical investigation. The assistant had deployed a server-side patch that intercepts hidden states at layers [3, 31, 59] during the prefill (EXTEND) forward pass, saving them as binary .pt files to /dev/shm/sglang_hs/. Initial tests were promising: a request with 13 prompt tokens produced tensors of shape [13, 7168] — exactly matching the token count and the model's hidden dimension. But earlier tests had shown a puzzling anomaly: a request with 10 prompt tokens produced only a 1-token dump. The discrepancy was traced to SGLang's radix cache, which reuses KV cache entries from previous requests. When a new request shares a prefix with a cached sequence, SGLang's EXTEND forward pass only processes the uncached suffix. The hidden state capture patch, which hooks into the forward pass, only sees these new tokens — the cached prefix's hidden states are never materialized. For EAGLE-3 training, which requires hidden states for every token position in the training sequence, this is catastrophic. The training data would be silently corrupted, with missing positions that could not be detected until the drafter failed to learn.

This discovery forced a fundamental decision: the server must be restarted with --disable-radix-cache. The assistant could not work around the issue with unique prompts alone, because even naturally unique training conversations might share system prompt prefixes. The only reliable solution was to disable caching entirely.

Anatomy of a Carefully Chosen Command

The command in <msg id=3385> is dense with deliberate choices. Let us examine each element:

SGLANG_HS_DUMP_DIR=/dev/shm/sglang_hs — This environment variable tells the custom patch where to write hidden state files. The choice of /dev/shm (shared memory) is intentional: it provides RAM-disk performance for the hundreds of gigabytes of tensor data that will be written during a 10K-sample extraction run.

NCCL environment variables — NCCL_PROTO=LL, NCCL_ALGO=Ring, NCCL_P2P_LEVEL=SYS, NCCL_MAX_NCHANNELS=16, NCCL_BUFFSIZE=16777216, NCCL_NTHREADS=512 — these are carried over from earlier performance tuning that achieved 90 tok/s single-stream throughput on this 8-GPU RTX PRO 6000 Blackwell system. The assistant assumes these settings remain optimal for the extraction workload, which is reasonable since the model and hardware are unchanged.

--disable-cuda-graph — CUDA graph capture is incompatible with the hidden state extraction patch, which needs to intercept intermediate tensors during the forward pass. CUDA graphs would capture and replay a fixed computation graph, bypassing the patch entirely.

--disable-custom-all-reduce — This disables SGLang's optimized all-reduce kernel. The assistant likely found that the custom all-reduce caused issues with the patch or with tensor parallelism on this specific hardware configuration.

--disable-radix-cache — The critical flag. Without it, hidden state extraction would silently produce incomplete data. This is the entire reason for the restart.

--mem-fraction-static 0.85 — Allocates 85% of available GPU memory to the KV cache. With radix cache disabled, every request computes its full KV cache from scratch, increasing memory pressure. The 0.85 fraction leaves headroom for the forward pass computations while maximizing throughput.

--tp-size 8 — Tensor parallelism across all 8 GPUs, unchanged from the working configuration.

Logging to a new file — The output is redirected to sglang_hs_dump_v3.log, indicating this is the third iteration of the extraction server. The v3 suffix signals a fresh start with a corrected configuration.

Assumptions and Unstated Risks

The assistant makes several assumptions in this command that deserve scrutiny. First, it assumes that disabling radix cache will not cause out-of-memory errors. With 8 GPUs each holding approximately 24 GB of memory (the RTX PRO 6000 Blackwell has 24 GB VRAM), and --mem-fraction-static 0.85 allocating ~20 GB per GPU for KV cache, the server must handle concurrent requests without exhausting memory. The assistant implicitly trusts that the extraction workload — sending one request at a time with max_running_requests=1 — will stay within bounds.

Second, the assistant assumes the dump directory /dev/shm/sglang_hs exists and is writable. The preceding messages show the assistant cleaned this directory with rm -rf, but did not explicitly recreate it. If the directory was deleted entirely rather than just emptied, the server would fail to write hidden states.

Third, the assistant assumes the NCCL tuning from the earlier benchmark session transfers correctly to the extraction workload. While the hardware is the same, the extraction workload has different characteristics: it processes one request at a time with short decode lengths (often max_tokens=1), whereas the benchmark measured sustained decode throughput. The NCCL settings optimized for streaming decode may not be ideal for the prefill-heavy extraction pattern.

Fourth, the assistant does not verify the server started successfully before proceeding. The command uses nohup and runs in the background, with output sent to a log file. The next steps in the conversation would need to check the log for errors or test with a sample request.

The Broader Significance

This message exemplifies a recurring pattern in machine learning engineering: the discovery that a seemingly innocuous optimization — radix cache — silently breaks a downstream process. The radix cache is designed to improve throughput by reusing KV cache entries, a pure win for inference. But for the hidden state extraction use case, it introduces a correctness bug that would corrupt training data without any obvious error signal. The assistant's debugging process — sending test requests, checking tensor shapes, comparing prompt token counts to dump sizes — demonstrates the kind of empirical validation required when building novel infrastructure.

The restart also marks a transition point in the session. Before this message, the assistant was in discovery mode: testing the patch, understanding the counter behavior, and diagnosing the cache issue. After this message, the assistant can proceed to the production extraction run — 10,000 samples of hidden states that will train a new EAGLE-3 drafter from scratch. The choice to use --disable-radix-cache is a conscious tradeoff: sacrificing inference throughput (since every request recomputes its full KV cache) for data correctness. In the context of EAGLE-3 training, where the quality of the drafter depends on complete and accurate hidden state supervision, this is the right call.

The command in <msg id=3385> is thus far more than a routine server restart. It is the resolution of a subtle debugging odyssey, a deliberate configuration choice that prioritizes correctness over performance, and the foundation upon which the next phase of the EAGLE-3 training pipeline is built. Without this restart — without the recognition that radix cache silently corrupts hidden state data — the entire training effort would have been built on a flawed foundation, producing a drafter with mysterious failure modes that could take weeks to diagnose.