The Per-Rank Pitfall: Debugging an SGLang Hierarchical Cache OOM

In the middle of an intensive optimization session for a large-scale inference pipeline, a single message captures a moment of sudden realization — the kind where a system failure snaps into focus as a simple parameter misunderstanding. The message at index 3855 reads:

439GB used, 9GB free, and some TP workers are defunct. It's OOMing. The --hicache-size 300 means 300GB per rank, not total. Kill it and try with the right size.

This short diagnostic, followed by a cleanup command, represents the culmination of a chain of reasoning that began with a performance bottleneck and ended with a system on the verge of memory exhaustion. To understand why this message matters, we need to trace the path that led to it and unpack the assumptions that made it necessary.

The Bottleneck That Started It All

The story begins with a throughput problem. The assistant was running inference on the Kimi-K2.5 model using SGLang with 8 GPUs, generating synthetic training data for an EAGLE-3 speculative decoding pipeline. The user reported that the system was "only doing 500tps" and expected it to be "4-5x faster" ([msg 3831]). Initial investigation revealed a more nuanced picture: the server was actually generating at 780-890 tokens per second, but only 46-56 requests were running concurrently despite a concurrency setting of 150 and 94-101 requests sitting in the queue ([msg 3833]).

The bottleneck became clear when the assistant examined the KV cache metrics. With --mem-fraction-static 0.85, the server could hold approximately 116,000 total tokens in its KV cache. With each request averaging over 4,000 tokens for the B2_opencodeinstruct dataset, only about 28 concurrent requests could fit at steady state. The KV cache was saturated — token usage was at 0.85-0.96, meaning the scheduler could not admit new requests because there was simply no room to store their key-value states ([msg 3834]).

The Hierarchical Cache Solution

The assistant evaluated several approaches to increase effective KV cache capacity. Simply increasing --mem-fraction-static offered marginal gains — going from 0.85 to 0.95 would only yield about 12% more tokens. The real opportunity lay in SGLang's hierarchical cache feature, which spills KV cache entries from GPU VRAM to host RAM when GPU memory is exhausted.

The math was compelling. The system had 449GB of host RAM, with approximately 408GB available after accounting for the operating system and other processes. If 300GB were allocated to hierarchical cache, the assistant calculated it could support approximately 410,000 total tokens — enough for roughly 100 concurrent requests at 4K average length, nearly doubling the effective throughput ([msg 3842]).

With the user's directive to "use all levers" ([msg 3848]), the assistant killed the existing server, verified all GPU memory was freed, and launched a new server with an aggressive configuration: --mem-fraction-static 0.93, --enable-hierarchical-cache, --hicache-size 300, and --hicache-write-policy write_through ([msg 3850]).

The Moment of Failure

The server began loading, progressing through checkpoint shards, and then appeared to hang. A health check timed out after 600 seconds ([msg 3852]). When the assistant finally inspected the server logs, it found that all TP ranks had completed CUDA graph capture with only 0.82GB of available memory remaining per GPU — a warning sign ([msg 3853]).

The critical diagnostic came next. A check of system memory showed 439GB used out of 449GB total, with only 9GB free ([msg 3854]). The assistant immediately recognized the pattern: "That's a problem — each TP rank is trying to allocate 300GB. 8 × 300GB = 2.4TB, far more than the 449GB total RAM."

This is the context for message 3855. The assistant had already identified the root cause in the previous message, but message 3855 crystallizes the understanding and initiates the corrective action. The 439GB usage confirms the diagnosis — the system is OOMing because --hicache-size 300 is interpreted as 300GB per tensor parallelism rank, not 300GB total across all ranks.

The Critical Assumption and Its Consequences

The mistake here is subtle and instructive. The assistant assumed that --hicache-size specified a total system-wide allocation — a single pool of 300GB of host RAM that all TP ranks would share for hierarchical cache overflow. This is a natural assumption from a user's perspective: you have 449GB of RAM, you want to dedicate 300GB of it to KV cache overflow, so you pass --hicache-size 300.

However, SGLang's implementation treats this parameter as per-rank. With TP=8, each of the eight ranks independently attempts to allocate 300GB of host memory for its hierarchical cache. The total demand becomes 2.4TB, which is 5.3× the available 449GB. The system cannot satisfy this demand, so it exhausts memory, some TP workers become defunct, and the server hangs in a partially-initialized state.

This kind of parameter semantics ambiguity is common in distributed systems. Some parameters are naturally global (total memory budget across all devices) while others are per-device (memory fraction per GPU). The assistant had previously worked with --mem-fraction-static, which is explicitly per-GPU — it controls what fraction of each GPU's memory is reserved for the KV cache. The hierarchical cache parameter might have been expected to follow a different pattern since it involves host memory, which is a shared resource rather than a per-device resource.

The assistant's earlier calculation in message 3842 had already divided the 300GB by 8 GPUs to compute per-GPU token capacity, suggesting an implicit awareness that the cache would be distributed. But the parameter itself was not divided — the assistant passed 300 as the raw value, expecting the framework to handle the distribution.

The Aftermath: Cleanup Challenges

Message 3855 dispatches a cleanup command — killing SGLang and Python processes, then checking memory and GPU state. But the aftermath reveals that cleanup is not straightforward. Even after killing all processes, 318GB of RAM remains in use, and GPUs 0 and 2 still show 96GB of allocated memory ([msg 3856]). The hierarchical cache allocation has leaked — the kernel-level CUDA context is stuck, holding both host memory and GPU memory hostage. The assistant attempts GPU resets, but they fail with "not supported" errors, likely because the system is running in a container environment ([msg 3859]).

This cascading failure demonstrates a second-order consequence of the parameter mistake: the OOM doesn't just fail gracefully — it leaves the system in a state that requires manual intervention to recover. The hierarchical cache, which was supposed to be a performance optimization, has instead created a resource management crisis.

Input and Output Knowledge

To fully understand message 3855, one needs several pieces of input knowledge. First, the concept of tensor parallelism (TP) in SGLang — the model is sharded across 8 GPUs, and each shard (rank) operates semi-independently, maintaining its own portion of the KV cache. Second, the hierarchical cache mechanism, which allows KV cache entries to be stored in host RAM and transferred to GPU on demand. Third, the memory topology of the system: 449GB of host RAM, 8× 98GB GPUs, and the Kimi-K2.5 model's memory footprint of approximately 68GB for weights.

The message creates several important pieces of output knowledge. It establishes that --hicache-size in SGLang is a per-rank parameter, not a total system parameter. It demonstrates that passing 300GB with TP=8 results in a 2.4TB allocation attempt that will OOM any system with less than that amount of RAM. It shows the diagnostic signature of this specific failure mode: high host memory usage, defunct TP workers, and GPUs that remain allocated even after process termination. And it provides a template for the corrective action: kill all processes, reset GPU state, and relaunch with a corrected parameter value — either --hicache-size 37 (300/8 ≈ 37.5GB per rank) or, more elegantly, --hicache-ratio which specifies the fraction of available host memory rather than an absolute size.

The Thinking Process Revealed

The reasoning visible in message 3855 and its surrounding context follows a classic diagnostic pattern. First comes observation: the server hung, health checks timed out, and a memory check shows 439GB used with only 9GB free. Second comes pattern recognition: this is an out-of-memory condition, and the hierarchical cache allocation is the most likely culprit since it was the only configuration change from the previously working setup. Third comes mechanism identification: the assistant realizes that --hicache-size is per-rank rather than total, and calculates the resulting 2.4TB demand. Fourth comes action: kill the server and prepare to relaunch with corrected parameters.

What makes this thinking process noteworthy is the speed of the realization. The assistant went from observing the hang to identifying the root cause in a single round of analysis ([msg 3853] to [msg 3854]), without needing to consult documentation or experiment further. The diagnostic was driven by a quantitative mismatch — the assistant knew the system had 449GB of RAM, observed 439GB in use, and immediately connected this to the 300GB parameter scaled by 8 ranks. This is the hallmark of a strong mental model of the system: the ability to predict what resource consumption should look like and detect when the observed consumption deviates from that prediction.

Lessons for Distributed Systems Debugging

This episode illustrates several principles for debugging parameter mistakes in distributed inference systems. First, always verify whether a resource parameter is per-device or total — the semantics can vary between seemingly similar flags. Second, when a system hangs during initialization, check resource consumption immediately; an OOM during startup looks very different from a logic deadlock. Third, be prepared for cascading cleanup failures — a parameter mistake that causes OOM may leave GPU contexts and pinned memory allocations that survive process termination.

The message at index 3855 is brief — barely a sentence of analysis followed by a cleanup command — but it represents the critical turning point where confusion gives way to understanding. It is the moment the assistant stops wondering what is wrong and starts acting on why it is wrong, turning a system failure into a learning opportunity about the semantics of distributed inference frameworks.