Tracing the KV Cache Allocation Formula: A Diagnostic Deep Dive into SGLang's Memory Management

The Message

In a single bash command, the assistant probed the SGLang inference engine's source code to understand why nearly 8 GB of GPU memory per device was going unused despite an aggressive memory reservation setting:

ssh root@10.1.230.174 'grep -n "max_total_num_tokens\|available_size\|rest_memory\|cell_size\|kv_cache_dtype\|token_size" /root/sglang/python/sglang/srt/model_executor/model_runner.py | head -30'

This produced output showing line numbers for key variables in the KV cache allocation pipeline:

587:        self.configure_kv_cache_dtype()
595:                self.max_total_num_tokens // 2
644:                num_tokens=self.max_total_num_tokens + self.page_size,
997:        if self.server_args.kv_cache_dtype == "fp8_e4m3":
1623:            return min(self.swa_max_total_num_tokens, self.max_total_num_tokens)
1625:            return self.max_total_num_tokens
1669:    def configure_kv_cache_dtype(self):
1670:        if self.server_args.kv_cache_dtype == "auto":
1678:                    sel...

The Context: A Memory Accounting Mystery

To understand why this message was written, we must step back into the conversation that produced it. The user had deployed a massive 547 GB Kimi-K2.5 INT4 model across eight RTX PRO 6000 Blackwell GPUs (96 GB each) using SGLang with EAGLE-3 speculative decoding. After hardening the deployment into a systemd service, the user noticed they were "kinda low on max parallel request context" — the number of concurrent requests the server could handle was constrained by KV cache capacity.

The assistant began investigating. Each GPU had 96 GB total, of which 72.33 GB was consumed by model weights, leaving 21.71 GB available. With --mem-fraction-static 0.88, the assistant expected roughly 19.1 GB to be reserved for KV cache. Yet the actual KV cache allocation was only 10.42 GB (159,277 tokens), with a puzzling 7.43 GB still free after everything — draft model weights, draft KV cache, and CUDA graphs were all accounted for. Something in the memory accounting was leaving headroom on the table.

The assistant's initial hypothesis, stated explicitly in the preceding message ([msg 5727]), was that the KV cache size might be "limited by max_running_requests=48 rather than memory." This was the spark that led to message 5728.

The Reasoning: Why Grep the Source Code?

The assistant faced a classic systems debugging problem: the observable behavior (7.43 GB free per GPU) contradicted the expected behavior based on the configuration parameters. The mem_fraction_static=0.88 flag was supposed to reserve 88% of available memory for KV cache, but the actual allocation was far lower. To resolve this discrepancy, the assistant needed to understand the actual allocation formula rather than relying on assumptions about what the flag does.

The decision to grep the source code rather than, say, run a memory profiler or add debug logging, reveals a specific debugging philosophy. The assistant chose to trace the deterministic logic of the code rather than measure the runtime state. This is a powerful approach when the system is deterministic and the code is accessible — it lets you reason about what must be happening rather than inferring from noisy measurements.

The grep pattern itself is carefully constructed. Each term targets a specific piece of the allocation puzzle:

• max_total_num_tokens — the final output of the allocation algorithm, the number of KV slots available
• available_size — how much memory is considered usable after reserving headroom
• rest_memory — the key intermediate value in the formula
• cell_size — the per-token memory footprint (how much one KV slot costs)
• kv_cache_dtype — the data type of the cache, which directly affects cell_size
• token_size — another name for per-token memory cost The head -30 limit shows the assistant expected a manageable number of hits — enough to find the allocation logic without being overwhelmed by noise from unrelated variable uses.

Input Knowledge Required

To understand this message, one needs considerable context about both the SGLang codebase and the broader system architecture:

1. SGLang's memory architecture: The KV cache is allocated during server startup through a profiling phase that measures available GPU memory after model weights are loaded, then applies the mem_fraction_static ratio to determine how many tokens can fit.
2. The mem_fraction_static parameter: This flag controls what fraction of available GPU memory (after weights) is reserved for KV cache. The assistant initially assumed it was a simple multiplier on available memory, but the grep would reveal it's actually applied as rest_memory = available_gpu_memory - total_gpu_memory * (1 - mem_fraction_static) — a formula that reserves a fraction of total GPU memory as headroom, not a fraction of available memory for KV.
3. The deployment topology: Eight GPUs connected via PCIe Gen5, running tensor parallelism (TP8), with both a target model and a draft model for speculative decoding. The draft worker adds its own memory pressure.
4. The specific file structure: The assistant already knew that KV cache allocation logic lived in model_runner.py (from prior investigation of tp_worker.py in [msg 5727]), and that the profile_max_num_token function was the key entry point.

The Output Knowledge Created

This message produced a map of the KV cache allocation code. The line numbers served as signposts for the next phase of investigation:

• Line 587: configure_kv_cache_dtype() — the dtype selection happens before allocation
• Line 595: self.max_total_num_tokens // 2 — a halving operation, likely for SWA (sliding window attention) or draft worker contexts
• Line 644: num_tokens=self.max_total_num_tokens + self.page_size — page alignment for the cache
• Line 997: kv_cache_dtype == "fp8_e4m3" — FP8 KV cache path, a potential optimization
• Line 1623-1625: max_total_num_tokens accessors, with SWA min operation
• Line 1669-1678: configure_kv_cache_dtype function definition These line numbers directly guided the subsequent investigation. In the very next message ([msg 5729]), the assistant grepped for profile_max_num_token to find the core allocation function. By [msg 5732], it was reading the actual formula from model_runner_kv_cache_mixin.py. And by [msg 5736], the assistant had reconstructed the full formula:

rest_memory = available_gpu_memory - total_gpu_memory * (1 - mem_fraction_static)

This revealed the root cause: mem_fraction_static=0.88 doesn't mean "use 88% of available memory for KV." It means "reserve 12% of total GPU memory as headroom, and give everything else to KV." Since total GPU memory is 96 GB, 12% is 11.47 GB — a fixed reservation that doesn't shrink even when the model weights consume most of the memory. The 7.43 GB of "wasted" space was actually this headroom reservation, partially consumed by draft model weights, draft KV cache, and CUDA graphs.

Assumptions and Their Consequences

The assistant made a critical assumption that turned out to be incorrect: that mem_fraction_static was a simple fraction of available memory. This assumption was reasonable — many memory management systems work this way. But SGLang's implementation uses a different formula that subtracts a fraction of total memory from available memory, which produces a smaller KV cache when the model is large relative to total GPU memory.

This misunderstanding shaped the entire investigation. If the assistant had known the formula upfront, it could have immediately calculated the expected KV cache size (10.24 GB) and confirmed it matched the observed 10.42 GB, rather than spending several messages tracing through source code. The discrepancy between expected (19.1 GB) and actual (10.42 GB) was entirely an artifact of the wrong mental model.

However, this "mistake" was productive. The source code investigation revealed not just the formula but also the code paths for FP8 KV cache, SWA handling, and the hierarchical cache interface — knowledge that proved immediately useful when the assistant later recommended enabling hierarchical cache and FP8 KV dtype as solutions.

The Broader Arc

Message 5728 sits at a pivot point in the conversation. Before it, the assistant was operating on intuition and observed memory numbers. After it, the assistant had precise, code-level understanding of the memory allocation logic. This knowledge directly informed the recommendations in [msg 5736]: push mem_fraction_static higher (risky), enable hierarchical cache (safe, uses abundant system RAM), or switch to FP8 KV cache (halves memory per token).

The user chose hierarchical cache ([msg 5737]: "What about hicache?"), and the assistant immediately added --enable-hierarchical-cache --hicache-ratio 2.0 --hicache-write-policy write_through --hicache-io-backend kernel to the systemd service, leveraging the 396 GB of free system RAM as an L2 KV cache tier.

Conclusion

This single grep command exemplifies a debugging methodology that prioritizes code understanding over measurement. When faced with a puzzling discrepancy between expected and observed behavior, the assistant went straight to the source — not to find a bug, but to understand the actual algorithm. The line numbers it discovered became a roadmap for the subsequent investigation, and the corrected mental model of mem_fraction_static transformed the user's problem from "why is memory wasted" to "which of these four solutions best fits my workload." In doing so, message 5728 turned a frustrating performance puzzle into a structured optimization decision with clear tradeoffs.