Reading the Source: Tracing SGLang's KV Cache Allocation Logic

In any complex debugging session, there comes a moment when speculation must yield to evidence. For the assistant in this opencode conversation, that moment arrived in message [msg 5732], when it stopped inferring how SGLang's KV cache memory allocation worked and went directly to the source code to find out. The message is deceptively simple — a single sed command piped over SSH — but it represents a critical turning point in a deep investigation into GPU memory utilization and KV cache capacity.

The Message

The assistant executed:

ssh root@10.1.230.174 'sed -n "270,340p" /root/sglang/python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py'

And the output revealed the following code snippet:

            )
            return

        swa_full_tokens_ratio = self.server_args.swa_full_tokens_ratio

        # Use unified memory-based allocation for all hybrid SWA models.
        #
        # Let:
        #   F = Full layer per-token memory
        #   S = SWA layer per-token memory (may differ from F)
        #   r = swa_full_tokens_ratio = swa_tokens / full_tokens
        #
        # The profile phase computed:
        #   cell_size = F * n_full + S * n_swa
        #   max_total_num_tok...

This is the SWA (sliding window attention) memory allocation logic from SGLang's KV cache mixin, a file responsible for computing how many tokens can fit in the KV cache given available GPU memory.

Why This Message Was Written

The message was the product of a growing frustration with incomplete information. In the preceding messages ([msg 5720] through [msg 5731]), the user had asked about options for RAM offloading of the KV cache, citing low "max parallel request context." The assistant had been diligently gathering data: checking GPU memory with nvidia-smi, inspecting system RAM with free -h, and combing through SGLang's startup logs to understand the KV cache allocation.

By message [msg 5727], the assistant had traced the memory math and found a puzzling discrepancy. The server was configured with --mem-fraction-static 0.88, meaning 88% of available GPU memory should be reserved for the KV cache. After loading the target model weights (72.33 GB on each 96 GB GPU), 21.71 GB remained available. Applying the 0.88 fraction should have yielded 19.10 GB for the KV cache. Yet the actual KV cache allocation was only 10.42 GB — barely half of what the simple calculation predicted.

The assistant hypothesized that something in the memory accounting was leaving headroom, perhaps constrained by max_running_requests=48 rather than actual memory pressure. It searched for where max_total_num_tokens was assigned ([msg 5731]), but the assignment wasn't in model_runner.py where it had been looking. The natural next step was to read the actual file that contained the KV cache allocation logic: model_runner_kv_cache_mixin.py.

The Thinking Process Visible in the Message

The message reveals a methodical, hypothesis-driven debugging approach. The assistant had already formed a theory — that the KV cache size was being limited by something other than raw memory — and needed to verify it by reading the source code. The choice of sed -n "270,340p" is telling: the assistant already knew approximately where the relevant logic lived, likely from earlier grep searches that had revealed function names and line numbers.

The code snippet that came back is centered on SWA (sliding window attention) allocation. The comments in the code explain the unified memory-based allocation model for hybrid SWA models, defining variables F (full layer per-token memory), S (SWA layer per-token memory), and r (the ratio of SWA tokens to full tokens). This was directly relevant because the Kimi-K2.5 model uses hybrid attention layers — some layers use full attention while others use sliding window attention — which complicates the memory calculation.

Assumptions Made

The assistant made several assumptions in this message. First, it assumed that the KV cache allocation logic was the right place to look for the discrepancy between expected and actual memory usage. This was a reasonable assumption given that profile_max_num_token is the function that determines how many tokens the KV cache can hold.

Second, the assistant assumed that reading lines 270-340 would capture the critical allocation logic. This was based on earlier grep results that had identified the file and likely function boundaries. The assumption proved correct — the snippet contained the SWA allocation comments and the beginning of the formula.

Third, the assistant assumed that the answer to the user's question about KV cache capacity lay in understanding SGLang's internal memory accounting, rather than in external factors like PCIe bandwidth or CPU memory speed. This assumption shaped the entire investigative approach.

Input Knowledge Required

To understand this message, the reader needs several pieces of context. They need to know that the Kimi-K2.5 INT4 model is deployed across 8 NVIDIA RTX PRO 6000 Blackwell GPUs using SGLang with tensor parallelism. They need to understand the concept of a KV cache — the key-value cache that stores attention computations for reuse across decoding steps — and why its size directly limits the number of concurrent requests and context lengths a server can handle.

The reader also needs to understand the mem-fraction-static parameter, which controls what fraction of available GPU memory is reserved for the KV cache after model weights are loaded. And they need to understand hybrid SWA (sliding window attention) models, which use different attention patterns for different layers, complicating the memory allocation formula.

Output Knowledge Created

This message produced several important pieces of knowledge. First, it confirmed that the KV cache allocation logic in SGLang uses a unified memory-based approach for hybrid SWA models, with separate per-token memory costs for full-attention and SWA layers. Second, it revealed the formula structure: cell_size = F * n_full + S * n_swa, where the cell size represents the memory cost of a single token across all attention layers.

The message also set the stage for the discovery that followed. In message [msg 5736], after reading more of the file (lines 340-420 and 116-200), the assistant found the key formula: rest_memory = available_gpu_memory - total_gpu_memory * (1 - mem_fraction_static). This formula revealed that SGLang reserves (1 - mem_fraction_static) of total GPU memory (not available memory) as headroom. With mem_fraction_static=0.88, that meant 12% of 96 GB = 11.47 GB was reserved regardless of actual usage, explaining why only 10.42 GB went to KV cache despite 21.71 GB being available.

Mistakes and Corrective Insights

The assistant's initial mental model of the memory accounting was incorrect. In message [msg 5727], it had calculated 0.88 × 21.71 = 19.10 GB reserved for target KV cache, assuming the fraction applied to available memory. The source code revealed that the fraction applies to total GPU memory instead, with the reserved portion subtracted from available memory. This is a subtle but important distinction: the formula reserves a fixed chunk of total memory as safety margin, rather than taking a percentage of whatever happens to be free after weights load.

This discovery had practical implications. The 7.43 GB of "wasted" headroom per GPU wasn't actually wasted — it was the 12% safety margin (11.47 GB) partially consumed by draft model weights (0.95 GB), draft KV cache (0.60 GB), and CUDA graph memory. The remaining ~7.4 GB was genuine headroom for activation tensors, temporary buffers, and runtime spikes.

The Broader Significance

This message exemplifies a pattern that recurs throughout the conversation: when logs and heuristics fail to explain observed behavior, the assistant goes to the source code. The willingness to read and interpret SGLang's internal implementation is what enables the deep optimizations that follow — from understanding KV cache limits to eventually enabling hierarchical cache and tuning FP4 backends.

For the user, this message was the turning point. The assistant could now give an informed answer about KV cache options, explaining why mem-fraction-static couldn't simply be increased, why FP8 KV cache dtype would double capacity, and why hierarchical cache was the most promising path forward. The source code reading transformed a vague "something's off" feeling into a precise understanding of SGLang's memory model, enabling the production deployment decisions that followed in subsequent messages.