The KV Cache Calculation: Expanding GLM-5 Context from 8K to 200K

In the middle of a sprawling coding session dedicated to deploying the massive GLM-5 model (744B parameters) across eight RTX PRO 6000 Blackwell GPUs, a seemingly simple user request triggered a moment of careful engineering reasoning. The user reported an error — max_tokens=32000 cannot be greater than max_model_len=max_total_tokens=8192 — and asked simply: "Setup 200k context" ([msg 2076]). The assistant's response, message 2080, is a compact but revealing example of capacity planning under tight memory constraints. In a single paragraph of reasoning followed by an edit action, the assistant calculates whether the GPUs have enough free memory to support a 25× increase in context length, from 8192 tokens to 200,000 tokens.

The Message

Here is the subject message in full:

[assistant] Let me do the math on KV cache. With MLA, the compressed KV per token per layer is (kv_lora_rank + qk_rope_head_dim) * 2 bytes = (512+64)*2 = 1152 bytes. With 78 layers, TP=8, that's 1152 * 78 / 8 = 11232 bytes/token/GPU. For 200k tokens: 11232 * 200000 = ~2.09 GB/GPU. We have ~4.3 GB free per GPU currently, so 200k should fit if we bump gpu-memory-utilization slightly. Let me just set --max-model-len 200000 and see if vLLM can fit it. [edit] /home/theuser/glm-kimi-sm120-rtx6000bw/vllm-glm5.service Edit applied successfully.

The Reasoning and Motivation

The message was written in direct response to a concrete operational problem. The user had attempted to request 32,000 output tokens from the deployed GLM-5 service, but vLLM rejected the request because the server was configured with max_model_len=8192 — the default value from the GGUF model file. The error message was unambiguous, but the user's follow-up request — "Setup 200k context" — was underspecified. It did not say how to achieve this, nor did it acknowledge the memory implications of a 25× context expansion on a model that already consumed 90.8 GiB of GPU memory across eight GPUs.

The assistant therefore had to bridge the gap between the user's goal and the practical constraints. The reasoning visible in the message shows a three-step process:

1. Understand the architecture: The assistant knows that GLM-5 uses Multi-head Latent Attention (MLA), which compresses the KV cache into a low-rank latent space. This is the same MLA architecture made famous by DeepSeek V2/V3, and it dramatically reduces the per-token memory footprint compared to standard multi-head attention. The key architectural parameters are kv_lora_rank=512 (the compressed latent dimension) and qk_rope_head_dim=64 (the rotary position embedding dimension that is kept separate from the compression).
2. Calculate the memory budget: Using these parameters, the assistant computes the per-token, per-layer memory requirement: (512 + 64) dimensions × 2 bytes per fp16 value = 1152 bytes. Multiplied by 78 layers and divided across 8 tensor-parallel GPUs, this yields 11,232 bytes per token per GPU. For 200,000 tokens, the total is approximately 2.09 GiB per GPU.
3. Compare against available headroom: The assistant had previously checked GPU memory and found approximately 4.3 GiB free per GPU ([msg 2077]). Since 2.09 GiB is less than 4.3 GiB, the conclusion is that 200k context should fit — with the caveat that gpu-memory-utilization might need to be bumped slightly to give vLLM access to more of the reserved memory.

Assumptions Made

The assistant's calculation rests on several assumptions, some explicit and some implicit:

Explicit assumptions:

• The MLA compressed KV cache size is exactly kv_lora_rank + qk_rope_head_dim per token per layer. This assumes that vLLM's implementation stores the full compressed latent plus the separate RoPE dimensions, which is the standard MLA formulation.
• The KV cache is stored in fp16 (2 bytes per element). This is the default for vLLM and is consistent with the model's dtype.
• Tensor parallelism divides the KV cache evenly across 8 GPUs. This is correct for MLA since each GPU stores a shard of the latent representation. Implicit assumptions:
• vLLM will accept --max-model-len 200000 and automatically allocate the necessary KV cache blocks. The assistant does not check whether vLLM's block-based KV cache management introduces additional overhead (e.g., padding to block boundaries, extra metadata per block).
• The existing gpu-memory-utilization=0.90 setting leaves enough headroom. The assistant notes that bumping it "slightly" may be needed, but does not specify the new value or verify that the model weights themselves still fit within the allocated fraction.
• No other memory consumers (CUDA graphs, NCCL buffers, temporary tensors during inference) will significantly increase the memory footprint beyond the KV cache estimate.
• The model's RoPE scaling configuration (rope_scaling: {'rope_theta': 1000000, 'rope_type': 'default'}) supports 200k positions without modification. The model's max_position_embeddings=202752 from the config check ([msg 2077]) supports this assumption.

Mistakes and Incorrect Assumptions

While the calculation is fundamentally sound, there are several potential issues worth examining:

1. KV cache block alignment overhead. vLLM does not allocate KV cache as a flat contiguous buffer. Instead, it uses a block-based paged attention mechanism where memory is allocated in blocks (typically 16 or 64 tokens per block). This means the actual memory usage could be slightly higher than the raw per-token calculation, especially if the block size does not evenly divide the sequence length. For 200,000 tokens with a block size of 64, the last block would be fully allocated but only partially used, wasting 48 tokens' worth of cache. This overhead is small (~0.03%) but worth noting.

2. The gpu-memory-utilization parameter. The assistant's plan to "bump gpu-memory-utilization slightly" assumes that the current 0.90 setting is the only thing preventing vLLM from using the free 4.3 GiB. However, vLLM's memory allocation is more complex: it reserves memory for weights, KV cache, and scratch space based on the utilization fraction. Increasing the fraction might cause vLLM to reallocate memory for other purposes (e.g., larger scratch buffers) rather than simply exposing the free memory for KV cache.

3. The KV cache size for the prefill phase. The calculation assumes a static KV cache size, but during the prefill phase, the entire 200k-token prompt must be processed, which means the KV cache must hold all 200k tokens simultaneously before any generation begins. The assistant's calculation correctly accounts for this (200k tokens × per-token size), but the prefill phase also requires additional temporary memory for attention computation (the QK^T product matrix, softmax, etc.) which could be substantial for 200k-token sequences.

4. Memory fragmentation. With 4.3 GiB free per GPU and a requirement of 2.09 GiB, there appears to be comfortable headroom. However, CUDA memory fragmentation — especially after multiple service restarts and model loads — could reduce the size of the largest contiguous free block. The assistant does not check for fragmentation.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several areas:

Multi-head Latent Attention (MLA): This is the attention mechanism introduced by DeepSeek V2 and adopted by GLM-5. Unlike standard multi-head attention (which stores a full key-value pair per head per token), MLA compresses the KV cache into a low-dimensional latent vector (the "kv_lora_rank") plus a separate RoPE component. This reduces the KV cache size by a factor of 4-8× compared to standard MHA, which is what makes 200k context feasible on 8 GPUs with only ~2 GB per GPU.

Tensor parallelism (TP): The model is sharded across 8 GPUs, meaning each GPU stores 1/8 of each layer's weights and KV cache. The division by 8 in the calculation reflects this.

vLLM's memory management: vLLM uses a gpu-memory-utilization parameter (0.0 to 1.0) to control how much of the available GPU memory it reserves for model weights, KV cache, and scratch space. The remaining fraction is left for the CUDA runtime, NCCL, and other overhead. The assistant's plan to adjust this parameter reflects an understanding of this memory budgeting system.

GGUF model format: The model is loaded from a GGUF file, which stores the architecture configuration (including MLA parameters) in its header. The assistant had previously inspected the model's config to find kv_lora_rank=512 and qk_rope_head_dim=64.

Output Knowledge Created

This message produces several concrete outputs:

1. A memory budget estimate: 2.09 GiB per GPU for 200k-token KV cache, which is the key feasibility data point.
2. A decision: The assistant decides to proceed with --max-model-len 200000 and a slight bump to gpu-memory-utilization. This decision is immediately executed via an edit to the systemd service file.
3. A testable hypothesis: The claim "200k should fit" is testable — the next message in the conversation will show whether vLLM accepts the configuration or crashes with an out-of-memory error.
4. A reusable calculation pattern: The formula (kv_lora_rank + qk_rope_head_dim) * 2 * num_layers / tp_size * seq_len is a general-purpose estimator for MLA KV cache memory that could be applied to any MLA-based model (DeepSeek V2/V3, GLM-5, etc.).

The Thinking Process

What makes this message particularly interesting is the visible reasoning structure. The assistant does not simply guess or try a random configuration. Instead, it:

1. Identifies the relevant architectural parameters: kv_lora_rank and qk_rope_head_dim are not generic model config fields — they are specific to MLA. The assistant knows which parameters matter for KV cache sizing.
2. Performs a back-of-the-envelope calculation: The math is done in plain text, step by step, with units tracked carefully (bytes → GiB). This is the kind of reasoning that distinguishes a considered configuration change from a blind attempt.
3. Compares against measured reality: The calculated 2.09 GiB is compared against the observed 4.3 GiB free from nvidia-smi. This grounds the theoretical calculation in actual system state.
4. Acknowledges the uncertainty: The phrase "if we bump gpu-memory-utilization slightly" and "see if vLLM can fit it" both acknowledge that the calculation is approximate and the real test is whether vLLM's memory allocator accepts the configuration.
5. Acts decisively: Despite the uncertainties, the assistant proceeds with the edit. This reflects a pragmatic engineering judgment: the calculation shows comfortable headroom (2.09 GiB needed vs 4.3 GiB available), so the risk of failure is low, and the only way to know for sure is to try.

Broader Context

This message sits at a pivotal moment in the session. The GLM-5 deployment had been a long and difficult journey spanning multiple segments: installing CUDA toolkits, resolving flash-attn build failures, patching vLLM's GGUF loader for the novel glm_moe_dsa architecture, debugging incoherent output from tensor parallelism sharding mismatches, and finally achieving ~55 tok/s throughput. The model was working correctly, tool calling was enabled, and the service was stable. The user's request to expand to 200k context was the next logical step toward a production-ready deployment — long-context reasoning being a key capability of the GLM-5 architecture.

The assistant's response demonstrates that even after all the heavy lifting of environment setup and model loading, the work of capacity planning and configuration tuning continues. Each new capability (tool calling, long context, higher throughput) requires careful analysis of the system's constraints and a reasoned approach to pushing its boundaries.