From 8K to 200K: Expanding Context for GLM-5 on vLLM

The Message in Context

In message [msg 2087] of a sprawling opencode session spanning GPU driver installation, flash-attn compilation, GGUF patching, and multiple model deployments, the assistant delivers a brief but consequential status update:

Working — max_tokens=32000 is now accepted (the model just generated 20 tokens and stopped naturally). The 200k context limit is in place.

Done. Changed --max-model-len from 8192 to 200000. The model's max_position_embeddings is 202752, and the MLA compressed KV cache is very memory-efficient (~1.1 KB/token/GPU across 78 layers with TP=8), so 200k fits comfortably within the available VRAM. Service is restarted and accepting requests with up to 200k context.

This message is the culmination of a focused sub-task triggered by a user error. The user had attempted to request max_tokens=32000 from the GLM-5 GGUF deployment, only to be met with a validation error: "max_tokens=32000 cannot be greater than max_model_len=max_total_tokens=8192. Please request fewer output tokens." The user's follow-up instruction was succinct: "Setup 200k context."

What makes this message interesting is not its length—it is a mere five sentences—but the density of reasoning, calculation, and system knowledge it encapsulates. It is a report-back message, the terminal node in a decision tree that began with a constraint violation and ended with a successful reconfiguration of a production inference server.

The Problem: A Hard-Coded Context Ceiling

The immediate trigger was a vLLM validation error. When the user sent a request with max_tokens=32000, vLLM's API server rejected it because the engine had been initialized with --max-model-len 8192. This parameter controls the maximum sequence length the engine will accept, encompassing both prompt and generated tokens. Setting max_tokens to 32000 was impossible because the total sequence length (prompt tokens + 32000 generated tokens) would exceed 8192 regardless of the actual prompt length.

The root cause was a conservative default. Earlier in the session ([msg 2078]), the assistant had discovered that the running service was configured with max_seq_len=8192. This was likely a safe starting value—enough for basic chat but far below the model's native capability. The GLM-5 model, as the assistant discovered by inspecting its configuration via HuggingFace's AutoConfig, has max_position_embeddings: 202752. The model architecture natively supports approximately 202K tokens of context. The 8192 limit was entirely an artifact of the deployment configuration, not a model constraint.

Investigation: Can 200K Fit?

Before making any changes, the assistant performed a critical feasibility calculation. The central question was: does the available GPU memory support a 200K-token context window?

The assistant's reasoning, visible in the preceding messages ([msg 2080]), is a model of pragmatic engineering analysis. The GLM-5 model uses Multi-Head Latent Attention (MLA), a memory-efficient attention mechanism that compresses the KV cache into a low-rank latent space. The compressed KV representation per token per layer is kv_lora_rank + qk_rope_head_dim = 512 + 64 = 576 elements. In fp16 (2 bytes per element), that is 1152 bytes per token per layer.

With 78 layers and tensor parallelism across 8 GPUs, the per-GPU memory footprint is:

1152 bytes/token/layer × 78 layers ÷ 8 GPUs = 11,232 bytes/token/GPU

For 200,000 tokens:

11,232 × 200,000 = ~2.09 GB/GPU

The assistant then checked the current GPU memory state via nvidia-smi ([msg 2077]), which showed approximately 4.2 GB free per GPU out of ~95.6 GB total. The model weights consumed roughly 90.8 GB per GPU (at 0.90 gpu-memory-utilization). With ~4.3 GB free, the 2.09 GB KV cache requirement for 200K context would fit comfortably, with headroom to spare.

This calculation involved several assumptions:

The MLA compressed KV cache formula is correct. The assistant assumed kv_lora_rank=512 and qk_rope_head_dim=64 based on the GLM-5 architecture. These values are standard for DeepSeek-style MLA but could theoretically differ in GLM-5's implementation.
vLLM's KV cache allocation matches the theoretical minimum. In practice, vLLM may allocate cache blocks in fixed-size chunks (typically 16 or 32 tokens per block), which introduces some overhead. The assistant's calculation of ~2.09 GB was a lower bound; the actual allocation could be slightly higher.
No other memory consumers appear at runtime. The assistant assumed that the memory used during model loading and CUDAGraph compilation would be freed or remain stable. This is generally true for vLLM's memory management, but peak memory during compilation can temporarily exceed steady-state usage.
The gpu-memory-utilization=0.90 setting provides enough headroom. The assistant did not adjust this parameter, implicitly trusting that the existing 90% utilization limit plus the 4.3 GB free would be sufficient. These assumptions were reasonable and ultimately validated by the successful deployment, but they represent engineering judgment rather than certainty. A more conservative approach might have involved incrementally increasing the context length and monitoring memory, but the assistant's calculation was solid enough to proceed directly to 200K.

Implementation: One Parameter Change

The implementation was remarkably simple: change --max-model-len from 8192 to 200000 in the systemd service file. The assistant edited the service file ([msg 2080]), deployed it via scp, reloaded systemd, and triggered a restart ([msg 2081]).

The model reload took approximately 7 minutes—51 GB of model weights per GPU plus AOT compilation of CUDAGraphs. The assistant monitored the restart process across multiple polling steps ([msg 2082], [msg 2083], [msg 2084], [msg 2085]), watching for errors, OOM failures, or configuration rejections.

The key validation point came when the logs showed max_seq_len=200000 was accepted without complaint ([msg 2082]). No OOM errors appeared. The model loaded successfully, consuming 51 GB per GPU for weights ([msg 2083]), and the server reached "Application startup complete" ([msg 2085]).

Verification: Proving It Works

The final verification in the subject message is elegant in its simplicity. The assistant sent a request with max_tokens=32000—the very parameter that had previously triggered the validation error—and confirmed it was accepted. The model generated only 20 tokens and stopped naturally (because the prompt "Say hello" requires a short response), but crucially, the request was not rejected.

This is a textbook example of a negative test case repurposed as a positive test: the same input that previously produced an error now succeeds. The assistant also implicitly verified that the model's output quality was unaffected by the context length change, though this was not explicitly tested in this message.

What This Message Reveals About the Session

The subject message sits at the intersection of several themes that run throughout this opencode session:

Resource-constrained engineering. The entire session is characterized by working within tight GPU memory budgets. From flash-attn compilation with reduced MAX_JOBS to avoid OOM, to careful KV cache calculations, to the eventual pivot from GLM-5 to Kimi-K2.5-NVFP4, the assistant consistently operates under the constraint of finite VRAM. This message is no exception—the feasibility of 200K context hinged on a precise memory budget calculation.

Pragmatic minimalism. The assistant could have over-engineered this change. It could have adjusted gpu-memory-utilization, added memory monitoring, or implemented graceful fallback if 200K didn't fit. Instead, it made a single parameter change, verified it worked, and reported success. This reflects a deep understanding of vLLM's architecture: --max-model-len is the primary control for context length, and if the KV cache fits, no other changes are needed.

Trust but verify. The assistant's process follows a consistent pattern: calculate, implement, monitor, verify. The calculation (KV cache size) informed the implementation (parameter change). The monitoring (journalctl polling) confirmed the model loaded without errors. The verification (test request) proved the system worked end-to-end. This four-step cycle appears repeatedly throughout the session.

The assistant as system integrator. This message demonstrates the assistant's role not as a code generator but as a system integrator who understands the interactions between model architecture (MLA), inference engine (vLLM), hardware (RTX PRO 6000 Blackwell GPUs), and deployment infrastructure (systemd). The assistant doesn't just know that --max-model-len exists; it knows why the change is safe, grounded in an understanding of MLA's memory efficiency and the specific GPU memory budget.

Deeper Technical Context: Why MLA Matters

The assistant's mention of "MLA compressed KV cache" is a crucial detail that deserves unpacking. Multi-Head Latent Attention, pioneered by DeepSeek and adopted by GLM-5, addresses one of the fundamental scaling challenges of long-context LLM inference: the KV cache grows linearly with both sequence length and the number of attention heads.

In standard multi-head attention with 64 layers, 128 heads, and a head dimension of 128, each token would require 64 × 128 × 128 × 2 = ~2 MB of KV cache in fp16. For 200K tokens, that would be 400 GB—impossible for any single GPU and impractical even across 8 GPUs.

MLA solves this by projecting the KV pairs into a low-dimensional latent space (rank 512 in GLM-5's case) and adding a small per-head RoPE component (64 dimensions). The compressed representation is 512 + 64 = 576 elements per token per layer, a reduction of roughly 28× compared to the full KV cache. This is what makes 200K context feasible on 8 GPUs with only ~2 GB of KV cache per GPU.

The assistant's calculation of "~1.1 KB/token/GPU" (which is 11,232 bytes, or ~11 KB—the message says 1.1 KB but the calculation shows 11.2 KB; this appears to be a minor error in the message where "KB" should be "KiB" or the decimal is misplaced) reflects this compressed representation. The exact value depends on the tensor parallelism sharding: with TP=8, each GPU stores 1/8 of the KV cache per layer, hence the division by 8 in the calculation.

What Could Have Gone Wrong

Several failure modes were avoided or mitigated:

OOM during compilation. The CUDAGraph compilation phase temporarily allocates additional memory for workspace buffers. If the 200K KV cache allocation combined with compilation workspace exceeded the ~4.3 GB free per GPU, the process would have crashed. The assistant mitigated this by monitoring the logs for OOM errors during the restart.

Fragmented memory. GPU memory can become fragmented after repeated load/unload cycles. The assistant had previously implemented an ExecStartPre cleanup step that waits for GPU memory to be freed and cleans stale shared memory segments ([msg 2062]). This reduced the risk of fragmentation-related allocation failures.

Configuration incompatibility. The --max-model-len 200000 parameter could theoretically conflict with other settings like --max-num-batched-tokens or --max-num-seqs. The assistant implicitly relied on vLLM's internal validation to catch such conflicts, and none materialized.

The model's actual context limit. The assistant checked max_position_embeddings=202752 from the HuggingFace config, but this value is a training-time parameter that may not reflect the model's true effective context. Models can exhibit quality degradation at long contexts even if the architecture supports them. The assistant did not test long-context quality, only availability.

Conclusion

Message [msg 2087] is a model of concise, confident engineering communication. It reports a successful outcome, summarizes the change made, provides the technical justification (MLA memory efficiency), and confirms the system is operational. The message's brevity is earned through the rigor of the preceding investigation: the KV cache calculation, the memory check, the configuration inspection, and the monitored restart.

This message also illustrates a broader pattern in AI-assisted system administration: the assistant acts as a knowledgeable engineer who can reason about system constraints, calculate resource requirements, make targeted configuration changes, and verify outcomes. The 200K context expansion was not a trivial change—it required understanding MLA architecture, vLLM's memory management, GPU memory budgets, and the model's native capabilities. The assistant synthesized all of these into a single parameter change and a five-sentence status report.

The expansion from 8K to 200K context represents a 25× increase in the model's usable context window, enabled entirely by understanding and working within the system's constraints. It is a small but satisfying victory in a session full of larger battles—flash-attn compilation, GGUF patching, Triton kernel debugging—and it demonstrates that sometimes the most impactful changes are the simplest ones, provided they are backed by solid reasoning.