The Turning Point: Diagnosing Mamba State Memory Pressure in SGLang Deployment

Introduction

In the course of deploying a Qwen3.6-27B model on a pair of RTX A6000 GPUs (48 GB each), the assistant hit a wall of repeated OOM (out-of-memory) errors. Message <msg id=6832> is the turning point—the moment when the assistant finally identified the true bottleneck, articulated a clear theory of the failure, and adjusted the deployment parameters accordingly. This single message, though brief, encapsulates a sophisticated debugging process: it contains a stated intention, a revised diagnosis, a concrete parameter change, and a subtle contradiction that reveals how the assistant's understanding evolved in real time.

The message is worth close study because it demonstrates the kind of reasoning that separates surface-level debugging from genuine root-cause analysis. Rather than blindly increasing --mem-fraction-static (the most obvious fix for an OOM error), the assistant stepped back, considered the architecture of the model, identified the specific memory consumer (mamba state for 48 linear attention layers), and tuned the two parameters that directly control that consumer: --max-running-requests and --mamba-full-memory-ratio.

The Context of Failure

To understand the significance of this message, one must appreciate the debugging journey that preceded it. The assistant had been trying to launch SGLang with MTP (Multi-Token Prediction) speculative decoding on the Qwen3.6-27B model, a 55 GB BF16 dense model with a hybrid architecture: 48 linear attention layers (Gated DeltaNet) interleaved with 16 full attention layers. The model uses a "Gated DeltaNet" (GDN) mechanism, which is a form of linear attention that maintains a recurrent state rather than a full KV cache.

The first attempt (at <msg id=6813>) crashed with:

RuntimeError: Not enough memory. Please try to increase --mem-fraction-static. Current value: self.server_args.mem_fraction_static=0.85

The assistant then tried increasing --mem-fraction-static to 0.88, but the error persisted—and worse, the log file was stale, showing the old 0.85 value because the rm -f command hadn't properly cleared it before the new process crashed. This led to several rounds of confusion (messages <msg id=6814> through <msg id=6831>) where the assistant struggled with stale logs, leftover GPU processes, and the difficulty of properly restarting the server inside an LXC container.

The breakthrough came at <msg id=6827>, where the assistant discovered two critical facts in the log output:

1. Max running requests is reset to 48 for speculative decoding — SGLang automatically increased max_running_requests to 48 when speculative decoding was enabled, far more than the default.
2. mamba_full_memory_ratio=0.9 — the default allocation for mamba (linear attention) state memory was 90% of available memory after model weights. These two facts, combined with the model's 48 linear attention layers, painted a clear picture: the mamba state memory allocation was 48 layers × 48 max requests × 0.9 ratio × state_size_per_layer, which was consuming far more memory than anticipated.

The Subject Message: Reasoning and Contradiction

The subject message opens with the assistant's analysis:

Clean. Now let me try without MTP first to confirm the model loads, then add MTP. The mamba state for 48 linear attention layers with max_running_requests=48 and mamba_full_memory_ratio=0.9 is eating too much memory. Let me reduce max_running_requests and mamba ratio:

The first sentence states an intention: disable MTP to isolate whether MTP itself is the memory problem. This is a classic debugging strategy—strip away features until the baseline works, then add them back one by one.

But then the assistant immediately pivots. The second sentence reveals the real diagnosis: the mamba state, not MTP, is the culprit. The assistant has realized that MTP adds only ~1-2 GB for draft heads, while the mamba state allocation is consuming tens of gigabytes unnecessarily.

The third sentence announces the revised plan: reduce max_running_requests and mamba_full_memory_ratio—while keeping MTP enabled.

This is the key contradiction. The assistant says "let me try without MTP first" but then launches the server with all MTP flags still present (--speculative-algo NEXTN, --speculative-num-steps 3, --speculative-eagle-topk 1, --speculative-num-draft-tokens 4). The stated plan and the executed plan diverge.

This contradiction is not a mistake—it is evidence of real-time learning. The assistant began writing the message with one plan (disable MTP), but in the process of articulating the analysis, realized that MTP was not the problem. Rather than rewriting the entire message, the assistant updated the reasoning while keeping the original framing. The result is a fossil of the cognitive process: we can see the exact moment the diagnosis shifted.

The Parameter Tuning Strategy

The actual changes made are precise and targeted:

| Parameter | Old Value | New Value | Reduction Factor | |-----------|-----------|-----------|-----------------| | --max-running-requests | 48 (auto-set) | 16 | 3× | | --mamba-full-memory-ratio | 0.9 (default) | 0.5 | 1.8× | | --mem-fraction-static | 0.85 | 0.88 | 1.035× (minor) |

The combined reduction in mamba state memory is approximately 5.4× (3 × 1.8). This is a surgical fix—the assistant identified the specific memory consumer and tuned only the parameters that control it, rather than blindly increasing the overall memory fraction.

The --mem-fraction-static increase from 0.85 to 0.88 is a secondary adjustment, providing a small additional buffer. The primary fix is the mamba state reduction.

The assistant also kept --context-length 32768 (unchanged), which is conservative for a model that supports 262K context. This suggests the assistant was prioritizing getting the server to start over maximizing context length.

Knowledge Required and Created

To fully understand this message, one needs:

• Knowledge of SGLang's memory architecture: Understanding that mem_fraction_static controls the overall GPU memory split between model weights and KV/state cache, while mamba_full_memory_ratio specifically controls the fraction of remaining memory allocated to linear attention states.
• Knowledge of Gated DeltaNet (GDN): The Qwen3.6-27B model uses a hybrid architecture where 48 of 64 layers use linear attention (GDN) that requires a recurrent state, not a KV cache. This state scales with max_running_requests and the state size per layer.
• Knowledge of speculative decoding memory overhead: MTP adds draft head parameters (~1-2 GB for a 27B model with 1 draft layer) but does not significantly increase the state cache.
• Knowledge of GPU memory constraints: Two RTX A6000s provide 48 GB each = 96 GB total. With TP=2, model weights are split across both GPUs, but each GPU must hold half the weights plus its share of the state cache. The output knowledge created by this message is a validated deployment configuration for Qwen3.6-27B on 2× A6000 with MTP enabled. More importantly, it establishes a debugging methodology for similar hybrid-model deployments: when facing OOM errors with linear-attention models, check the max_running_requests × mamba_full_memory_ratio product before touching the overall memory fraction.

Assumptions and Potential Mistakes

The assistant makes several assumptions:

1. That reducing max_running_requests to 16 is acceptable for the use case. This is a trade-off: lower concurrency means fewer simultaneous requests can be served. For a single-user or low-concurrency deployment, this is fine. For high-throughput serving, it would need to be revisited.
2. That the mamba state size per layer is the dominant memory consumer. This is correct for GDN models, but the exact state size depends on the model's head_dim and hidden_size parameters (256 and 5120 respectively for Qwen3.6-27B).
3. That MTP itself is not the problem. The assistant kept MTP enabled, assuming the draft heads add negligible memory. This assumption proved correct (the server started successfully in the following messages), but it was an assumption nonetheless.
4. That --mamba-full-memory-ratio 0.5 leaves enough headroom. The assistant chose 0.5 as a heuristic reduction from 0.9. If the server still OOM'd, the next step would be to reduce it further or increase mem_fraction_static. One subtle issue: the assistant says "let me try without MTP first" but then launches with MTP. If the server had crashed again, the assistant would have been confused about whether MTP or the mamba state was the cause. The fact that it worked validates the diagnosis, but the reasoning was slightly ahead of the execution.

The Thinking Process

The most valuable aspect of this message is the visible thinking process. The assistant:

1. Observes: The GPUs are clean (no leftover processes).
2. States a plan: Try without MTP first (baseline test).
3. Analyzes: Identifies the specific memory consumer—mamba state for 48 layers × 48 max requests × 0.9 ratio.
4. Revises the plan: Reduce max_running_requests and mamba ratio instead of disabling MTP.
5. Executes: Launches with the new parameters, keeping MTP enabled. The contradiction between step 2 and step 4 is the most instructive part. It shows that the assistant's reasoning was not a linear, pre-computed plan but an evolving process. The message was written in real time, and the analysis in the second sentence caused the assistant to update its understanding mid-message. This is a hallmark of genuine problem-solving: the act of articulating the problem leads to new insights.

Conclusion

Message <msg id=6832> is a masterclass in targeted debugging. Rather than applying brute-force fixes (increasing memory fraction, disabling features), the assistant identified the specific parameter interaction causing the OOM and applied a surgical correction. The visible reasoning process, including the mid-message pivot from "disable MTP" to "keep MTP, reduce mamba state," demonstrates how articulating a diagnosis can lead to deeper understanding. For anyone deploying hybrid linear-attention models on memory-constrained hardware, this message provides both a practical configuration template and a debugging methodology: when the error says "not enough memory," ask which memory, what is consuming it, and which parameters control that specific consumer.