When Speculation Goes Too Far: Diagnosing an OOM Crash in SGLang's EAGLE Decoding

Introduction

In the high-stakes world of large language model inference, every millisecond counts. Speculative decoding—a technique where a lightweight draft model predicts multiple future tokens in parallel while the main model verifies them—has emerged as one of the most effective ways to reduce per-request latency. But as with any optimization, pushing parameters too aggressively can backfire spectacularly. Message [msg 6532] captures a moment when the assistant, in the midst of an exhaustive speculative decoding tuning session for the Qwen3.5-122B-A10B model running on SGLang, discovers that its latest experiment has ended not with improved throughput, but with a SIGKILL from the Linux OOM killer.

This message is a masterclass in diagnostic reasoning under pressure. It shows how the assistant connects a process termination signal to its root cause—excessive KV cache consumption from over-aggressive speculation—and immediately pivots to gather the data needed to confirm the hypothesis. To understand this message fully, we must trace the experimental arc that led to this crash, examine the assumptions that were tested, and appreciate the resource management constraints that govern modern LLM serving infrastructure.

The Message

The subject message reads:

It got SIGKILLed — probably OOM killed (no swap on this machine). With steps=10, the draft tokens per request are 11, eating too much KV cache. Let me check: [bash] ssh root@10.1.230.174 'journalctl -u sglang-qwen.service --since "5 min ago" --no-pager | grep -E "max_total_num_tokens|max_running"' 2>&1 Mar 09 22:35:06 llm-two sglang-qwen[16414]: [2026-03-09 22:35:06] server_args=ServerArgs(model_path='/shared/models/Qwen3.5-122B-A10B', tokenizer_path='/shared/models/Qwen3.5-122B-A10B', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=30000, fastapi_root_path='', grpc_mode=Fals...

The message is deceptively brief—two sentences of analysis followed by a diagnostic shell command and its output. But within those sentences lies a remarkable chain of inference. The assistant has received a SIGKILL signal, knows the machine has no swap, understands the relationship between speculative steps and KV cache pressure, and immediately formulates a testable hypothesis.

The Experimental Arc: Why We Were Here

To appreciate this message, we must understand what came before. The assistant had been engaged in a systematic optimization campaign for the Qwen3.5-122B-A10B model running on a Proxmox-hosted machine with 4× RTX PRO 6000 Blackwell GPUs (the machine at 10.1.230.174, hostname "llm-two"). The model uses a Mixture-of-Experts architecture with 256 experts and a per-expert intermediate size of just 256, making the MoE GEMM kernels memory-bound rather than compute-bound. Having ruled out MoE kernel tuning as a meaningful optimization path, the assistant turned to speculative decoding—specifically, SGLang's EAGLE (EAGLE-3) implementation with MTP (Multi-Token Prediction) speculation.

The tuning journey was methodical and data-driven. Starting from the default --speculative-num-steps=1 (which drafts 2 tokens per step with topk=1), the assistant progressively increased the speculation depth:

The Crash: What Actually Happened

The assistant dutifully edited the systemd service file to set --speculative-num-steps 10, deployed the change, and waited for the model to load. What followed was a confusing sequence of events. In message [msg 6530], the assistant initially misinterpreted the situation: seeing logs showing 270-293 tok/s throughput with an acceptance rate of 0.92, it concluded "Wait — it actually ran!" and assumed the crash was from the benchmark script timing out. But when it checked the service endpoint, it got "NOT READY."

Message [msg 6531] revealed the truth: the service was in an "activating (auto-restart)" state with "Result: signal." The process had been terminated by a signal—specifically SIGKILL, which cannot be caught or handled by the process. On a Linux system, SIGKILL from an out-of-memory condition is the classic signature of the OOM killer.

The Diagnostic Reasoning

This is where message [msg 6532] shines. The assistant connects three pieces of evidence into a coherent explanation:

  1. The signal: SIGKILL, which is what the OOM killer sends when the system runs out of memory.
  2. The system configuration: "no swap on this machine." Without swap, the OOM killer has no buffer—once physical memory is exhausted, processes are killed immediately.
  3. The parameter change: "With steps=10, the draft tokens per request are 11, eating too much KV cache." The reasoning is elegant. With speculative_eagle_topk=1, SGLang automatically adjusts speculative_num_draft_tokens to speculative_num_steps + 1. So steps=10 means 11 draft tokens per request. Each draft token consumes KV cache space proportional to the model's hidden dimension and number of layers. With 11 draft tokens per request instead of the default 2, each concurrent request consumes roughly 5.5× more KV cache. The max_running_requests parameter, which was already down to 17 at steps=3, would have been even lower at steps=10—but the assistant hadn't checked the exact value before the crash. The assistant then runs a targeted diagnostic command: grep the journal for max_total_num_tokens and max_running to see what values SGLang computed for this configuration. The output confirms the server_args line is present, though the full values are truncated in the message. The assistant is looking for the specific resource allocation numbers that would confirm the KV cache budget was exceeded.

Assumptions and Their Validity

The assistant makes several assumptions in this message, most of which are well-founded:

Assumption 1: SIGKILL = OOM killer. This is the most common cause of SIGKILL on a server process, but not the only one. A user could manually send SIGKILL with kill -9, or another system watchdog could terminate the process. However, given the context—the assistant had just launched the service with an aggressively high speculation parameter, and the machine has no swap—the OOM killer hypothesis is by far the most probable.

Assumption 2: The KV cache is the memory bottleneck. This is correct for LLM serving. The KV cache for a 122B-parameter model with 262K context length is enormous. Each additional draft token multiplies the KV cache footprint per sequence. With 11 draft tokens per step, the KV cache for a single request could be 5-10 GB depending on the model's layer count and hidden dimension.

Assumption 3: The crash happened during or after model loading. The assistant's earlier confusion (thinking the model had run successfully in msg [msg 6530]) suggests the process may have survived long enough to serve some requests before being killed. This is consistent with an OOM scenario where memory pressure builds gradually as requests accumulate.

One subtle incorrect assumption deserves note: in message [msg 6530], the assistant initially assumed the crash was caused by "my benchmark script timing out" rather than OOM. This was a reasonable hypothesis—the benchmark script sends requests and waits for responses, and a timeout could indicate a network issue or a hung process. But the SIGKILL signal told a different story, and the assistant corrected course quickly.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of speculative decoding: How EAGLE/MTP works, the relationship between num_steps and num_draft_tokens, and how draft tokens consume KV cache.
  2. SGLang server architecture: How max_running_requests and max_total_num_tokens are computed from available GPU memory, and how speculative decoding parameters affect these limits.
  3. Linux process management: What SIGKILL means, how the OOM killer works, and why "no swap" is relevant.
  4. The experimental context: The assistant had been systematically tuning steps from 1 through 5, tracking throughput improvements and resource tradeoffs at each level.
  5. The hardware constraints: 4× RTX PRO 6000 Blackwell GPUs with finite VRAM, no swap space on the host.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

  1. A confirmed upper bound: Steps=10 (11 draft tokens) exceeds the memory budget on this hardware configuration. This is a concrete data point for future deployments.
  2. A diagnostic methodology: The assistant demonstrates how to investigate a SIGKILL crash in an LLM serving context—check the signal, consider the OOM killer, identify the resource-hungry parameter, and grep for the relevant server configuration.
  3. A cautionary data point: The relationship between speculation depth and memory pressure is not linear in its practical impact. While steps=4 (5 draft tokens) was sustainable and highly beneficial, steps=10 (11 draft tokens) was catastrophic.
  4. The journalctl command pattern: Using grep -E "max_total_num_tokens|max_running" to extract the specific server configuration from the startup logs is a reusable diagnostic technique.

The Thinking Process

The assistant's reasoning, visible in the transition from message [msg 6531] to [msg 6532], follows a classic diagnostic arc:

  1. Observation: The service is in "auto-restart" with "Result: signal."
  2. Hypothesis generation: SIGKILL + no swap + aggressive speculation parameter = OOM killer.
  3. Causal chain: Steps=10 → 11 draft tokens per request → KV cache per request increases by ~5.5× → memory exhausted → OOM killer terminates the process.
  4. Evidence gathering: Run journalctl to capture the server_args line, which contains the computed max_total_num_tokens and max_running_requests values.
  5. Confirmation: The server_args output confirms the configuration was applied, and the assistant can now see the exact resource allocation that led to the crash. What's particularly impressive is the speed of this reasoning. The assistant had just moments earlier believed the model was running successfully (msg [msg 6530]). When the service status check revealed the SIGKILL, it immediately reformulated its understanding and produced a coherent explanation.

Conclusion

Message [msg 6532] is a small but revealing window into the realities of LLM inference optimization. It demonstrates that pushing any single parameter to its extreme, even one that has shown consistent improvement at lower values, can hit hard resource ceilings. The assistant's diagnostic response—quick, hypothesis-driven, and targeted—turns a crash into a learning opportunity. The knowledge gained here (steps=10 is too aggressive for this hardware) is immediately actionable and informs the optimal configuration that will be deployed.

In the broader context of the session, this crash effectively ended the speculation depth exploration. The assistant had already found that steps=4 (277 tok/s) and steps=5 (282 tok/s) were near the peak, and the crash at steps=10 confirmed that further increases were not viable. The optimal configuration for this model and hardware would be steps=4 or steps=5, depending on whether single-request throughput or aggregate throughput at higher concurrency was the priority.

This message also serves as a reminder that in production LLM serving, memory is always the constraining resource. Every optimization that improves latency or throughput does so by trading against memory consumption. The art of serving is finding the right balance, and sometimes the only way to find the edge is to step over it.