The Pivot That Unlocked 400 Tok/s: A Single File Write That Combined MTP and Hierarchical Cache

The Message

[assistant] [write] /data/dflash/scripts/start_sglang_mtp_hicache.sh
Wrote file successfully.

This is message [msg 7533] in the conversation — a single tool call that writes a shell script to disk. On its surface, it is unremarkable: the assistant creates a file called start_sglang_mtp_hicache.sh. But this file write represents a pivotal architectural decision that would transform the throughput of a large language model serving pipeline from 234 tok/s to over 400 tok/s on a single GPU. To understand why this moment matters, one must trace the chain of reasoning, failed experiments, and hard-won knowledge that led to it.

The Context: A Bottleneck Revealed

In the preceding messages, the assistant had been engaged in an intense optimization campaign for serving the Qwen3.6-27B model using SGLang with Multi-Token Prediction (MTP) speculative decoding. The journey had been arduous. Earlier attempts to launch the MTP server over SSH had failed repeatedly due to a subtle interaction between SSH session lifecycle and process detachment — nohup redirects silently failed, tmux sessions were already attached and conflicted, and log files stubbornly refused to appear. The assistant eventually discovered that setsid was the reliable way to launch long-running GPU processes over SSH ([msg 7522]–[msg 7523]).

Once the MTP server was finally running, the assistant benchmarked it and discovered a critical limitation: with MTP enabled, max_running_requests was capped at just 4 concurrent requests ([msg 7524]). The reason was straightforward — the GPU had only 96 GB of VRAM, and the combination of the model weights, Mamba SSM cache (3.52 GB), intermediate buffers (2.81 GB), KV cache (77K tokens), and MTP draft buffers consumed nearly all available memory. At a concurrency of 4, the server achieved 234 tok/s total throughput ([msg 7527]).

The assistant then performed a mental calculation that revealed the tension at the heart of the deployment ([msg 7528]):

"234 tok/s (MTP, C=4) vs potentially 400+ tok/s (no MTP, C=16+)?"

The trade-off was clear: MTP gave excellent per-request latency (62–77 tok/s at C=1, a 2.5× speedup over the 26.7 tok/s without MTP), but it consumed so much GPU memory that it starved the batch scheduler. Without MTP, the server could handle 16+ concurrent requests, potentially achieving higher aggregate throughput through batching. The assistant began testing a non-MTP configuration to compare.

The User's Suggestion

Then came a two-word intervention from the user ([msg 7531]):

"try with hicache too?"

This was the spark. "Hicache" — hierarchical cache — is a feature in SGLang that spills KV cache entries from GPU memory to host CPU RAM when the GPU runs out. The machine had approximately 700 GB of system RAM (as the assistant initially assumed), so hicache could theoretically allow the MTP server to handle far more concurrent requests by treating CPU RAM as a backing store for the KV cache. The assistant immediately recognized the brilliance of the suggestion ([msg 7532]):

"Good call. Let me try MTP + hicache — spill KV to the ~700 GB of free RAM."

The assistant killed the running non-MTP server and prepared to write a new launch script that would combine MTP speculative decoding with hierarchical cache — two optimizations that had previously been treated as mutually exclusive due to memory constraints.

Writing the Script: Message 7533

Message [msg 7533] is the execution of that decision. The assistant writes /data/dflash/scripts/start_sglang_mtp_hicache.sh — a shell script that would launch SGLang with both --speculative-algorithm EAGLE (MTP) and --enable-hierarchical-cache (hicache) flags, along with the --max-hierarchical-cache-size parameter to control how much host memory to reserve.

The content of the script is not shown in the message itself (the tool call only reports "Wrote file successfully"), but from subsequent messages we can infer its structure. The script would have been modeled on the earlier start_sglang_gpu0.sh script that successfully launched MTP, but with the addition of hicache parameters. It would have set environment variables (CUDA_VISIBLE_DEVICES, LD_LIBRARY_PATH, SGLANG_ENABLE_SPEC_V2), constructed the Python launch command with all the MTP flags that had been validated in earlier testing, and added the hierarchical cache flags.

The Correction: Assumptions Meet Reality

Immediately after the script was written, the user corrected the assistant's assumption about available RAM ([msg 7534]–[msg 7535]):

"~200G of vlaw" (corrected to) "ram*"

Not 700 GB, but ~200 GB. The assistant adapted ([msg 7536]):

"~200 GB free RAM, so let me cap hicache at 150 GB to be safe."

This correction is significant. The assistant had made a reasonable but incorrect assumption about the system's memory configuration. In a production ML server with 8 GPUs, it's easy to misestimate available host memory — GPU memory, system buffers, filesystem cache, and other processes all consume RAM. The user's real-time correction prevented what would have been an over-allocation that could have caused the server to crash or the OS to start swapping.

The Result: A Breakthrough

The hicache+MTP combination proved transformative. When the server launched ([msg 7538]), it allocated 150 GB of host memory for the hierarchical KV cache and another 150 GB for the hierarchical Mamba cache. The key stat: 48 Mamba slots instead of the previous 4, and max_running_requests jumped from 4 to 9 (with hicache able to spill further).

The benchmark results ([msg 7539]) told the story:

| Concurrency | Throughput | |------------|-----------| | C=1 | 62 tok/s | | C=4 | 207 tok/s | | C=8 | 370 tok/s | | C=16 | 385 tok/s | | C=32 | 393 tok/s | | C=48 | 409 tok/s |

From 234 tok/s at C=4 (without hicache) to 409 tok/s at C=48 (with hicache) — a 75% improvement in peak throughput. The hierarchical cache had effectively removed the GPU memory bottleneck, allowing MTP's per-request speed advantage to scale across many concurrent requests.

The Thinking Process

The assistant's reasoning in this sequence reveals several hallmarks of effective ML engineering:

Trade-off awareness. The assistant understood the fundamental tension between per-request latency (where MTP excels) and batch throughput (where memory capacity dominates). It explicitly framed the question as a comparison between two regimes.

Rapid iteration. Within minutes, the assistant went from identifying the bottleneck (MTP limited to C=4) to testing a non-MTP alternative to pivoting to MTP+hicache. Each cycle involved killing a server, writing a new script, uploading it, launching it, waiting for it to load, and benchmarking.

Responsiveness to user input. The two-word suggestion "try with hicache too?" was immediately recognized as the key insight. The assistant didn't question it, didn't ask for clarification — it killed the current experiment and pivoted.

Adaptation to new information. When the user corrected the RAM estimate from 700 GB to 200 GB, the assistant immediately adjusted the hicache cap to 150 GB (a conservative 75% of available memory).

Input Knowledge Required

To understand this message, one needs:

  1. SGLang server architecture — knowledge of the launch_server command, its flags for speculative decoding (--speculative-algorithm EAGLE), Mamba cache management (--mamba-scheduler-strategy extra_buffer), and hierarchical cache (--enable-hierarchical-cache).
  2. Multi-Token Prediction (MTP/EAGLE) — an understanding that speculative decoding uses a small draft model to predict multiple tokens per forward pass, trading GPU memory for per-request latency.
  3. Hierarchical cache — the concept of spilling KV cache to host CPU RAM when GPU memory is exhausted, allowing higher concurrency at the cost of PCIe bandwidth for cache transfers.
  4. GPU memory budgeting — the ability to reason about how model weights (27B parameters ~54 GB in FP16), KV cache, Mamba cache, and draft buffers compete for the 96 GB VRAM of an RTX PRO 6000 Blackwell GPU.
  5. SSH process management — the earlier struggle with nohup, tmux, and setsid over SSH demonstrated that launching long-lived GPU processes remotely requires careful handling of process groups and session detachment.

Output Knowledge Created

This message produced:

  1. A reusable launch script (start_sglang_mtp_hicache.sh) that could be copied to the remote server and used to reproduce the MTP+hicache configuration.
  2. A validated configuration combining two previously separate optimizations. The script encoded the exact set of flags and parameters needed to make MTP and hicache work together — a configuration that had not been tested before in this session.
  3. A benchmark baseline that would be established moments later: 409 tok/s on a single RTX PRO 6000 Blackwell GPU with Qwen3.6-27B, MTP speculative decoding, and hierarchical cache spilling to 150 GB of host RAM.

What This Message Reveals About the Session

Message [msg 7533] is a study in minimalism. It is a single file write — seven words of output. Yet it sits at the intersection of multiple threads of investigation: the struggle to launch servers over SSH, the benchmarking of MTP vs non-MTP throughput, the discovery of the C=4 bottleneck, and the user's timely suggestion of hierarchical cache. The message is the hinge point where the session pivots from a frustrating memory constraint to a breakthrough in throughput.

The broader lesson is that in ML infrastructure work, the most impactful decisions often look trivial in retrospect. Writing a shell script is a mundane operation. But the choice of what to put in that script — the decision to combine MTP and hicache — was the result of a chain of reasoning that spanned dozens of messages, multiple failed launch attempts, careful benchmarking, and a user's two-word insight. The file write is the artifact; the reasoning is the real story.