The Pivot to Hierarchical Cache: A Split-Second Architectural Decision in ML Serving

The Message

## Agent Reasoning
The user wants me to try with hierarchical cache (hicache) which spills KV cache to CPU RAM. This would allow much higher concurrency. Let me also benchmark the non-MTP server first since it's already running, then try MTP + hicache.

Actually, let me just go straight to what matters: MTP + hicache. The non-MTP server has max_running_requests=19 with 96 slots Mamba cache. With hicache, the KV cache overflows to RAM, so we can have way more concurrent requests.

Let me kill the current server and launch MTP + hicache.
Good call. Let me try MTP + hicache — spill KV to the ~700 GB of free RAM:
[bash] ssh -p [REDACTED] root@[REDACTED] "pkill -9 -f sglang; sleep 2" 2>&1
(no output)

Context: The Performance Ceiling

To understand why this message matters, we must first grasp the situation that produced it. The conversation up to this point had been a long, grueling optimization campaign for deploying the Qwen3.6-27B model with speculative decoding (MTP — Multi-Token Prediction) on a single NVIDIA RTX PRO 6000 Blackwell GPU. The team had successfully launched an MTP-enabled SGLang server after overcoming numerous hurdles: broken nohup processes over SSH, tmux session conflicts, and argument parsing issues with the --mamba-scheduler-strategy extra_buffer flag.

The MTP server was running and producing respectable per-request throughput of 62–77 tok/s, roughly 2.5× faster than the non-MTP baseline of 26.7 tok/s at a concurrency of 1. But there was a catch: the MTP configuration was severely memory-constrained. The server logs showed max_running_requests=4, meaning only four concurrent requests could be processed simultaneously. When the team benchmarked at maximum concurrency (C=4), they achieved 234 tok/s total. The bottleneck was GPU memory — the 96 GB available on a single Blackwell GPU had to accommodate the model weights, the Mamba cache (3.52 GB SSM + 2.81 GB intermediate ≈ 6.4 GB), the KV cache, and the MTP draft buffers. Something had to give, and it was the KV cache allocation.

The team had just killed the MTP server and launched a non-MTP comparison server to see whether higher concurrency without speculation could beat the MTP throughput. That non-MTP server was loading as the user interjected with a two-word suggestion: "try with hicache too?"

The Message's Reasoning: A Moment of Reassessment

The assistant's response to this suggestion is remarkable for what it reveals about decision-making under uncertainty. The message begins with a reasoning block that shows an internal debate unfolding in real time:

"The user wants me to try with hierarchical cache (hicache) which spills KV cache to CPU RAM. This would allow much higher concurrency. Let me also benchmark the non-MTP server first since it's already running, then try MTP + hicache."

>

"Actually, let me just go straight to what matters: MTP + hicache."

This is the crux of the message — a split-second pivot. The assistant initially plans to follow the logical sequential approach: benchmark the already-running non-MTP server (which took time to launch), then try MTP + hicache. But then it corrects itself, recognizing that the non-MTP benchmark is a distraction. The real question is whether MTP can be made viable at high concurrency through hierarchical cache spilling. If MTP + hicache works well, the non-MTP comparison is irrelevant. If it doesn't, they can always come back to the non-MTP baseline.

This kind of reasoning — recognizing when to abandon a planned experiment in favor of a more informative one — is a hallmark of efficient experimental design. The assistant correctly identifies that the marginal value of the non-MTP benchmark is low compared to the cost of delaying the MTP + hicache test.

The Hierarchical Cache Concept

Hierarchical cache (hicache) is a feature in SGLang that allows KV cache entries to be spilled to CPU RAM when GPU memory is exhausted. Normally, the KV cache lives entirely in GPU memory for fast access during autoregressive decoding. But for models with large context lengths or high concurrency, the KV cache can consume tens of gigabytes. Hicache provides a safety valve: when the GPU's preallocated KV cache pool fills up, older entries are evicted to system RAM. If a request needs a spilled entry later, it's reloaded back to the GPU — at the cost of PCIe transfer latency.

The key insight the assistant grasps immediately is that hicache could break the memory bottleneck that limited MTP to just 4 concurrent requests. The non-MTP server had max_running_requests=19 with 96 Mamba cache slots, but the MTP server was capped at 4 because the Mamba cache + MTP buffers consumed memory that would otherwise go to KV cache. With hicache, the KV cache could overflow to the ~700 GB of free system RAM, allowing many more concurrent requests even with the MTP memory overhead.

Assumptions and Their Validity

The message makes several assumptions, some explicit and some implicit:

Assumption 1: ~700 GB of free RAM is available. This is based on the system configuration — a machine with 8× RTX PRO 6000 Blackwell GPUs (each 96 GB) and presumably 1 TB+ of system RAM. The assistant assumes that after accounting for OS overhead and other processes, roughly 700 GB of RAM is free for KV cache spilling. This is a reasonable assumption given the hardware profile, but it doesn't account for potential memory pressure from other processes (the training pipeline, data preprocessing, etc.).

Assumption 2: Hicache will actually improve throughput. This is the central hypothesis being tested. The assistant assumes that hicache will allow higher concurrency, and that higher concurrency will translate to higher total throughput. However, hicache introduces PCIe transfer latency for spilled entries, which could degrade per-request latency. The trade-off is between higher concurrency (more requests in flight) and higher per-request latency (due to cache misses). The assistant implicitly assumes the net effect is positive, but this is an empirical question.

Assumption 3: The non-MTP server's benchmark can be deferred. The assistant decides to skip the non-MTP benchmark and go straight to MTP + hicache. This assumes that the non-MTP server can be killed and relaunched later if needed, and that no time-sensitive information would be lost. This is reasonable given that the non-MTP server was just a comparison baseline, not a production service.

Assumption 4: MTP + hicache is a valid configuration in SGLang. The assistant assumes that hierarchical cache works correctly with speculative decoding. This is not guaranteed — hicache was designed for standard autoregressive decoding, and its interaction with the Mamba cache and MTP draft buffers could introduce bugs or performance pathologies. The assistant doesn't express doubt about this, but it's a significant unknown.

The Knowledge Input: What You Need to Understand This Message

To fully grasp this message, a reader needs familiarity with several concepts:

  1. Speculative Decoding (MTP/EAGLE): A technique where a small "draft" model predicts multiple future tokens, and the large "target" model verifies them in parallel. This trades increased memory usage for higher per-request throughput. The Mamba cache stores the draft model's state.
  2. KV Cache: A key-value cache that stores attention computations from previous tokens, avoiding recomputation during autoregressive decoding. It grows linearly with sequence length and batch size.
  3. Hierarchical Cache (Hicache): A memory management strategy that spills KV cache entries from GPU to CPU RAM when GPU memory is exhausted, allowing higher concurrency at the cost of potential PCIe transfer latency.
  4. Concurrency vs. Throughput: The fundamental trade-off in serving systems. Higher concurrency (more simultaneous requests) can increase total throughput but may increase per-request latency. The optimal concurrency depends on the model, hardware, and serving system's scheduling efficiency.
  5. SGLang Server Architecture: The assistant is working with SGLang's launch_server, which manages model loading, memory allocation, request scheduling, and the HTTP API endpoint.

The Knowledge Output: What This Message Creates

This message creates several important outputs for the ongoing project:

  1. A clear experimental direction: The team will now test MTP + hicache, which is the most promising configuration for achieving both good per-request latency (from MTP) and high total throughput (from hicache-enabled concurrency).
  2. A terminated non-MTP server: The pkill -9 -f sglang command kills the non-MTP server that was loading, freeing GPU memory for the new experiment. This is a destructive action — the non-MTP benchmark results are lost — but it's a calculated trade-off.
  3. An implicit hypothesis: The message frames the hypothesis that MTP + hicache will outperform both MTP without hicache (limited to C=4) and non-MTP with high concurrency. This hypothesis will be tested in subsequent messages.
  4. A decision point documented: The reasoning block captures the assistant's thought process, serving as documentation for why the non-MTP benchmark was abandoned. This is valuable for future analysis of the project's experimental methodology.

The Thinking Process: A Microcosm of Engineering Decision-Making

The reasoning block in this message is a beautiful example of how experienced engineers make decisions under uncertainty. It follows a pattern:

  1. Interpret the suggestion: "The user wants me to try with hierarchical cache (hicache) which spills KV cache to CPU RAM. This would allow much higher concurrency."
  2. Form a plan: "Let me also benchmark the non-MTP server first since it's already running, then try MTP + hicache."
  3. Reassess the plan: "Actually, let me just go straight to what matters: MTP + hicache."
  4. Justify the pivot: "The non-MTP server has max_running_requests=19 with 96 slots Mamba cache. With hicache, the KV cache overflows to RAM, so we can have way more concurrent requests."
  5. Execute decisively: "Let me kill the current server and launch MTP + hicache." The key insight in step 3 is the recognition that the non-MTP benchmark is a distraction. The assistant realizes that the MTP + hicache configuration is the most informative experiment to run next, and that delaying it to run a less informative experiment is suboptimal. This is a form of optimal experimental design — prioritizing experiments by their expected information gain. The message also reveals the assistant's understanding of the system's memory architecture. It knows that the non-MTP server has max_running_requests=19 (from the server args output in the previous message), and it knows that hicache allows KV cache overflow to ~700 GB of RAM. This systems-level understanding is what enables the quick decision.

Conclusion

This message, though brief and containing only a single bash command, represents a critical decision point in the optimization campaign. The assistant's willingness to abandon a planned experiment (the non-MTP benchmark) in favor of a more promising direction (MTP + hicache) demonstrates adaptive reasoning under uncertainty. The reasoning block captures the essence of efficient experimental design: identify the most informative experiment, run it first, and don't let sunk costs (the time already spent launching the non-MTP server) distort your priorities.

The subsequent messages would reveal whether this bet paid off — whether MTP + hicache could break the concurrency bottleneck and deliver the throughput gains the team was seeking. But this message itself is a testament to the kind of split-second architectural decisions that define successful optimization campaigns.