The no_buffer Pivot: Diagnosing and Eliminating a Mamba State Bottleneck in Batch Inference

Introduction

In the course of a sprawling ML infrastructure session spanning environment setup, model deployment, and training data generation, one message stands out as a turning point where a careful diagnostic observation translated into a concrete optimization that nearly doubled throughput. The message, sent by the AI assistant in an opencode session, reads:

1,170 tok/s per GPU with 64 concurrent requests — up from 670 tok/s. That's 9,360 tok/s aggregate. The client-side is still ramping (1.7 req/s at 291 completed). Let me wait a bit longer for steady state:

This brief status update, followed by a bash command to verify steady-state metrics, encapsulates the payoff of a multi-step diagnostic and remediation process. To understand its significance, we must trace the reasoning chain that led to this moment, examine the assumptions that underpinned the fix, and evaluate what the message reveals about the assistant's thinking process.

The Context: A Data Generation Pipeline Under Pressure

The message belongs to a larger effort to generate 193K diverse training prompts for a DFlash drafter model. The pipeline involved serving the Qwen3.6-27B model (a Mamba-hybrid architecture) across 8× RTX PRO 6000 Blackwell GPUs using SGLang, then feeding prompts through these servers for batch inference. The user had explicitly stated an expectation: the PRO 6000 cards, with roughly half the memory bandwidth and half the compute of an NVIDIA B200, should achieve about 25% of the B200's throughput—roughly 6,250 tokens per second aggregate.

Earlier in the session, the assistant had observed throughput of only ~3,900 tok/s aggregate, or about 15.6% of B200 performance ([msg 9574]). The user's correction in [msg 9573]—"We should be at ~25% of a B200 at least"—set the stage for a targeted optimization effort.

The Diagnostic Chain: Finding the Real Bottleneck

The assistant's reasoning in the messages preceding the subject message reveals a systematic diagnostic approach. In [msg 9575], the assistant queried per-server metrics and found a critical clue: while KV cache utilization was only 20% (full token usage: 0.21), the mamba state memory was at 61% utilization. This was the bottleneck. The extra_buffer mamba scheduler strategy, which allocates extra memory for branching and overlap scheduling in Mamba's linear attention, was consuming memory that could otherwise host additional concurrent requests.

The assistant calculated that with 32 concurrent requests per GPU and ~670 tok/s decode throughput, the system was underperforming by roughly 40% relative to theoretical capacity. The max_running_requests was only 37 per GPU, constrained by mamba state memory rather than KV cache.

This led to a clear hypothesis: switching from extra_buffer to no_buffer would free mamba state memory, allowing more concurrent requests to fit in GPU memory, and thereby increase throughput. The tradeoff was that no_buffer eliminates the overlap scheduling that reduces latency variance, but for a batch generation pipeline where raw throughput matters more than per-request latency, this was an acceptable sacrifice.

The Fix and Its Immediate Results

In [msg 9579], the assistant relaunched all 8 SGLang servers with --mamba-scheduler-strategy no_buffer and --mem-fraction-static 0.90. The result, visible in [msg 9584], was that max_running_requests jumped from 37 to 72 per GPU—nearly a 2× increase in capacity.

The subject message then captures the first verification of this change's impact. With 64 concurrent requests (approaching the new capacity limit), per-GPU decode throughput had risen from ~670 tok/s to ~1,170 tok/s—a 75% improvement. Aggregate throughput reached ~9,360 tok/s, exceeding the user's 25%-of-B200 target of ~6,250 tok/s. The ETA for the generation job dropped from ~107 hours to ~62 hours.

Assumptions Embedded in the Message

The message makes several implicit assumptions that are worth examining. First, it assumes that the no_buffer strategy is safe for this workload—that the elimination of overlap scheduling won't cause instability or request failures. The assistant had verified this implicitly by checking server health ([msg 9582]) before resuming generation.

Second, the message assumes that the client-side generation script will ramp up to fill the newly available capacity. The assistant notes "the client-side is still ramping (1.7 req/s at 291 completed)," acknowledging that the full potential hasn't been reached yet. The follow-up check in the same message shows 1,397 completed at 2.9 req/s, confirming the ramp is progressing.

Third, the assistant assumes that the observed per-GPU throughput of 1,170 tok/s is a stable figure, not a transient spike. The fact that the system has 64 running requests and is still ramping suggests the throughput may increase further as more requests fill the pipeline.

Input Knowledge Required

To fully understand this message, one needs knowledge of several domains:

Mamba hybrid architectures: The Qwen3.6-27B model uses Mamba's linear attention layers alongside traditional attention. Mamba's state management is fundamentally different from KV cache, requiring separate memory allocation and scheduling strategies.
SGLang's mamba scheduler strategies: The extra_buffer strategy allocates extra memory for Mamba states to enable overlap scheduling (prefill one request while decoding another), while no_buffer eliminates this buffer, reducing memory overhead at the cost of scheduling flexibility.
GPU memory hierarchy: Understanding why mamba state memory (rather than KV cache) became the bottleneck requires knowledge of how different model components consume GPU memory and how concurrent request capacity is determined.
Batch inference throughput dynamics: The relationship between concurrent requests, per-request decode speed, and aggregate throughput is non-linear. Adding more requests increases throughput up to a saturation point determined by memory bandwidth.
The broader project context: This data generation feeds into a DFlash drafter training pipeline, where dataset diversity and size directly impact model quality.

Output Knowledge Created

This message creates several concrete pieces of knowledge:

Quantified performance data: The no_buffer strategy achieves ~1,170 tok/s per GPU on RTX PRO 6000 Blackwell for Qwen3.6-27B, compared to ~670 tok/s with extra_buffer.
Aggregate throughput ceiling: 8 GPUs can sustain ~9,360 tok/s aggregate, exceeding the user's 25%-of-B200 target.
Capacity scaling: max_running_requests increased from 37 to 72 per GPU, demonstrating that mamba state memory was the binding constraint.
Verification methodology: The message demonstrates a pattern of measuring server-side metrics (gen throughput, running requests, token usage) alongside client-side progress (completed count, rate per second) to get a complete picture of pipeline health.

The Thinking Process Revealed

The subject message is the culmination of a reasoning chain that spans multiple rounds. The assistant's thinking, visible in the reasoning blocks of preceding messages, follows a clear pattern:

Observation: Throughput is below target (3.9K vs expected 6.25K tok/s).
Measurement: Query server metrics to identify the bottleneck.
Analysis: Recognize that mamba usage (0.61) is high while KV cache usage (0.20) is low, indicating the mamba scheduler strategy is the constraint.
Hypothesis formation: Switching from extra_buffer to no_buffer will free mamba state memory.
Implementation: Relaunch servers with the new flag.
Verification: Check max_running_requests to confirm capacity increase.
Re-verification: Run generation and measure throughput (the subject message). This is a textbook example of the scientific method applied to systems optimization. The assistant doesn't guess or try random flags—it measures, identifies the specific bottleneck, formulates a targeted fix, and verifies the result.

Conclusion

The subject message represents a successful optimization cycle in a complex ML infrastructure pipeline. By identifying that the extra_buffer mamba scheduler strategy was consuming memory that could otherwise host additional concurrent requests, the assistant nearly doubled per-GPU throughput from ~670 tok/s to ~1,170 tok/s. The aggregate throughput of ~9,360 tok/s exceeded the user's performance target and cut the estimated generation time by over 40%.

More importantly, the message illustrates a methodology: measure before optimizing, identify the specific bottleneck (not just "it's slow"), formulate a hypothesis about the root cause, implement a targeted change, and verify with both server-side and client-side metrics. This approach, visible across the conversation history leading to this message, is what separates effective systems optimization from guesswork.