Benchmarking the Qwen3.6-27B GDN Hybrid: Validating Production Performance on Modest Hardware

Introduction

In the sprawling ecosystem of large language model deployment, the gap between a model that loads successfully and a model that performs reliably under production conditions can be vast. Message 6895 in this opencode session represents the critical validation milestone for a Qwen3.6-27B deployment—the moment when weeks of infrastructure setup, driver installation, framework debugging, and configuration tuning crystallize into concrete numbers that answer the fundamental question: Does this actually work well enough?

The message is a benchmark results summary, delivered by the AI assistant after executing an extensive performance evaluation on a freshly deployed Qwen3.6-27B model running on two NVIDIA RTX A6000 GPUs. It is deceptively concise—a few tables and bullet points—but beneath that surface lies a rich story of architectural discovery, hardware constraints, framework evolution, and the relentless pursuit of production-ready performance.

The Message in Full

[assistant] ## Benchmark Results: Qwen3.6-27B on 2x RTX A6000 (kpro5)

**SGLang 0.5.11, TP=2, MTP (NEXTN steps=3, topk=1, draft=4), BF16**

### Throughput Sweep (input=1000, output=500)

| C | Agg tok/s | Per-req tok/s | TTFT(s) | Fail |
|---:|----------:|--------------:|--------:|-----:|
| 1 | 70.2 | 73.5 | 0.32 | 0 |
| 2 | 137.7 | 70.8 | 0.20 | 0 |
| 4 | 207.1 | 66.9 | 0.34 | 0 |
| 8 | 402.9 | 58.8 | 0.43 | 0 |
| 16 | 465.7 | 55.4 | 4.46 | 0 |
| 32 | 470.6 | 54.3 | 14.85 | 0 |
| 64 | 503.3 | 54.0 | 37.61 | 0 |
| 100 | 504.9 | 53.8 | 61.35 | 0 |

- **Peak aggregate**: ~505 tok/s at C=64-100
- **Single-request**: 73.5 tok/s with MTP (accept rate: 100%, accept len: 4.0)
- Aggregate saturates around C=32, graceful TTFT degradation after that

### Long Context (C=1, output=200, 128K context window)

| Input | Per-req tok/s | TTFT(s) |
|------:|--------------:|--------:|
| 1K | 73.6 | 0.39 |
| 10K | 73.1 | 2.59 |
| 30K | 71.5 | 6.17 |
| 60K | 69.2 | 10.09 |
| 100K | 66.3 | 15.07 |
| 120K | 64.9 | 9.81 |

- Decode speed barely degrades with context — **73.6 tok/s at 1K vs 64.9 tok/s at 120K** (only 12% drop). This is the GDN hybrid architecture advantage: 48 of 64 layers use linear attention with O(1) state, so KV cache is only needed for the 16 full-attention layers.
- TTFT scales roughly linearly with input length (~15s at 100K)
- **120K input works perfectly** on 2x A6000 with room to spare

Why This Message Was Written

The immediate trigger was a direct user request in message 6879: "Run benchmarks, 1-100 batch, then also for longer context 30-60-100k." This is a natural and essential step after any model deployment—the user needs to know what performance to expect before routing real traffic to the server.

But the deeper motivation goes beyond simply satisfying a request. This message represents a validation gate. The deployment of Qwen3.6-27B had been fraught with challenges. Earlier in the session, the assistant had discovered that SGLang 0.5.9 produced degenerate output with this model's GDN hybrid attention architecture, requiring an upgrade to 0.5.11. The MTP (Multi-Token Prediction) speculative decoding configuration required specific flags (--mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1) that weren't obvious from documentation. The model itself is a 27-billion-parameter BF16 behemoth weighing 52GB, shoehorned onto two 48GB A6000s with tensor parallelism—a configuration that left only ~6.5GB of free memory after model loading. Every aspect of this deployment was operating at the edge of feasibility, and the benchmarks were the final proof that the entire stack was functioning correctly.

The message also serves as a knowledge artifact for the broader project. The assistant is not just reporting numbers; it is building a performance profile that will inform future decisions about context window sizing, concurrency limits, and hardware requirements for similar deployments. The todo list at the bottom of the message marks all benchmark-related tasks as completed, signaling that this phase of work is definitively closed.

The Benchmarking Journey

To fully appreciate the summary message, one must understand the work that preceded it. The assistant did not simply run a single command and report results. The benchmarking unfolded across multiple messages (6880–6894) in a progressive, discovery-driven process.

First, the assistant created a custom benchmark script (bench_qwen36_kpro5.py) adapted from an earlier benchmark used for a different model. The script supported two modes: a throughput sweep across concurrency levels 1–100 with fixed input/output lengths, and a long-context mode that varied input length while measuring decode speed and time-to-first-token (TTFT).

The throughput sweep ran first (message 6884), producing the core numbers that would later appear in the summary. Critically, it reported zero failures across all concurrency levels—a strong signal that the server was stable under load.

The long-context benchmarking was more iterative. The server was initially configured with a 32K context window. The assistant ran tests at 1K, 5K, 10K, 20K, and 30K input lengths (message 6886), noting that the full token usage metric was only 0.12 (12%) at 30K—far lower than expected. This was the first clue that the GDN hybrid architecture's linear attention layers were dramatically reducing KV cache consumption.

Emboldened by this discovery, the assistant progressively expanded the context window: first to 65K (message 6888–6890), then to 128K (message 6891–6893). Each restart required stopping the server, editing the systemd service file, and waiting for the model to reload and warm up. At each step, the benchmarks confirmed that the model could handle the larger context without running out of memory. The final test at 120K input (message 6893) validated that the deployment could handle context lengths far beyond what the assistant initially thought possible.

This progressive expansion reveals a key aspect of the assistant's reasoning: it was operating under an initial assumption that long context would be memory-constrained on 2x A6000s, and it was pleasantly surprised to discover otherwise. The summary message in 6895 is where this discovery is consolidated and explained.

Decisions Made

Several important decisions are embedded in this message, some explicit and some implicit.

Benchmark parameter selection: The assistant chose input=1000, output=500 for the throughput sweep, and output=200 for the long-context tests. These are relatively short generation lengths, which means the throughput numbers are heavily influenced by prefill efficiency rather than pure decode speed. For a more complete picture, one would want to test longer outputs, but these parameters are reasonable for a quick validation benchmark and align with common benchmarking practices.

Concurrency levels: The sweep covered C=1, 2, 4, 8, 16, 32, 64, 100—a geometric progression that captures both the low-concurrency regime (where per-request throughput is maximized) and the high-concurrency regime (where aggregate throughput saturates). The choice to go up to 100 concurrent requests is aggressive for a 2-GPU setup and demonstrates confidence in the server's stability.

Context length progression: The decision to incrementally increase the context window from 32K to 65K to 128K was a real-time adaptive choice, not pre-planned. The assistant observed the low KV cache utilization at 30K and correctly inferred that the GDN architecture would allow much larger contexts. This is a good example of data-driven decision-making during the benchmarking process.

Presentation format: The assistant chose to present the data in two clean tables with clear headers, followed by interpretive bullet points. This is a deliberate choice to make the results accessible to the user while also providing the technical depth needed for informed decision-making. The inclusion of the Fail column (all zeros) is a subtle but important signal of reliability.

Assumptions and Their Validity

Every benchmark rests on assumptions, and this message is no exception.

The most significant assumption is that 1000 input tokens and 500 output tokens are representative of real usage. For a model deployed with tool-calling and reasoning capabilities, actual usage patterns may involve much longer reasoning chains (the assistant observed earlier that the model produced 1720 tokens for a simple prime-number question, most of which was reasoning content). If real workloads generate significantly more output tokens, the throughput numbers would shift, potentially changing the optimal concurrency configuration.

The assistant assumed that MTP speculative decoding with topk=1 and 4 draft tokens is the optimal configuration. The 100% accept rate and 4.0 accept length suggest that the draft tokens are always accepted, which is excellent—but it also means the benchmark is measuring the combined performance of the base model and drafter together. The "100% accept rate" with topk=1 is somewhat tautological: with only one candidate path, the verification step has no alternative to fall back to, so acceptance is guaranteed by construction. The meaningful question is whether the draft tokens match what the base model would have generated, and the benchmark doesn't directly measure this.

The assistant assumed that the benchmark script's methodology is sound. The script sends requests concurrently using Python threads, measures wall-clock time, and computes throughput as total tokens divided by total time. This is a reasonable approach but may not capture nuances like tail latency or the distribution of response times across requests.

Perhaps the most interesting assumption was the initial belief that 60K/100K context would OOM. In message 6886, the assistant wrote: "For 60K and 100K we'd need to restart with higher context, but that'll likely OOM on 2x A6000 with this model." This assumption was based on experience with standard transformer architectures where KV cache scales linearly with context length. The GDN hybrid architecture defied this expectation, and the assistant correctly updated its mental model mid-benchmark.

Input Knowledge Required

To fully understand this message, a reader needs knowledge spanning several domains.

Model architecture: The Qwen3.6-27B uses a GDN (Gated Dense Network) hybrid architecture with 64 layers, of which 48 use linear attention (with O(1) state) and 16 use full softmax attention. This is the key insight that explains the long-context performance—linear attention layers don't require KV cache, so the memory footprint grows much more slowly with context length than in a pure transformer.

Speculative decoding: MTP (Multi-Token Prediction) with NEXTN is a form of speculative decoding where a lightweight drafter model proposes multiple tokens per step, and the base model verifies them in parallel. The topk=1 and draft=4 parameters mean the drafter proposes a single 4-token path per step. The accept len: 4.0 means all 4 draft tokens are accepted on average.

Hardware context: Two RTX A6000 GPUs (48GB each) with tensor parallelism (TP=2), running on the kpro5 host. The model is in BF16 precision. The available memory after model loading was approximately 6.5GB, which is remarkably tight.

SGLang framework: The server runs SGLang 0.5.11 with specific configuration flags for MTP support. The --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1 environment variable were required workarounds discovered earlier in the session.

Benchmarking metrics: The reader must understand the distinction between aggregate tok/s (total throughput across all concurrent requests) and per-req tok/s (average throughput per individual request), as well as TTFT (time to first token) which measures prefill latency.

Output Knowledge Created

This message generates several important pieces of knowledge that extend beyond the immediate deployment.

Performance envelope: The model delivers 73.5 tok/s single-request throughput and peaks at ~505 tok/s aggregate at high concurrency. These numbers define the capacity of this hardware configuration and can be used for capacity planning.

Context scaling behavior: The decode speed degrades by only 12% from 1K to 120K context (73.6 → 64.9 tok/s), which is exceptional. For comparison, a pure transformer of similar size would typically see 50%+ degradation over this range due to KV cache overhead. This quantifies the GDN hybrid advantage in concrete terms.

Memory efficiency validation: The fact that 120K context works on 2x A6000 with "room to spare" is a non-obvious result that challenges conventional wisdom about memory requirements for long-context LLM inference.

Stability certification: Zero failures across all concurrency levels and context lengths provides strong evidence that the deployment is production-ready.

Configuration baseline: The specific SGLang version (0.5.11), MTP parameters, and server flags are documented as a working configuration that others can reproduce.

Anomalies and Nuances

A careful reader will notice an anomaly in the long-context table: the TTFT for 120K input (9.81s) is lower than for 100K input (15.07s). This is counterintuitive—longer inputs should take longer to prefill. Several explanations are possible: the 120K measurement may have benefited from KV cache reuse from the 100K test (if the server cached some prefix), or the measurement may have experienced less system load interference. The assistant did not comment on this anomaly, which is a minor oversight. In a rigorous benchmark, one would run multiple trials and report means or medians.

The "accept rate: 100%" deserves careful interpretation. With topk=1, the speculative decoding system proposes exactly one candidate path. The verification step accepts or rejects this single path—it cannot choose among alternatives. A 100% accept rate with topk=1 means the single proposed path was always accepted, but this is expected behavior when the drafter is well-calibrated. The more interesting question—how often does the drafter's proposal match what the base model would have generated?—is not directly measured. The accept length of 4.0 (all 4 draft tokens accepted on average) suggests good alignment, but this metric conflates drafter quality with the verification algorithm's leniency.

The assistant also presents the MTP configuration as a success story, but it's worth noting that the alternative—no speculative decoding—was not benchmarked for comparison. The 73.5 tok/s single-request throughput is with MTP enabled; the baseline without speculation might be meaningfully lower. The user cannot assess the actual speedup from MTP without this comparison.

The GDN Hybrid Architecture Advantage

The most significant insight in this message is the demonstration of the GDN hybrid architecture's practical advantages. The assistant's explanation—"48 of 64 layers use linear attention with O(1) state, so KV cache is only needed for the 16 full-attention layers"—is the key that unlocks understanding of the entire benchmark.

In a standard transformer, KV cache grows as 2 × n_layers × n_heads × d_head × seq_len × precision, which for a 27B model at 128K context would require tens of gigabytes per request. With only 16 attention layers requiring KV cache, the memory footprint is reduced by 75%. This is what makes 120K context feasible on 2x A6000 with only 6.5GB of free memory.

The decode speed degradation of only 12% from 1K to 120K is a direct consequence of this architecture. In a standard transformer, decode speed degrades because the attention computation grows with sequence length (O(n) per decode step). With 75% of layers using linear attention (O(1) per decode step), the overall attention cost scales much more gently.

This architectural insight transforms the benchmark from a simple performance report into a validation of a specific design choice. The numbers are not just "good for 2x A6000"—they are evidence that the GDN hybrid approach delivers on its promise of efficient long-context inference.

Conclusion

Message 6895 is a masterclass in concise technical communication. In a few dozen lines, it captures the culmination of an extensive deployment effort, distills complex performance data into actionable insights, and educates the reader about the architectural reasons behind the results. It serves as both a validation gate for the current deployment and a reference point for future capacity planning.

The message also reveals the assistant's growth during the benchmarking process—from initial skepticism about long-context feasibility to confident understanding of the GDN architecture's memory efficiency. This learning journey, compressed into the gap between the initial assumption ("likely OOM") and the final conclusion ("room to spare"), is perhaps the most valuable knowledge artifact in the entire exchange.

For anyone deploying large language models on constrained hardware, this message offers a concrete data point and a methodological template. It demonstrates that thorough benchmarking requires not just running scripts, but actively interpreting results, challenging assumptions, and adapting the evaluation to the architecture's unique characteristics. The numbers themselves will age as hardware improves, but the approach—systematic, curious, and grounded in architectural understanding—remains timeless.