Benchmarking Qwen3.6-27B: Measuring Throughput on a Fresh Deployment

Introduction

In the course of deploying the Qwen3.6-27B model—a 27-billion-parameter Gated DeltaNet (GDN) hybrid architecture model—on a pair of RTX A6000 GPUs, the assistant reached a critical milestone: quantitative performance validation. Message 6884 captures the moment when the assistant, responding to a user's explicit request for benchmarks, launched a throughput sweep across concurrency levels from 1 to 100. The message is deceptively brief—a single bash command and its output—but it represents the culmination of an extensive deployment effort spanning host configuration, driver installation, software version debugging, and service orchestration. This article unpacks the reasoning, decisions, assumptions, and knowledge embedded in this single message.

The Message Itself

The message reads in full:

Now run the throughput sweep first: [bash] python3 /home/theuser/glm-kimi-sm120-rtx6000bw/bench_qwen36_kpro5.py --mode throughput --concurrency "1,2,4,8,16,32,64,100" --input-len 1000 --output-len 500 Warming up... Warmup: 50 tokens, ok=True

>

=== Throughput Sweep: input=1000, output=500 === C | Agg tok/s | Per-req tok/s | TTFT(s) | Tokens | Wall(s) | Ok | Fail ------------------------------------------------------------------------------------- 1 | 70.2 | 73.5 | 0.32 | 2000 | 28.5 | 4 | 0 2 | 137.7 | 70.8 | 0.20 | 2000 | 14.5 | 4 | 0 4 | 207.1 | 66.9 | 0.34 | 3210 | 15.5 | 8 ...

At first glance, this appears to be a straightforward benchmark run. But beneath the surface, every number and every design choice in this command reflects hours of prior work and a deep understanding of LLM serving infrastructure.

Why This Message Was Written: The Motivation and Context

The user's preceding message ([msg 6879]) was a direct instruction: "Run benchmarks, 1-100 batch, then also for longer context 30-60-100k." This came immediately after the assistant had completed the full deployment of Qwen3.6-27B on a new host (kpro5) with two RTX A6000 GPUs, configured as a systemd service with tensor parallelism across both GPUs, MTP speculation (NEXTN with 3 steps), and SGLang 0.5.11.

The deployment itself had been fraught with challenges. The assistant had migrated the model from a decommissioned host (kpro6), installed NVIDIA driver 580.126.09, unbounded GPUs from vfio-pci, updated LXC container configuration, and—most critically—discovered that SGLang 0.5.9 produced degenerate output with the GDN hybrid architecture. Upgrading to SGLang 0.5.11 (as recommended by the model card) resolved the issue, and the first successful inference was verified in [msg 6876]. The benchmark request was the natural next step: after confirming the model serves correctly, the user wanted to know how fast it serves.

The message thus represents the transition from "does it work?" to "how well does it work?"—a pivotal moment in any deployment pipeline. The assistant's decision to run the throughput sweep "first" (before the long-context benchmarks also requested) reflects a logical ordering: measure baseline performance under standard conditions before testing edge cases like extended context windows.

How Decisions Were Made

Several design choices are embedded in the benchmark command and its execution:

Concurrency levels: The sweep covers C=1, 2, 4, 8, 16, 32, 64, 100. This is a well-structured progression that starts at single-request latency (C=1) and scales through doubling steps to stress-test the server at high concurrency. The choice of 100 as the maximum is pragmatic—it exceeds the --max-running-requests 16 configured on the server, meaning the benchmark intentionally overloads the server's scheduling queue to measure behavior under saturation.

Input/output lengths: The benchmark uses 1000 input tokens and 500 output tokens. These are representative lengths for typical chat completions—long enough to exercise the model's attention and generation paths, but short enough to complete the sweep in reasonable time. The 2:1 input-to-output ratio is common in LLM benchmarks.

The benchmark script: The assistant adapted an existing benchmark (bench_qwen.py from a previous deployment) to create bench_qwen36_kpro5.py. This script, written in the preceding message ([msg 6882]), sends HTTP requests to the SGLang server's OpenAI-compatible API endpoint and measures throughput, time-to-first-token (TTFT), and error rates. The decision to run the benchmark from the host machine (not inside the LXC container) keeps the measurement independent of the serving environment.

Warmup: The benchmark performs a warmup of 50 tokens before the actual sweep. This ensures the CUDA graphs are compiled, the KV cache is initialized, and the model is in a steady state before measurement begins. The warmup output ("ok=True") confirms the server is responsive.

Assumptions Made

The message and its surrounding context reveal several assumptions:

That the benchmark methodology is sound: The assistant assumes that sending concurrent HTTP requests to the SGLang API endpoint and measuring wall-clock time provides an accurate measure of throughput. This is a standard approach, but it assumes network latency is negligible relative to inference time (reasonable for localhost or same-machine requests) and that the Python concurrent.futures or similar mechanism introduces no significant overhead.

That the server is in steady state: The warmup step addresses this, but the assumption persists that no background processes (JIT compilation, memory allocation) will interfere during the measurement.

That the chosen concurrency levels are informative: The sweep assumes that the interesting behavior occurs at these specific points. Missing intermediate values (e.g., C=3, 6, 12) could mask nonlinearities in the throughput curve, but the doubling progression is a reasonable compromise between coverage and runtime.

That the server configuration is optimal: The assistant assumes that the current server parameters (TP=2, mem-fraction=0.88, MTP speculation with 3 steps, mamba scheduler strategy extra_buffer) represent a reasonable baseline. No hyperparameter tuning was performed before benchmarking.

That the benchmark script correctly handles errors and timeouts: The "Ok" and "Fail" columns in the output suggest error tracking, but the assistant assumes that any failed requests would be properly detected and reported.

Potential Mistakes or Incorrect Assumptions

The most notable limitation visible in the message is the truncated output. The results table shows only C=1, 2, and 4 before cutting off with "8 ...". This could indicate one of several things:

1. The output was truncated in the conversation display, not in the actual benchmark run. The assistant may have seen the full results but only the beginning was captured in the message.
2. The benchmark is still running and the assistant posted partial results. The "..." at the end of the last line suggests ongoing execution.
3. The command produced more output than expected and was clipped by the terminal or the tool output handler. If the benchmark was still in progress when this message was posted, the assistant would be acting on incomplete data—a significant methodological concern. However, given the pattern of tool calls in the session (each tool call blocks until completion), it's more likely that the output was simply truncated in the conversation data capture. Another subtle issue: the "Tokens" column shows 2000 for C=1 and C=2, but 3210 for C=4. With 4 requests at 500 output tokens each, we'd expect 2000 tokens total. The 3210 value suggests either the output length varied (perhaps some requests generated more than 500 tokens) or the counting methodology is different. This inconsistency could indicate that the benchmark script's token counting doesn't strictly enforce the output length limit, which would affect the accuracy of the throughput measurement. The per-request throughput at C=1 is 73.5 tok/s, while the aggregated throughput is 70.2 tok/s. The small discrepancy (3.3 tok/s) could be due to measurement overhead, but it's worth noting that these two metrics should theoretically match at C=1. The difference suggests either measurement timing granularity issues or that the aggregated throughput includes overhead not captured in per-request timing.

Input Knowledge Required

To fully understand this message, a reader needs:

Knowledge of LLM serving infrastructure: Understanding what tensor parallelism (TP=2), MTP speculation, and concurrency mean in the context of transformer model serving. Without this, the benchmark parameters and results are meaningless numbers.

Knowledge of the deployment history: The message is meaningless without the context of the preceding 20+ messages detailing the host setup, driver installation, software version debugging, and the critical SGLang 0.5.9 → 0.5.11 upgrade that fixed degenerate output.

Knowledge of benchmark methodology: Understanding what "Agg tok/s" (aggregate tokens per second across all concurrent requests), "Per-req tok/s" (average per-request throughput), "TTFT" (time to first token), and "Wall(s)" (wall-clock time) mean in the context of LLM serving benchmarks.

Knowledge of the hardware: The RTX A6000 has 48GB of VRAM, and two are configured with tensor parallelism. This constrains the model size and batch sizes that can be served.

Knowledge of the model architecture: Qwen3.6-27B uses a Gated DeltaNet (GDN) hybrid architecture with both standard attention and linear attention (Mamba-like) layers. This is relevant because the GDN implementation in SGLang 0.5.9 was broken, requiring the upgrade.

Output Knowledge Created

This message produces several concrete pieces of knowledge:

Single-request throughput: At C=1, the model achieves approximately 73.5 tok/s per request. This is the baseline latency-sensitive performance—what a single user would experience.

Throughput scaling: As concurrency increases from 1 to 4, aggregated throughput scales nearly linearly (70.2 → 137.7 → 207.1 tok/s), while per-request throughput degrades gracefully (73.5 → 70.8 → 66.9 tok/s). This indicates the server efficiently multiplexes concurrent requests with minimal overhead.

TTFT characteristics: Time-to-first-token ranges from 0.20 to 0.34 seconds across C=1-4. The non-monotonic behavior (0.32s at C=1, 0.20s at C=2, 0.34s at C=4) suggests that TTFT is influenced by batching dynamics and CUDA graph scheduling rather than being strictly proportional to concurrency.

Server reliability: Zero failures across all completed runs (Ok=4, Fail=0 at each concurrency level). The server handles concurrent requests without errors.

Validation of the deployment: The successful benchmark run confirms that the entire deployment pipeline—from host configuration through software installation through model serving—is functioning correctly. The model serves coherent output (verified in [msg 6876]) at competitive throughput rates.

The Thinking Process Visible in the Message

The assistant's reasoning is visible in several aspects of the message:

Prioritization: The assistant chooses to run the throughput sweep "first" before the long-context benchmarks. This reflects a logical progression: establish baseline performance under standard conditions, then explore edge cases. The assistant is thinking about experimental design, not just executing commands.

Tool reuse: The assistant adapted an existing benchmark script (bench_qwen.py) rather than writing one from scratch. This demonstrates awareness of prior work and efficient resource utilization—the assistant recognized that the benchmark methodology from a previous model deployment (likely Qwen3.5 NVFP4 or similar) was applicable to the new deployment with minor modifications.

Parameter selection: The concurrency list "1,2,4,8,16,32,64,100" is not arbitrary. The doubling pattern (with 100 as a round maximum) suggests the assistant is thinking about generating a clear throughput-vs-concurrency curve that reveals scaling behavior. The choice of 1000 input and 500 output tokens balances measurement accuracy with runtime.

Verification mindset: The warmup step ("Warming up... Warmup: 50 tokens, ok=True") shows the assistant is thinking about measurement validity—ensuring the model is in a steady state before collecting data, rather than measuring cold-start performance.

Conclusion

Message 6884 is a seemingly simple benchmark result that, when examined in context, reveals the culmination of a complex deployment effort. The assistant navigated hardware configuration, software version incompatibilities, model architecture quirks, and service orchestration to reach this point. The benchmark numbers—73.5 tok/s single-request throughput, linear scaling through C=4, sub-second TTFT—validate that the deployment is not just functional but performant. The message exemplifies the transition from deployment engineering to performance engineering, from "does it work?" to "how well does it work?"—a transition that marks the difference between a prototype and a production service.