Benchmarking the Beast: Measuring Throughput for Large-Scale LLM Generation on Blackwell GPUs
Introduction
In the high-stakes world of large language model training, data quality is everything. When a team discovers that 87% of their meticulously curated 914K-sample training dataset contains essentially empty responses—just six tokens of "thinking\n\nresponse\nOK.<|im_end|>"—the only option is to start over. This is the exact predicament faced in Segment 44 of this opencode session, and it sets the stage for one of the most technically dense messages in the conversation: a throughput benchmark that would determine the fate of an entire training pipeline.
Message 7462 captures a pivotal moment where the assistant, having just verified that Qwen3.6-27B can generate proper thinking-mode completions on a single GPU, pivots to the critical question: how fast can we do this at scale? The answer would determine whether the team could regenerate 914K completions on their existing 4× RTX PRO 6000 Blackwell node, or whether they would need to provision alternative hardware—a decision with significant time and cost implications.
The Crisis That Led to This Moment
To understand why message 7462 matters, we need to step back. The team had been building a DFlash (Drafting with Flash Attention) speculative decoding system for Qwen3.6-27B. The core idea was elegant: train a lightweight drafter model that predicts hidden states from the target model, enabling faster autoregressive generation through speculative decoding. The training pipeline required hidden state extractions from the target model processing real-world prompts.
The original dataset consisted of 913,786 prompts drawn from ShareGPT conversations, already tokenized and stored in S3. But when the team finally inspected the tokenized data, they discovered a catastrophe: 87% of samples had a loss_mask sum of exactly 6 tokens. The responses were essentially empty—just the model saying "OK" without any meaningful content. The hidden state extraction pipeline, which had been running for days and producing 645 GB of data, was worthless for DFlash training because the model hadn't actually generated anything useful.
The only fix was to regenerate all 902,087 completions using Qwen3.6-27B with thinking mode enabled, ensuring the model produced rich reasoning traces and substantive responses. But this meant running the model over nearly a million prompts, each requiring thousands of tokens of generation. The question of where to run this generation—and how long it would take—became the central engineering challenge.
Message 7462: The Benchmark Begins
Message 7462 is an assistant message that arrives after a series of preparatory steps: killing the old extraction pipeline, installing SGLang 0.5.11, launching a test server on GPU 0, and verifying that a single request produces proper thinking-mode output. The assistant now faces a fork in the road. Let's examine the message in full:
## Agent Reasoning Good - the generation is working: - 26 input tokens → 1115 output tokens (2540 chars reasoning + 1167 chars content) - The thinking is ~600 tokens, the content is ~500 tokens
>
Now I need to: 1. Upload the prompts to the training machine 2. Run a throughput benchmark with concurrent requests 3. Set up all 4 SGLang instances
>
Let me start the upload of prompts and run the benchmark in parallel. Generation works with thinking. Let me upload prompts and run the throughput benchmark in parallel:
>
[bash] # Upload prompts to training machine ssh -p 19248 root@154.59.156.20 'mkdir -p /workspace/dflash/data/raw_prompts' scp -P 19248 /data/dflash/q36-27b/raw_prompts/all_prompts_sharegpt.jsonl root@154.59.156.20:/workspace/dflash/data/raw_prompts/ 2>&1 (no output)
>
[bash] # Throughput benchmark on GPU0 - test at various concurrency levels ssh -p 19248 root@154.59.156.20 ' /workspace/dflash/venv/bin/python3 -c " import asyncio, aiohttp, time, json, sys
>
SERVER = \"http://localhost:30000\" PROMPTS = [ \"Write a Python function to merge two sorted lists efficiently.\", \"Explain the difference between TCP and UDP with code examples.\", \"Create a React component for a modal dialog with animations.\", \"What is the time complexity of quicksort? Prove it.\", \"Write a bash script that monitors disk usage and sends alerts.\", \"Implement a binary search tree with insert, delete, and search.\", \"How does garbage collection work in Python? Give detailed examples.\", \"Write a Dockerfile for a FastAPI application with multi-stage build.\", ]
>
async def send_one(session, sem, prompt, max_tokens=2048): payload = { \"model\": \"default\", \"messages\": [{\"role\": \"user\", \"content\": prompt}], \"max_tokens\": max_tokens, \"temperature\": 0.6, \"top_p\": 0.95, \"stream\": False, } async with sem: t0 = time.time() async with session.post(f\"{SERVER}/v1/chat/completions\", json=payload, timeout=aiohttp.ClientTimeout(total=300)) as r: data = await r.json() dt = time.time() - t0 u = data[\"usage\"] return u[\"prompt_tokens\"], u[\"completion_tokens\"], dt
>
async def bench(concurrency, n_requests): sem = asyncio.Semaphore(concurrency) conn = aiohttp.TCPConnector(limit=concurrency, limit_per_host=concurrency) async with aiohttp.ClientSession(connector=conn) as session: t0 = time.time() tasks = [send_one(session, sem, PROMPTS[i%len(PROMPTS)]) for i in range(n_requests)] results = await asyncio.gather(*tasks, return_exceptions=True) total_time = time.time() - t0
>
ok = [r for r in results if not isinstance(r, Exception)] errs = len(results) - len(ok) total_in = sum(r[0] for r in ok) total_out = sum(r[1] for r in ok) avg_out = total_out / len(ok) if ok else 0 out_tps = total_out / total_time req_ps = len(ok) / total_time
>
print(f\"C={concurrency:3d} n={n_requests:3d} | {len(ok)} ok {errs} err | \" f\"out_tok/s={out_tps:7.1f} req/s={req_ps:5.2f} | \" f\"avg_out={avg_out:6.0f} total_out={total_out:7d} | {total_time:.1f}s\") return out_tps
>
print(\"Benchmarking single GPU throughput...\") print(\"=\"*100)
>
# Warm up asyncio.run(bench(1, 1)) print(\"--- Warmup done ---\")
>
# Test various concurrency levels for c in [1, 4, 8, 16, 32]: n = max(c 2, 8) asyncio.run(bench(c, n)) print(\"=\"100) " 2>&1 ' 2>&1
Anatomy of the Agent's Reasoning
The agent reasoning section reveals a clear, methodical thought process. The assistant has just received results from a single-request test (message 7461) showing that a 26-token prompt produces 1115 output tokens with rich reasoning content. The assistant correctly interprets this as validation that the generation pipeline works end-to-end.
The reasoning then identifies three parallel workstreams: uploading prompts, benchmarking throughput, and setting up all four SGLang instances. This is a classic optimization pattern in distributed systems—identify independent work that can proceed concurrently to minimize wall-clock time. The assistant chooses to upload prompts and run the benchmark simultaneously, which is sensible since the benchmark doesn't depend on the prompts being present (it uses hardcoded test prompts).
What's notable is what the reasoning doesn't say. There's no explicit calculation of required throughput or estimation of total generation time. The assistant seems to be operating under the assumption that the benchmark will reveal whether the Blackwell node is fast enough, and the decision about whether to use it or seek alternatives will follow from the data. This is a data-driven approach: measure first, decide second.
The Benchmark Script: A Technical Deep Dive
The benchmark script embedded in the bash command is a compact but sophisticated piece of engineering. Let's examine its design choices.
Asyncio and aiohttp: The script uses Python's asyncio framework with the aiohttp library for asynchronous HTTP requests. This is the correct choice for benchmarking an LLM inference server, where the bottleneck is network I/O waiting for the server to complete generation. A synchronous approach would waste CPU cycles polling for responses and wouldn't accurately measure the server's capacity under concurrent load.
Semaphore-based concurrency control: The asyncio.Semaphore(concurrency) pattern ensures that exactly concurrency requests are in flight at any time. When a request completes, the semaphore releases a slot, allowing the next queued request to proceed. This is more realistic than simply firing all requests at once, which could overwhelm the server and produce misleading results.
TCP connector tuning: The script creates a TCPConnector with limit=concurrency and limit_per_host=concurrency. This prevents the HTTP client from opening more connections than the intended concurrency level, which would defeat the purpose of the semaphore. It's a subtle but important detail that shows attention to measurement accuracy.
Warm-up phase: The script runs a single warm-up request before the real benchmarks. This is crucial for LLM serving because the first request often incurs cold-start costs: CUDA graph compilation, KV cache initialization, and model warm-up. Without the warm-up, the first benchmark iteration would be polluted by one-time startup overhead.
Multi-level concurrency sweep: The script tests concurrency levels of 1, 4, 8, 16, and 32. This sweep is designed to find the saturation point—the concurrency level where throughput stops increasing and may even decrease due to contention for GPU memory, KV cache slots, or Mamba state cache. The choice of powers of two plus 32 (a common maximum for GPU memory-constrained deployments) is standard practice in systems benchmarking.
Metrics collection: For each run, the script records output tokens per second (out_tok/s), requests per second (req/s), average output length, and error count. The focus on output tokens per second is appropriate because the generation task is output-heavy—the model needs to produce long reasoning traces followed by responses, and the input prompts are relatively short.
What the Benchmark Reveals (and What It Doesn't)
The benchmark results are not shown in message 7462 itself—they would appear in the next message after the commands complete. However, from the chunk summary we know the outcome: the Blackwell node achieved approximately 400 tok/s per GPU with MTP (speculative decoding) and hierarchical cache enabled. This translates to roughly 1,600 tok/s across all 4 GPUs with data parallelism.
But this benchmark has significant limitations that the assistant either doesn't acknowledge or chooses to ignore for expedience:
Prompt diversity: The benchmark uses only 8 short, generic coding prompts. The actual dataset contains 914K prompts spanning tool calls, multi-turn conversations, creative writing, and open-ended questions. Real prompts vary wildly in length (from a few tokens to thousands), topic, and complexity, all of which affect generation speed. The benchmark's homogeneous prompt set likely overestimates throughput.
Output length distribution: The benchmark requests max_tokens=2048 for every prompt, but the actual generation task has a variable output length distribution. The assistant's earlier test produced 1115 output tokens from a 26-token input, but real prompts may produce shorter or much longer outputs. The benchmark doesn't model this distribution.
No speculative decoding: The chunk summary mentions that the final configuration used MTP (Multi-Token Prediction) speculative decoding, which provides 12-45% throughput improvement. But this benchmark runs without MTP enabled. The assistant's reasoning mentions "Set up all 4 SGLang instances" as a future step, suggesting MTP would be configured later. This means the benchmark results understate the achievable throughput.
Single GPU only: The benchmark tests only GPU 0, not all 4 GPUs in parallel. The assistant assumes linear scaling with data parallelism, but real-world scaling is rarely perfect. Communication overhead, load imbalance, and NUMA effects can reduce scaling efficiency.
No memory pressure testing: The benchmark doesn't test with long prompts that would stress the KV cache or Mamba state cache. The earlier server startup log showed max_running_requests=16 despite the --max-running-requests 128 flag, indicating memory pressure was already a constraint. The benchmark's short prompts don't trigger this bottleneck.
The Broader Impact
The benchmark results from message 7462 fed directly into a critical decision. With ~400 tok/s per GPU on the Blackwell node, the team calculated that generating 902K completions would take approximately 16.5 days—and that's assuming perfect linear scaling across 4 GPUs. This was unacceptably slow, especially since it would block the GPUs from their primary purpose: training the DFlash drafter.
This realization triggered the pivot to B200 NVL hardware. The team provisioned a 7× B200 NVL node (183 GB each, NVLink mesh) that could deliver an estimated 15,000–30,000 tok/s at roughly the same cost per token, cutting wall time to 1-2 days. The B200 generation run ultimately succeeded, producing 902,087 completions with full Qwen3.6-27B thinking traces (1.64B output tokens, 7.25 GB in S3).
But the story doesn't end there. The generated data led to another discovery: storing hidden states offline would require approximately 90 TB of storage (5 layers × 5120 hidden dimensions × BF16 × 2000 average tokens × 902K samples). This was completely impractical, forcing yet another pivot to an online training architecture where hidden states are extracted on-the-fly during the target model forward pass and fed directly to the drafter, eliminating storage entirely.
Conclusion
Message 7462 captures a moment of technical decision-making that exemplifies the iterative, data-driven nature of ML engineering. A single benchmark—carefully designed with async concurrency control, warm-up phases, and multi-level throughput sweeps—provided the data needed to make a multi-million-dollar infrastructure decision. The assistant's methodical approach, from verifying single-request quality to measuring throughput at scale, reflects the engineering discipline required to build production ML systems.
The benchmark also reveals the assumptions and blind spots that inevitably accompany such measurements. The homogeneous prompt set, the absence of speculative decoding, the single-GPU limitation, and the lack of memory pressure testing all represent potential sources of error. Yet the benchmark was "good enough" to inform the decision: the Blackwell node was too slow, and the B200 alternative was worth pursuing.
In the end, the benchmark from message 7462 didn't just measure throughput—it shaped the entire trajectory of the DFlash training pipeline, triggering a cascade of architectural decisions that would ultimately lead to a more elegant online training design. Sometimes the most valuable thing a benchmark can tell you is that you're asking the wrong question about hardware, and the right answer involves a completely different approach.