The Bridge Between Deployment and Measurement: Rewriting a Benchmark for vLLM's API

In the sprawling, multi-threaded narrative of deploying Qwen3.6-27B with DFlash speculative decoding on a pair of RTX A6000 GPUs, message [msg 6957] occupies a deceptively modest footprint. It consists of just two lines of assistant text and a file-write confirmation:

[assistant] Let me rewrite the benchmark to use the OpenAI-compatible /v1/completions endpoint which works with both vLLM and SGLang: [write] /home/theuser/glm-kimi-sm120-rtx6000bw/bench_qwen36_vllm.py Wrote file successfully.

Beneath this brevity lies a critical inflection point. The assistant has just spent dozens of messages wrestling DFlash speculative decoding into a working state on vLLM 0.20.1 — resolving flash-attn version conflicts, crafting a drafter config from scratch, debugging a near-zero acceptance rate, and finally verifying coherent output with proper reasoning content ([msg 6953]). The deployment works. But without a benchmark, "works" is an anecdote, not a data point. The entire speculative decoding effort — the hours of debugging, the unmerged PRs, the config crafting — is aimed at one measurable goal: throughput improvement over the MTP baseline. The benchmark is the instrument that will answer whether that goal has been achieved.

The API Mismatch That Made a New File Necessary

The immediate trigger for this message is a failure that occurred just moments earlier. In [msg 6955], the assistant attempted to run the existing benchmark script, bench_qwen36_kpro5.py, against the newly launched vLLM server. The result was immediate and unambiguous:

Warming up...
  Warmup: 0 tokens, ok=False
  Error: HTTP Error 404: Not Found

The 404 error was not a server failure — vLLM was running and healthy, as confirmed by the smoke test in [msg 6953]. The problem was an API endpoint mismatch. The existing benchmark script, written for the earlier SGLang-based deployment, used the /generate endpoint. SGLang exposes this custom endpoint for text generation, but vLLM follows the OpenAI-compatible API specification, which uses /v1/completions (for raw completions) and /v1/chat/completions (for chat-style interactions). The old script was sending requests to a route that simply didn't exist on the new server.

This is a classic problem in the ML serving ecosystem: despite convergence toward OpenAI-compatible APIs, each framework still has its own idiosyncrasies. SGLang's /generate endpoint, vLLM's /v1/completions, TGI's /generate_stream — they all serve the same purpose but speak different dialects. When the assistant migrated from SGLang to vLLM (a necessary move to access DFlash speculative decoding, which is only available in vLLM), the benchmarking infrastructure needed to follow.

Reading Before Writing: Understanding the Existing Code

Before writing the new benchmark, the assistant took a deliberate step that reveals its engineering methodology. In [msg 6956], it read the existing bench_qwen36_kpro5.py file. This is a crucial detail: the assistant did not write the new benchmark from a blank slate or from first principles. It studied the existing code to understand its structure — the concurrency sweep logic, the input/output length control, the warmup procedure, the result formatting — and then adapted that structure to the new API endpoint.

The read operation reveals the assistant's assumption that the benchmark's logic was sound and reusable; only the transport layer needed changing. The concurrency sweep pattern (testing throughput at concurrency levels 1, 2, 4, 8, 16, 32) and the token-counting methodology were worth preserving. The assistant recognized that the old script's architecture — making concurrent requests, measuring wall time, computing aggregate tokens per second — was framework-agnostic. The only SGLang-specific part was the URL path and request format.

The OpenAI-Compatible Choice

The assistant's decision to target /v1/completions rather than /v1/chat/completions is itself noteworthy. The smoke test in [msg 6953] had used the chat endpoint successfully, producing a full reasoning + content response. But for benchmarking, the assistant chose the simpler completions endpoint. This reflects an understanding that benchmarking raw throughput is best done without the overhead of chat-template processing, tool-call parsing, or multi-turn state management. The completions endpoint accepts raw token inputs and returns raw token outputs — a cleaner signal for measuring decode speed.

The phrase "which works with both vLLM and SGLang" in the assistant's message reveals another layer of strategic thinking. By targeting the OpenAI-compatible endpoint, the new benchmark is not just a vLLM adapter — it's a portable measurement tool. If the deployment later switches back to SGLang (or if a future comparison requires testing both frameworks), the same benchmark script will work without modification. This is a small but meaningful investment in infrastructure reusability.

What the New Benchmark Enables

The file written in this message, bench_qwen36_vllm.py, is the instrument that will quantify the entire DFlash integration effort. In the subsequent message ([msg 6958]), the assistant runs it and begins to see results:

=== Throughput Sweep: input=1000, output=500 ===
   C |  Agg tok/s |  Per-req tok/s |  TTFT(s) |   Tokens |  Wall(s) |   Ok | Fail
-------------------------------------------------------------------------------------
   1 |       17.7 |           18.1 |     0.64 |     2000 |    112.8 |    4 |    0
   2 |       30.4 |           16.0 |     1.49 |     2000 |     65.7 |    4 |    0

These numbers — 17.7 tok/s at concurrency 1, 30.4 tok/s at concurrency 2 — are the first quantitative data points for DFlash performance on this hardware. They are low compared to the MTP baseline (73.5 tok/s single-request from earlier SGLang benchmarks), which immediately signals that DFlash's acceptance rate is poor. This data will drive the next phase of investigation: why is the drafter being rejected so often? Is it the layer-ID offset bug? The sliding window attention issue? Or a fundamental drafter quality problem?

Without the benchmark written in this message, those questions would remain unasked. The assistant would have a working deployment but no way to measure whether it was an improvement. The benchmark transforms "it runs" into "it runs at X tok/s," which is the difference between a demo and a deployable system.

The Broader Pattern: Infrastructure Follows Architecture

This message exemplifies a recurring pattern in the opencode session: every architectural change requires corresponding infrastructure changes. When the model changed from GLM to Qwen, the serving configuration changed. When the serving framework changed from SGLang to vLLM, the benchmark changed. When the GPU topology changed, the tensor parallelism settings changed. Each layer of the system — model, serving framework, hardware, measurement — is coupled to the others, and changing one layer ripples through the rest.

The assistant's response to the 404 error is not to patch the old script with a URL replacement, but to write a new file. This is a deliberate choice that favors clarity over minimalism. A new file with a new name (bench_qwen36_vllm.py vs bench_qwen36_kpro5.py) makes the framework dependency explicit in the filename. It preserves the old script for potential SGLang comparisons. It avoids the complexity of conditional logic that would branch on the serving framework. The cost is a few extra lines of code; the benefit is a cleaner separation of concerns.

Conclusion

Message [msg 6957] is, on its surface, a routine engineering action: write a file. But in the context of the broader session, it is the moment when the assistant bridges the gap between "the deployment works" and "the deployment performs." The benchmark written here is the measurement instrument that will drive the next phase of debugging and optimization. It embodies the assistant's understanding that in ML systems engineering, deployment and measurement are not separate phases — they are interleaved, and each requires infrastructure that must be actively maintained as the system evolves. The 404 error that triggered this message was not a setback; it was a signal that the infrastructure had fallen out of sync with the architecture, and the assistant's response was to bring them back into alignment.