The Clean Slate: How Deleting Invalid Benchmark Results Saved an ML Evaluation Pipeline

In the middle of a complex benchmarking session for speculative decoding on 8× RTX PRO 6000 Blackwell GPUs, a single bash command appeared that, on its surface, looks trivial: a file deletion. But this moment — message <msg id=11267> in the conversation — represents a critical inflection point where rigorous experimental methodology triumphed over the temptation to accept plausible-looking but fundamentally flawed data.

The Message

The assistant executed:

# Delete all failed results so they re-run
rm -f bench_results/tp1-linear.json bench_results/tp1-b8.json bench_results/tp1-b12.json bench_results/tp1-b15.json bench_results/tp1-b16.json bench_results/tp1-b32.json bench_results/tp1-b64.json
ls bench_results/

Output:

tp1-auto.json

Seven JSON files, each representing a benchmark configuration for the Qwen3.6-27B model with DFlash and DDTree speculative decoding, were deleted in a single stroke. Only tp1-auto.json survived — a result from a different configuration that was not affected by the underlying issue.

Why This Message Was Written: The Discovery of Invalid Data

To understand why this deletion was necessary, we must look at the reasoning that immediately preceded it in <msg id=11266>. The assistant had been debugging a perplexing failure: the SGLang DFlash service was crashing during startup when configured with max_running_requests=8, but worked perfectly with max_running_requests=4. Both configurations used the same mem_fraction_static=0.75 and mamba_full_memory_ratio=0.5 parameters.

By carefully comparing the memory allocation logs from the working and failing runs, the assistant uncovered a critical insight. The working service (with max_running_requests=4) allocated max_mamba_cache_size: 24 — that is, 24 Mamba state slots — while the failing service (with max_running_requests=8) only allocated max_mamba_cache_size: 2. This was deeply counterintuitive: one would expect more concurrent requests to require more Mamba cache slots, not fewer.

The assistant's reasoning traced the problem to the intermediate_ssm_state_cache, which stores intermediate Mamba states across multiple tokens or steps during speculative decoding. In the working configuration, each slot consumed approximately 0.63 GB of memory. In the failing configuration, each slot consumed roughly 1 GB — and crucially, the intermediate state cache was vastly larger per slot (1.125 GB/slot vs 0.469 GB/slot). This suggested that the max_running_requests parameter influenced how SGLang calculated the per-slot memory requirements for the Mamba cache, possibly through interaction with the mamba_scheduler_strategy='extra_buffer' setting or the DDTree algorithm's own memory calculations.

The consequence was catastrophic for the benchmarks that had already been run: with max_running_requests=8, the Mamba cache was squeezed to only 2 slots, and the KV cache was limited to a mere 11,893 tokens. The working configuration, by contrast, supported 24 Mamba slots and 87,061 KV tokens. Any throughput measurements taken under the broken configuration were not just slightly off — they were fundamentally invalid, reflecting an artificially constrained system that could not properly exercise the speculative decoding pipeline.

The Decision Process: A Methodical Pivot

The assistant's thinking in <msg id=11266> reveals a careful, multi-step decision process. First came the diagnostic phase: comparing memory allocation logs line by line, calculating per-slot memory costs, and identifying the max_running_requests parameter as the root cause. Then came the trade-off analysis: simply matching the working configuration (max_running_requests=4) would prevent testing at higher concurrency levels, which was one of the primary goals of the benchmark plan. The assistant recognized this tension and resolved to start with the conservative setting, validate that benchmarks produce correct results, and then incrementally push the concurrency higher while monitoring memory pressure.

The deletion in <msg id=11267> is the execution of that decision. Rather than attempting to salvage the existing results — perhaps by annotating them as "run with broken config" or trying to post-process them — the assistant chose a clean break. This is a hallmark of rigorous experimental practice: when you discover a systematic error in your measurement setup, you discard the contaminated data and start fresh. The comment in the command itself — "Delete all failed results so they re-run" — makes this intention explicit.

Assumptions and Input Knowledge

This message rests on several critical assumptions. First, that the configuration error (max_running_requests=8 instead of 4) was indeed the sole cause of the benchmark invalidity, and that no other latent issues would corrupt the re-runs. Second, that the corrected configuration would produce stable, meaningful results — an assumption validated by the working dflash-smoke service that had been observed running successfully with max_running_requests=4. Third, that the benchmark runner script would correctly regenerate the deleted files when re-executed, which required the script to be idempotent (or at least tolerant of missing result files).

The input knowledge required to understand this message is substantial. One must grasp the architecture of speculative decoding with DFlash and DDTree, where a small draft model proposes multiple candidate tokens and a large target model verifies them in parallel. The Mamba cache stores the state of the draft model's selective state-space layers, and its size directly constrains how many speculative tokens can be generated per step. The KV cache stores the target model's key-value pairs for attention computation. The interaction between max_running_requests, max_mamba_cache_size, and memory partitioning (mem_fraction_static, mamba_full_memory_ratio) is a subtle SGLang-specific detail that the assistant had to reverse-engineer from log output. Without this understanding, the deletion would look like random housekeeping rather than a precise experimental correction.

Output Knowledge Created

The immediate output of this message is a clean benchmark results directory containing only tp1-auto.json — a result from a configuration that was not affected by the max_running_requests issue. But the more important output is the knowledge that the assistant has identified and corrected a systematic measurement error. The seven deleted files represent not just data but a failed experimental configuration that has been diagnosed and will not be repeated.

This message also creates a clear audit trail. Any future reader of the conversation can see that the TP1 linear and DDTree benchmark results were deleted because they were known to be invalid, and that the re-runs (which appear later in the conversation) were generated with the corrected max_running_requests=4 configuration. This transparency is essential for reproducible research.

The Broader Context: Benchmarking at Scale

This deletion occurred within a larger benchmarking campaign that spanned multiple model architectures (Qwen3.6-27B, Kimi K2.6), multiple speculative decoding techniques (DFlash linear, DDTree, EAGLE-3), and multiple tensor parallelism configurations (TP1, TP4, TP8). The assistant had already overcome significant infrastructure challenges — an LXC container with missing nvidia-uvm cgroup permissions after a host reboot, CUDA initialization failures, and PCIe/NUMA tuning issues. The discovery that the benchmark results themselves were contaminated by a configuration error must have been frustrating, but the assistant's response was measured and methodical.

The decision to delete rather than annotate or ignore the invalid results reflects a commitment to data integrity that is often easier to abandon under time pressure. Benchmarking large language models on multi-GPU systems is expensive — each run consumes significant time and computational resources. The temptation to say "these results are probably close enough" or "we can note the issue in a footnote" is real. By choosing deletion and re-run, the assistant prioritized correctness over expedience.

Conclusion

Message <msg id=11267> is a deceptively simple moment in a complex conversation. A single rm -f command, seven JSON files deleted, one file remaining. But behind that command lies a sophisticated diagnostic process: comparing memory allocation logs, calculating per-slot cache costs, identifying a counterintuitive interaction between max_running_requests and Mamba cache sizing, and making the disciplined choice to discard invalid data. It is a reminder that in experimental science — whether in a physical laboratory or a cloud GPU cluster — the most important tool is not the instrument but the judgment to know when your measurements are wrong and the integrity to start over.