Enforcing Sliding Window Attention and Building the Benchmark Harness for DFlash Speculative Decoding

The Message in Context

In the ongoing effort to optimize Kimi K2.6 with DFlash speculative decoding on 8× RTX PRO 6000 Blackwell GPUs, the assistant had just delivered a comprehensive summary of three critical bug fixes—a CUDA graph sizing mismatch, a DDTree custom-mask corruption that silently broke wide trees, and temperature sampling via tree-structured rejection sampling—when the user issued a new directive:

"can you sweep budgets/topk? Also make sure we have sliding window attention on the drafter, up to 2k context. Benchmark at small and longer contexts, add parallel requests to benchmark matrix - we're at optimising stage now. Ideally eval on some coding too"

Message 11697 is the assistant's response to this request. It is a pivotal message in the optimization phase of the project, containing both the analytical reasoning that led to a key architectural decision and the first concrete implementation step: writing a comprehensive benchmark harness. The message is deceptively simple—it consists of an Agent Reasoning section followed by a single file write—but it encapsulates the transition from debugging to systematic optimization, and it reveals the assistant's deep understanding of the speculative decoding stack's attention mechanics.

The Subject Message

The full content of the subject message (index 11697) is as follows:

[assistant] ## Agent Reasoning The draft attention mechanism uses RadixAttention with a sliding window size pulled from the model configuration, since the DFlash draft model doesn't expose its own window size method. The compact draft cache size is controlled separately through a command-line parameter, though that explanation appears to be cut off.

The explicit way to enforce a 2k sliding window is by setting the draft cache size limit to 2048 tokens, which constrains the draft to attend only to recent tokens while also reducing memory usage at longer contexts. I'm planning to run a comprehensive benchmark across different context lengths (short, medium, long), concurrency levels, and budget/topk configurations to measure how the 2k window affects acceptance rates and throughput. The key questions are whether the smaller window helps or hurts performance at long context, and which budget/topk settings work best at each combination of context and concurrency. I'll test a few specific configurations including the 2k window with different budgets and a baseline without the window restriction, using real coding prompts with varying prefix lengths.

Writing benchmark harness... The draft window (--speculative-dflash-draft-window-size 2048) gives the drafter a true 2k sliding window on its KV (and saves memory at long context). Let me build a proper optimization benchmark harness (context × concurrency, with coding prompts + correctness), then sweep configs. First the harness: [write] /home/theuser/glm-kimi-sm120-rtx6000bw/bench_ddtree_matrix.py Wrote file successfully.

Why This Message Was Written: The Reasoning and Motivation

The message was written in direct response to the user's request, but the reasoning within it reveals a much deeper motivation. The user asked for two things: (1) ensure sliding window attention on the drafter up to 2k context, and (2) sweep budgets/topk across a benchmark matrix. The assistant's reasoning shows it understood that these two requests are deeply intertwined.

The sliding window question was not trivial. The drafter model (Kimi-K2.6-DFlash-tmp-long) was trained with a 2048-token sliding window—5 of its 6 attention layers use sliding_attention with sliding_window=2048. However, the assistant had discovered in the preceding messages (msg 11691–11696) that SGLang's draft runner was showing draft_window_size=None and compact_cache=False. This meant the draft KV cache was retaining the full context rather than clamping to the 2048-token window the model was trained with. For long contexts, this is both computationally wasteful and potentially out-of-distribution—the drafter was attending to tokens beyond its training window, which could degrade the quality of draft predictions.

The assistant's reasoning traces through the architecture: the DFlash draft model uses RadixAttention (SGLang's custom attention backend), and the draft window size can come from either the model's own get_attention_sliding_window_size() method or from the model_config.sliding_window_size. Since the DFlash draft model doesn't expose its own window size method, the explicit control is --speculative-dflash-draft-window-size. The assistant correctly identifies that setting this to 2048 creates a true sliding window on the draft KV cache—not just an attention mask, but a physical cache compaction that discards tokens older than the window.

The benchmark motivation was optimization-phase thinking. The assistant's reasoning explicitly frames the key questions: "whether the smaller window helps or hurts performance at long context, and which budget/topk settings work best at each combination of context and concurrency." This is a mature optimization mindset—not just testing one configuration, but building a matrix that spans context length (short/medium/long), concurrency (single to high), and the two DDTree hyperparameters (budget and top-k). The assistant also planned to include coding correctness evaluation, directly addressing the user's "Ideally eval on some coding too."

How Decisions Were Made

The decision-making in this message is subtle but important. The assistant had to choose between two approaches to enforce the sliding window:

Modify the DFlash draft model to expose get_attention_sliding_window_size() and have SGLang use the model's built-in sliding window configuration.
Use the existing --speculative-dflash-draft-window-size flag to clamp the draft KV cache to 2048 tokens. The assistant chose option 2. This was the pragmatic decision: it required no code changes to the draft model, it leveraged an existing SGLang feature (the compact draft cache), and it provided the same behavioral effect—the draft would only attend to the last 2048 tokens. The reasoning shows the assistant understood that the compact cache mechanism (_compute_compact_draft_seq_lens) naturally implements the sliding window by truncating the draft's KV to min(seq_len, N). The decision to write a standalone benchmark harness rather than use ad-hoc commands was also deliberate. The assistant recognized that systematic optimization requires reproducible measurements across multiple dimensions. The harness would need to handle service reconfiguration, warmup, and result collection—all of which would be error-prone if done manually.

Assumptions Made

The message reveals several assumptions:

That the compact draft cache is semantically equivalent to sliding window attention. The assistant assumes that clamping the KV cache to 2048 tokens produces the same behavior as a model that was trained with sliding window attention of 2048. This is a reasonable assumption—if the drafter only sees the last 2048 tokens, it can only attend to those tokens—but it's worth noting that the attention mechanism itself (RadixAttention) might handle windowing differently than the HuggingFace reference implementation.
That the benchmark dimensions chosen are the right ones. The assistant plans to test "short, medium, long" contexts and "different concurrency levels" and "budget/topk configurations." The specific values (60, 1024, 4096, 8192 tokens for context; 1, 8, 32, 64 for concurrency; 8/4, 16/4, 4/4, 12/6 for budget/topk) would be defined in the harness file, but the reasoning doesn't enumerate them—they're assumed to be reasonable coverage of the parameter space.
That the drafter's acceptance rate is the right optimization target. The assistant frames the benchmark around throughput (tok/s) and acceptance length, implicitly assuming that these are the metrics the user cares about. This aligns with the user's focus on "optimising" but assumes no other constraints (e.g., memory usage, latency tail, power consumption).

Mistakes or Incorrect Assumptions

The reasoning contains one notable self-aware observation: "though that explanation appears to be cut off." This refers to the assistant's understanding of how the compact draft cache size is controlled. The assistant had been tracing through SGLang's source code in the preceding messages and encountered an incomplete picture. The dflash_worker.py file showed the compact cache logic, but the full chain from command-line parameter to cache behavior wasn't entirely clear. The assistant's reasoning acknowledges this gap but proceeds with the practical solution anyway—a pragmatic tradeoff.

A more subtle potential mistake is the assumption that the sliding window must be enforced via the compact cache. In reality, the DFlash draft model's attention layers might already be applying the sliding window mask through RadixAttention's internal logic, even without the compact cache being enabled. The draft_window_size=None in the logs might have been misleading—it could refer to the cache size rather than the attention window. If RadixAttention was already masking out tokens beyond 2048 during attention computation, then enabling the compact cache would be redundant (though still beneficial for memory). The assistant doesn't fully resolve this ambiguity in the reasoning.

Input Knowledge Required

To understand this message, the reader needs:

Knowledge of speculative decoding with DFlash/DDTree. The message assumes familiarity with the draft-verify loop, the concept of a "drafter" model that predicts multiple tokens per step, and the budget/top-k hyperparameters that control tree construction.
Understanding of SGLang's architecture. The message references RadixAttention, the DFlashDraftModel, the compact draft cache, and the --speculative-dflash-draft-window-size flag. Without knowing that SGLang uses a custom attention backend and manages KV caches separately from the model definition, the reasoning would be opaque.
Knowledge of the project's hardware context. The 8× RTX PRO 6000 Blackwell GPUs, PCIe interconnect, and the CT200 remote machine are all part of the ongoing context. The benchmark harness is written for this specific setup.
The history of the three bug fixes. The message builds directly on the CUDA graph sizing fix, the DDTree mask corruption fix, and the temperature sampling implementation from the preceding messages (msg 11685–11689).

Output Knowledge Created

This message creates:

The benchmark harness file (bench_ddtree_matrix.py), which is the primary output. This file would contain the logic for cycling through context lengths, concurrency levels, and budget/topk configurations, running the service, collecting throughput and acceptance metrics, and optionally evaluating coding correctness.
The architectural decision to use --speculative-dflash-draft-window-size 2048 rather than modifying the draft model. This decision shapes all subsequent benchmarking and optimization work.
The benchmark plan implicit in the reasoning: test across context lengths (short, medium, long), concurrency levels (single to high), and budget/topk configurations, with and without the 2k window, including coding evaluation.

The Thinking Process

The Agent Reasoning section reveals a structured, methodical thought process. The assistant begins by summarizing what it learned from the preceding investigation: the draft attention uses RadixAttention, the window size comes from model configuration, and the compact cache is controlled separately. It acknowledges a gap in understanding ("that explanation appears to be cut off") but doesn't let it block progress—it identifies the practical solution anyway.

The reasoning then shifts to planning mode. It frames the key experimental questions, enumerates the parameter dimensions, and commits to building a proper harness rather than ad-hoc testing. The phrase "Writing benchmark harness..." marks the transition from analysis to action.

The final paragraph is a concise summary of the decision: "The draft window (--speculative-dflash-draft-window-size 2048) gives the drafter a true 2k sliding window on its KV (and saves memory at long context)." This shows the assistant has synthesized the investigation into a clear, actionable conclusion.

What's notable is what's not in the reasoning: there's no debate about whether to enable the window, no analysis of tradeoffs between different window sizes, no consideration of whether the window should match the training configuration exactly. The assistant treats the 2048 window as a given—the drafter was trained with it, so inference should match. This is a reasonable engineering judgment, but it leaves unexplored the possibility that a larger window (or no window) might actually perform better at certain context lengths due to the additional information available to the drafter.

Conclusion

Message 11697 is a turning point in the optimization phase. It marks the moment when the assistant stopped debugging and started systematically optimizing, armed with the knowledge that the DDTree mask corruption was fixed and the CUDA graphs were working. The decision to enforce the 2k sliding window via the compact draft cache was pragmatic and correct, and the benchmark harness would provide the data needed to make informed tradeoffs between budget, top-k, context length, and concurrency.

The message also illustrates a key characteristic of effective AI-assisted engineering: the ability to hold a complex mental model of the system (RadixAttention, compact cache, draft model architecture, CUDA graphs) while simultaneously planning a multi-dimensional optimization campaign. The assistant's reasoning shows it tracing through the attention mechanics, identifying the control lever, and immediately integrating it into a broader benchmark plan—all within a single message.