From Migration to Training: The Long Arc of Speculative Decoding Optimization

Introduction

The opencode coding session documented in this chunk represents one of the most technically dense and multifaceted sequences in the entire conversation. Spanning host migration, driver installation, container configuration, model deployment, speculative decoding investigation, and the construction of a training data pipeline, this chunk tells a story of systematic engineering under real-world constraints. At its heart is a single question: how do you make a 27-billion-parameter language model serve tokens faster? The answer, it turns out, involves everything from PCIe passthrough topology to unmerged GitHub pull requests to the architecture of a custom hidden state extraction pipeline running on 8× RTX PRO 6000 Blackwell GPUs.

This article synthesizes the major arcs of this chunk: the migration of Qwen3.6-27B from a decommissioned host to kpro5, the successful deployment with MTP speculative decoding, the deep investigation into DFlash and DDTree speculative decoding methods, and the eventual pivot toward training a better drafter model—a pivot that required building a high-throughput offline hidden state extraction pipeline from scratch.

Part I: The Migration — Moving a 55GB Model Across a Cluster

The chunk opens with a decisive infrastructure pivot. The user announces that kpro6—the Proxmox host that had been the primary compute node for the project—has been decommissioned ([msg 6747]). The LXC container CT129, which held the production inference environment, has been migrated to kpro5, a host with a heterogeneous GPU mix: RTX 3090s already allocated to running virtual machines, and two free RTX A6000s (48GB each) available for the inference workload.

The assistant's first task is to make kpro5 ready for GPU-accelerated ML inference. This proves to be a multi-layered challenge. The host has no NVIDIA driver installed—it is running the open-source nouveau driver, which is incompatible with CUDA-based ML frameworks. The assistant must install NVIDIA driver 580.126.09, but this requires kernel headers for the Proxmox kernel, which are not present ([msg 6751]). A chain of dependencies unfolds: install kernel headers, install build tools, blacklist nouveau, download and install the NVIDIA driver, and verify with nvidia-smi ([msg 6762]).

But the driver installation is only half the battle. The two RTX A6000s are bound to vfio-pci—a kernel driver used for PCIe passthrough to virtual machines. The assistant must unbind them, a process that involves identifying the correct PCI device addresses, checking which GPUs are currently in use by VMs (the 3090s must remain bound), and selectively unbinding only the A6000s ([msg 6766] through [msg 6771]). This requires careful reading of the Proxmox IOMMU groups and PCI topology to avoid disrupting running workloads.

Once the GPUs are visible to the host driver, the assistant turns to the LXC container configuration. CT129 was originally configured for a different host with different GPU PCIe addresses and device major numbers. The assistant updates the container config to expose the two A6000s with the correct cgroup device mappings ([msg 6774]), then starts the container and immediately hits a driver/library version mismatch: the host has driver 580.126.09, but the container's userspace libraries are from an older driver version ([msg 6779]). Resolving this requires installing matching CUDA and NVIDIA userspace libraries inside the container, a process that involves navigating the Debian/Ubuntu package repositories and avoiding the "virtual package trap" where nvidia-driver-* meta-packages pull in incompatible dependencies ([msg 6783]).

Part II: Deployment and the SGLang Version Epiphany

With the infrastructure stabilized, the assistant downloads the Qwen3.6-27B model—a 52GB BF16 model spread across 15 safetensor shards. The download itself is an ordeal: it stalls at 33GB due to a HuggingFace Hub issue, requiring the assistant to diagnose and resume the download using huggingface-cli ([msg 6806]). After verification, the model is ready.

The first deployment attempt uses SGLang 0.5.9 with MTP (Multi-Token Prediction) speculation. The server starts, CUDA graphs capture successfully, but the output is degenerate—endless repetitions of "lambda lambda lambda" ([msg 6834]). The assistant embarks on an extensive debugging journey, testing hypotheses about FlashInfer vs Triton attention backends ([msg 6855]), MTP speculation corruption ([msg 6838]), and process lifecycle issues in LXC containers ([msg 6859]). The server repeatedly crashes or produces gibberish, and the assistant chases false leads through stale logs ([msg 6845]), silent process exits ([msg 6844]), and systemd service configuration problems ([msg 6870]).

The breakthrough comes when the assistant re-reads the model card and discovers that Qwen recommends SGLang 0.5.11 specifically for the GDN (Gated DeltaNet) hybrid attention architecture ([msg 6866]). Upgrading from 0.5.9 to 0.5.11 resolves the degenerate output immediately. The GDN hybrid architecture—which combines 16 standard attention layers with 48 linear attention (Mamba-style) layers—requires specific handling in the serving framework that was only added in 0.5.11.

With the correct version, the deployment succeeds. The assistant configures the server with tensor parallelism across both A6000s, MTP speculation with NEXTN steps=3 (producing 4 draft tokens per step), and a 32K context window. A smoke test confirms coherent reasoning and correct Python code generation ([msg 6877]).

Part III: Benchmarking — Validating Production Readiness

The user's next instruction is to run benchmarks: a throughput sweep from concurrency 1 to 100, followed by long-context tests at 30K, 60K, and 100K tokens ([msg 6879]). The assistant adapts an existing benchmark script from a previous deployment ([msg 6881]) and runs the sweep.

The results are excellent. At concurrency 1, the model achieves 73.5 tok/s per request. Aggregate throughput scales nearly linearly with concurrency, reaching over 500 tok/s at C=64-100. Time-to-first-token (TTFT) remains under 0.5 seconds up to C=8, and there are zero failures across all concurrency levels ([msg 6884]). The deployment is stable under load.

The long-context benchmarks prove even more impressive. The assistant discovers that the server's 32K context limit can be extended—the GDN hybrid architecture's linear attention layers consume O(1) memory per token, so KV cache grows only with the 16 full-attention layers, not the full 64-layer depth. After restarting the server with a 128K context limit, the assistant benchmarks at 30K, 60K, and 120K input lengths ([msg 6885]). Decode speed barely degrades: 73.6 tok/s at 1K input versus 64.9 tok/s at 120K input, a mere 12% drop. This is a remarkable validation of the hybrid architecture's memory efficiency claim.

Part IV: The DFlash Investigation — When Research Meets Production

With MTP speculation working well, the assistant turns to a more ambitious speculative decoding method: DFlash. DFlash uses a separate drafter model—a smaller transformer that predicts the target model's hidden states—to generate draft tokens that the target model then verifies. The promise is higher acceptance rates than MTP's fixed-length speculation.

The assistant acquires the gated z-lab/Qwen3.6-27B-DFlash drafter safetensors from HuggingFace, creates a configuration file (initially guessing the target_layer_ids parameter), and deploys vLLM 0.20.1 with DFlash enabled. The result is catastrophic: an acceptance rate of approximately 1.1%, meaning the drafter's predictions are almost never accepted by the target model. At this rate, DFlash is worse than no speculation at all.

The assistant launches a deep investigation, reading the vLLM DFlash proposer source code, the DDTree reference implementation, and the z-lab HuggingFace repositories. Three root causes emerge:

1. Layer-ID offset bug: vLLM's hidden state extraction was missing a +1 offset in layer IDs, meaning it was extracting states from the wrong layers. This was fixed by vLLM PR #40727, but the fix was not yet merged into the release branch the assistant was using.
2. Sliding window attention (SWA) layers ignored: The Qwen3.6-27B drafter uses SWA layers, but vLLM's DFlash proposer was treating all layers as full-attention, ignoring the sliding window structure. This was fixed by PR #40898, which was still unmerged.
3. Eagle cache drop issues: The eagle cache—which stores hidden states for reuse across drafting steps—was being dropped prematurely, forcing recomputation. The assistant installs vLLM from the unmerged PR #40898 branch, confirms all three fixes are present, and verifies that the SWA layer type handling is active. However, the acceptance rate remains low, suggesting either additional bugs or that the drafter model itself—labeled "still under training" on HuggingFace—is not yet converged.

Part V: DDTree and the Architectural Wall

The assistant then investigates DDTree, a tree-based speculative decoding method that extends DFlash by generating multiple candidate draft sequences organized as a tree, then verifying all paths in parallel. The promise is higher acceptance rates through diversity—if one draft path is wrong, another might be right.

However, the assistant discovers a fundamental architectural limitation in vLLM: its verification pipeline uses a linear-chain rejection sampler, not a tree-walk sampler. Even in vLLM's EAGLE tree mode, the tree attention is only used during the drafting phase, not for verifying multiple candidate paths. Implementing true DDTree verification would require writing a new tree-walk rejection kernel from scratch—a significant engineering effort.

The assistant evaluates the DDTree authors' standalone reference implementation, successfully patching it for the Qwen3.6-27B GDN hybrid model. The benchmark confirms DDTree works correctly, but the acceptance rate improvement over DFlash is marginal (1.67 vs 1.59). The bottleneck is not the verification algorithm—it is the drafter model's quality. The drafter, still under training, simply does not predict the target model's hidden states accurately enough.

Part VI: The Pivot to Training — Building a Better Drafter

Faced with the conclusion that the drafter model is the primary bottleneck, the assistant pivots from deployment to training. The goal: train a new DFlash drafter that achieves higher acceptance rates, enabling both DFlash and DDTree to deliver meaningful speedups.

This pivot requires three major workstreams: dataset curation, training environment setup, and hidden state extraction.

Dataset curation: The assistant assembles a comprehensive 913K-sample dataset designed to align the drafter with the target model's agentic use case. The dataset mixes general instruction following (OpenOrca), code generation (Evol-CodeAlpaca, Magicoder), agentic coding traces (Agentic-Coding-Trajectories), and a significant 114K tool-calling subset (Glaive Function Calling v2, Qwen3.5 Tool Calling v2). The data is converted to ShareGPT format and tokenized using the vllm-project/speculators pipeline, which requires a custom patch for Qwen3.6's strict chat template.

Training environment: The assistant sets up the training environment on an 8× RTX PRO 6000 Blackwell node (96GB each, 1.9TB disk). The environment is provisioned with uv, speculators, and vLLM. The 55GB Qwen3.6-27B model is downloaded from HuggingFace in approximately 10 seconds (the Blackwell node has fast internet). The tokenized dataset (913,786 samples, 1.3 GB) and the DFlash drafter checkpoint are transferred. A test training run (100 samples, 1 epoch) is successfully launched, with the vLLM server serving hidden states on GPUs 0-3 and the DFlash training running on GPUs 4-7.

Hidden state extraction: The DFlash training process requires hidden states from the target model for each token in the training corpus. The speculators library provides an online vLLM pipeline for this, but it is fundamentally incompatible with Qwen3.6-27B's GDN hybrid KV cache. The assistant pivots to a custom offline extraction using HuggingFace Transformers.

The initial extraction runs at only 7–11 samples/s per GPU with high CPU sys overhead. The bottleneck is per-sample safetensors writes and individual GPU→CPU copies. The assistant's key breakthrough is batching the hidden state capture entirely on GPU—concatenating all samples in a batch before a single .cpu() transfer. This eliminates 2,725 individual copies per batch and boosts throughput to 140–155 samples/s per GPU (aggregate ~600/s across 4 GPUs)—a 20× improvement.

The pipeline is further hardened with async S3 uploads via subprocess to offload storage, installation of flash-linear-attention (FLA) to reduce kernel overhead from GDN layers, and backpressure logic that pauses extraction when /dev/shm exceeds 80% capacity. Marker-based resume ensures the pipeline can survive interruptions without losing progress. A Flask monitoring UI shows per-shard progress and GPU statistics in real time.

After the original compute instance is killed, the assistant re-provisions everything on a new 4× RTX PRO 6000 Blackwell node, installs dependencies (including building causal-conv1d for CUDA 13.0), pre-warms Triton kernels, and restarts extraction. The pipeline runs at high GPU utilization with low CPU overhead, processing the 914K-sample dataset in approximately 30–60 minutes.

Part VII: Themes and Lessons

Several overarching themes emerge from this chunk.

The gap between research and production: DFlash and DDTree are published research with open-source implementations, but deploying them in a production serving framework requires navigating unmerged PRs, custom configuration files, and subtle alignment issues between the reference implementation and the serving framework's internals. The near-zero acceptance rate was entirely a deployment integration failure, not a model quality issue.

The importance of model card information: The SGLang version debugging saga—days of chasing degenerate output that was fixed by upgrading from 0.5.9 to 0.5.11—could have been avoided if the assistant had checked the model card earlier. The model card explicitly recommended 0.5.11 for GDN hybrid attention support. This is a lesson in the value of upstream documentation.

The compounding value of systematic benchmarking: The throughput and long-context benchmarks produced a comprehensive characterization of the deployment's performance: 73.5 tok/s single-request throughput, linear scaling to 500+ tok/s aggregate, and only 12% decode speed degradation at 120K context. These numbers validated both the hardware choice (2× RTX A6000) and the software configuration (SGLang 0.5.11 with MTP speculation).

The necessity of custom infrastructure: The speculators library's online vLLM pipeline could not handle GDN hybrid models, forcing the assistant to build a custom offline extraction pipeline. The resulting 20× throughput improvement—achieved through GPU-side batching, async I/O, and careful kernel selection—demonstrates that generic tools often need adaptation for specific model architectures.

The pivot as a pattern: This chunk contains multiple pivots—from kpro6 to kpro5, from SGLang 0.5.9 to 0.5.11, from DFlash deployment to DFlash debugging, from DDTree evaluation to drafter training. Each pivot represents a recognition that the current approach is blocked and a redirection of effort toward a more promising path. The assistant's willingness to pivot—and its ability to do so without losing momentum—is a defining characteristic of effective engineering.

Conclusion

This chunk spans an extraordinary range of engineering activity: from the physical layer of GPU passthrough and driver installation, through the software layer of serving framework configuration and debugging, to the research layer of speculative decoding algorithm evaluation, and finally to the data layer of training dataset curation and hidden state extraction. Each layer revealed its own challenges, and each challenge required a different kind of expertise to overcome.

The thread that connects all of these activities is the pursuit of faster inference. MTP speculation provided a 2× speedup with minimal engineering effort. DFlash and DDTree promised further gains but required drafter models that are still under development. The assistant's response to this reality—to build the infrastructure to train better drafters—represents a long-term investment in speculative decoding performance. The hidden state extraction pipeline, running at 600 samples/s with robust monitoring and async S3 uploads, is the foundation for that investment.

In the end, this chunk is a case study in the full lifecycle of ML inference optimization: deploy, benchmark, identify bottlenecks, investigate alternatives, and when the alternatives are blocked by model quality, build the infrastructure to improve the model itself. It is a testament to the depth and breadth of engineering required to push the frontier of LLM serving performance.## References

[1] "The Architecture of State: A Comprehensive Status Document in a Multi-System ML Deployment" — Documents the state of the deployment before the pivot to kpro5.

[2] "The Pivot: Redirecting an AI Cluster from DGX Spark to KPro5" — The user's message announcing kpro6 decommissioning and the migration to kpro5.

[134] "The Benchmark Command: Validating Qwen3.6-27B Production Readiness" — The user's request for benchmarks and the assistant's response.

[135] "From Deployment to Validation: The Benchmarking Pivot in Message 6880" — Analysis of the transition from deployment to benchmarking.

[136] "The Art of Reuse: Reading a Benchmark File as a Template for Adaptation" — How the assistant adapted an existing benchmark script.

[137] "The Pivot to Evaluation: Benchmarking Qwen3.6-27B After a Hard-Won Deployment" — The decision to create separate throughput and long-context benchmark scripts.

[138] "The Invisible Scaffold: How a Single Todo-Update Message Reveals the Architecture of AI-Assisted Work" — The role of structured task tracking in the session.

[139] "Benchmarking Qwen3.6-27B: Measuring Throughput on a Fresh Deployment" — Detailed analysis of the throughput benchmark results.

[140] "The Pivot: From Throughput to Long-Context Benchmarking" — The transition to long-context evaluation and constraint awareness.