The Moment the Data Spoke: How a 77% Coding Skew Reshaped a Speculative Decoding Project

Introduction

In the life of a machine learning project, there are moments of quiet revelation that change everything. They rarely arrive with fanfare. More often, they come as a single line of output from a hastily written script — a number that stops you cold and forces you to reconsider the assumptions underpinning weeks of work. For the DFlash speculative decoding drafter project documented in this opencode session, that moment arrived at message index 9411, when an assistant executed a simple Python script to sample random training completions and discovered that 77.2% of the training data consisted of code generation tasks.

This article examines that single message in detail: why it was written, how its decisions were made, what assumptions it carried, and how its output reshaped the trajectory of an entire project. The message itself is unassuming — a bash invocation running an inline Python script — but its consequences were profound, triggering a strategic pivot from architecture and optimization tuning to data-centric improvements that would define the project's next phase.

Context: The Performance Gap That Wouldn't Close

To understand why message 9411 was written, we must first understand the pressure that built up behind it. The project was training a DFlash block-diffusion speculative decoding drafter for the Qwen3.6-27B language model. The goal was to match or approach the performance of a reference model from "z-lab," which achieved a DDTree-8 acceptance length (τ) of 12.38 — meaning the speculative decoder could, on average, generate about 12 tokens per verification step when using an 8-path tree.

The team had been through multiple iterations. Version 3 (v3) had plateaued at a training accuracy of 0.251 after 20,000 steps. Version 6 (v6) had fixed several critical bugs — using the correct target layer (layer 63 instead of layer 61), the proper 5-layer fully connected architecture, hard cross-entropy loss instead of soft KL divergence, and the correct gamma value of 4.0 — and was showing dramatically better convergence. Then came the DDTree experiment, which added gamma=10 weighting, sliding window attention, CAP auxiliary loss, and other optimizations.

Yet despite all this progress, a gap remained. In message 9399, the assistant had performed a detailed trajectory analysis comparing v3, v6, and the DDTree experiment against the z-lab reference. The extrapolation was sobering: even in the optimistic case, the DDTree experiment might reach only 73–89% of z-lab's DDTree-8 τ. The assistant had identified a likely culprit:

Main gap to z-lab: Training data diversity. Z-lab trained on Nemotron + CodeAlpaca (diverse), we train on coding completions only. This likely accounts for a significant portion of the remaining gap.

This was a hypothesis, not a fact. The assistant knew what datasets had been used to generate the completions — the prompt files in /data/dflash/q36-27b/raw_prompts/ contained millions of prompts from various sources — but the actual composition of the completions (the model-generated responses used for training) had never been audited. The user's response in message 9400 was direct: "Read docs/etc in /data/dflash to remind what exact train datasets were used."

The Investigation: Tracing a Dead End

The assistant began by exploring the project's documentation and data directories. It read PROGRESS.md and PLAN.md (message 9402), which described the data pipeline in broad strokes but didn't catalog the final mix. It listed the completions_raw/ directory (message 9403), finding 1,805 JSONL files. It inspected a sample completion's structure (message 9404), confirming the ShareGPT-style format with messages, input_tokens, output_tokens, and finish_reason fields — but crucially, no source or dataset label.

The assistant then examined the prompt files (messages 9405–9410). It found 3.2 million prompts across seven JSONL files, including all_prompts.jsonl, all_prompts_sharegpt.jsonl, extra_prompts.jsonl, and tool_calling_extra.jsonl. It searched for dataset source markers like "OpenOrca," "CodeAlpaca," or "ShareGPT" in the generation scripts. It inspected individual prompt records for source or dataset fields. Every path led to the same dead end: the prompts had no provenance metadata. The combined_prompts.jsonl entries showed source=unknown. The extra_prompts.jsonl contained Romanian text, negotiation tasks, JavaScript problems, and tool-calling examples — but without labels, the overall mix was impossible to determine from the prompt files alone.

This was the critical juncture. The documentation couldn't answer the question. The prompt files couldn't answer the question. The only way to understand the data composition was to look at the actual completions — the model-generated responses that were being fed into training.

The Message: A Pragmatic Empirical Audit

Message 9411 is the assistant's response to this impasse. It executes a bash command that runs an inline Python script designed to sample and classify the training completions:

import json, random
random.seed(42)
files = [f'/data/dflash/completions_raw/completions_{i:06d}.jsonl' for i in range(0, 1805, 200)]
counts = {'code': 0, 'tool': 0, 'general': 0, 'translation': 0, 'math': 0}
samples = 0
for fn in files:
    try:
        with open(fn) as f:
            for line in f:
                d = json.loads(line)
                msgs = d.get('messages', [])
                if len(msgs) < 2: continue
                user_msg = msgs[0].get('content', '')[:500].lower()
                asst_msg = msgs[1].get('content', '')[:500].lower()
                samples += 1
                if any(k in user_msg for k in ['<returncode>', 'def ', 'function', 'class ', 'import ', 'code', 'implement', 'python', 'javascript', 'typescript', 'rust', 'golang']):
                    counts['code'] += 1
                elif any(k in user_msg for k in ['tool', 'function_call', 'functions']):
                    counts['tool'] += 1
                elif any(k in user_msg for k in ['translate', 'traduire', 'übersetze']):
                    counts['translation'] += 1
                elif any(k in user_msg for k in ['solve', 'calculate', 'math', 'equation', 'integral']):
                    counts['math'] += 1
                else:
                    counts['general'] += 1
    except: pass

The design of this script reveals the assistant's thinking. It uses systematic stratified sampling: taking every 200th file from the 1,805-file range (files at indices 0, 200, 400, ..., 1600), yielding 9–10 files. This is a reasonable compromise between coverage and speed — it would sample roughly 1,000 completions from the full dataset of millions, enough for a statistically meaningful estimate. The random seed is fixed for reproducibility.

The classification is keyword-based and greedy: it checks the user message (first 500 characters, lowercased) against lists of keywords for each category, with code checked first. This prioritization means that any message containing both code-related and math-related keywords would be classified as code — a deliberate choice reflecting the assumption that coding tasks are the dominant category and should be captured preferentially.

The categories themselves reveal the assistant's mental model of the data landscape: code, tool (function calling), general (everything else), translation, and math. These are the dimensions the assistant expected to find variation in. Notably absent are categories like "reasoning," "creative writing," "QA," or "instruction following" — suggesting the assistant already had a strong prior that the data was heavily skewed toward coding and technical tasks.

The Result: A Number That Changed Everything

The script returned:

Sampled 956 completions
  code: 738 (77.2%)
  general: 147 (15.4%)
  tool: 56 (5.9%)
  translation: 9 (0.9%)
  math: 6 (0.6%)

77.2% code. The hypothesis was confirmed with stark clarity. The training data was overwhelmingly dominated by code completions — shell command outputs, code edits, function implementations, and debugging sessions. General chat, tool calling, translation, and math together accounted for less than a quarter of the data.

This number had immediate and profound implications. It explained why the DDTree experiment's trajectory, while promising, still lagged behind the z-lab reference. The reference model had been trained on a diverse mix including Nemotron (general text) and CodeAlpaca (code), giving it broad coverage across task types. The team's model, by contrast, was essentially a code specialist — it might excel at predicting the next token in a Python function, but it would struggle with the diverse prompts that real-world deployment would throw at it.

The 77% figure also explained a subtle pattern the assistant had noted earlier: the DDTree experiment's top-1 accuracy was lower than v6 at comparable steps, but its DDTree-8 streak was already ahead. A code-specialized model might develop strong local pattern-matching for code syntax (helping with the streak metric, which rewards consistent multi-token predictions) while lacking the broader language understanding needed for top-1 accuracy on diverse inputs.

Assumptions and Limitations

The audit in message 9411 was a pragmatic, quick-and-dirty investigation, and it carried several assumptions worth examining.

First, the keyword-based classification is inherently noisy. The keyword list for code includes terms like "code," "implement," and "function" — words that could appear in non-coding contexts. A user asking "Can you implement a function to calculate..." would be classified as code, but so might "Explain how the function of a compiler works" (a general explanation task). The '<returncode>' keyword is a strong signal for shell command outputs, but the others are fuzzier. The assistant implicitly assumed that false positives in the code category would be offset by false negatives (coding tasks that don't use any of the listed keywords), and that the overall proportions would be approximately correct.

Second, the classification uses only the user message, not the assistant response. This is a significant limitation. The user message might say "Write a Python function," but the assistant's response could be a general explanation rather than code. Conversely, a user message like "What's wrong with this code?" followed by a detailed debugging response would be classified as general (no code keywords in the user message) when it's actually a code-related task. The assistant chose to classify based on the user message because it's typically shorter and more indicative of the task type, but this choice introduces systematic bias.

Third, the sampling strategy assumes the files are roughly homogeneous in composition. Taking every 200th file is a form of systematic sampling that works well when files are ordered arbitrarily or by time. But if the files were ordered by some criterion correlated with content — for example, if the first 500 files were all code and the next 500 were all general — the sample could be biased. The assistant had no way to verify this without inspecting all files, and the random.seed(42) call is actually unused in the sampling (the file list is deterministic), suggesting the assistant may have intended to sample randomly within files but defaulted to a simpler approach.

Fourth, the categories are coarse and overlapping. A tool-calling task that involves writing a Python function to call an API could legitimately be classified as both "tool" and "code." The greedy ordering (code checked first) means such tasks end up in the code bucket, potentially undercounting tool usage. Similarly, a math problem phrased as "Write a Python function to calculate the integral" would be classified as code, not math.

Despite these limitations, the audit was good enough for its purpose. The 77% figure was so extreme that even with significant classification noise, the conclusion was robust: the training data was overwhelmingly skewed toward code. A more rigorous analysis — say, using an LLM to classify each sample, or manually labeling a validation set — would have been more accurate, but it would also have taken hours. The assistant needed an answer in minutes, and the keyword-based approach delivered.

The Strategic Pivot

The output of message 9411 didn't just confirm a hypothesis; it catalyzed a strategic reorientation. In the messages that followed, the assistant would go on to research and author a comprehensive data expansion plan (DATA_EXPANSION.md), identifying diverse datasets like Infinity-Instruct-0625, Nemotron-PT-v2, Hermes FC, and Atum09 to broaden the mix to 46% coding, 26% general, 11% math, and 9% agent tasks. The current training run was halted to prioritize data generation on a separate machine (CT200).

This pivot represents a classic machine learning narrative: the team had spent weeks optimizing architecture, loss functions, and hyperparameters, only to discover that the data was the binding constraint. The 77% coding skew wasn't something that better gamma scheduling or a more sophisticated loss function could fix. It was a fundamental property of the training distribution, and the only way to address it was to generate more diverse data.

The pivot also reflects a mature understanding of where performance gains actually come from in deep learning. Architecture innovations and loss function tweaks can squeeze out incremental improvements, but data diversity — ensuring the model sees a representative sample of the tasks it will encounter at inference time — is often the dominant factor. The z-lab reference model's 0.920 accuracy wasn't just a product of better architecture; it was trained on a fundamentally richer data diet.

Input Knowledge Required

To fully understand message 9411, one needs familiarity with several pieces of context:

The DFlash/DDTree project architecture: A block-diffusion speculative decoding drafter for Qwen3.6-27B, trained using hidden states extracted from the base model.
The data pipeline: Prompts were collected from various sources, fed to Qwen3.6-27B to generate completions with thinking traces, tokenized, and stored in S3. The completions are in ShareGPT format with messages arrays.
The training infrastructure: Multi-GPU training on 8× RTX PRO 6000 Blackwell GPUs, with separate roles for target model GPUs and drafter GPUs.
The trajectory analysis from message 9399: The extrapolation showing a gap to z-lab, and the hypothesis that data diversity was the cause.
The z-lab reference: A state-of-the-art DFlash drafter achieving DDTree-8 τ of 12.38, trained on Nemotron + CodeAlpaca. Without this context, the message appears to be a simple data audit. With it, it becomes a pivotal decision point in a complex engineering project.

Output Knowledge Created

Message 9411 produced several forms of knowledge:

Empirical data composition: The first quantitative measurement of the training data's category distribution, showing 77.2% code.
Validation of a hypothesis: Confirming that the data diversity gap identified in message 9399 was real and substantial.
A decision trigger: Providing the evidence needed to halt the current training run and pivot to data expansion.
A baseline for comparison: Establishing a reference point that the subsequent data expansion plan (targeting 46% coding) could be measured against.
An analytical methodology: Demonstrating a lightweight, scriptable approach to data auditing that could be applied to future datasets.

Conclusion

Message 9411 is a masterclass in pragmatic empirical investigation. Faced with a question that documentation couldn't answer and prompt files couldn't resolve, the assistant designed a simple, fast, and effective audit that cut through the ambiguity. The script wasn't perfect — its keyword-based classification was noisy, its sampling was heuristic, and its categories were coarse — but it was good enough to drive a high-stakes decision.

The 77% figure it revealed was the fulcrum on which the project turned. Before this message, the team was optimizing architecture and training dynamics. After it, they were optimizing data. This is the kind of insight that doesn't come from reading papers or tweaking hyperparameters — it comes from looking directly at your data and asking the uncomfortable question: "What am I actually training on?"

In the end, the most important tool in a machine learning engineer's arsenal isn't a better optimizer or a larger model. It's the willingness to stop, sample, and look.