Tokenization at Scale: How 902,087 Completions Became 1.87 Billion Training Tokens in 6.5 Minutes

The Message

In a single, data-dense assistant message (global index 7726), the AI reported the successful completion of a massive tokenization pipeline:

Done in 6.5 minutes total. Results:

>

| Metric | Value | |---|---| | Samples | 902,087 | | Total tokens | 1.866B (1,865.8M) | | Loss tokens | 1.633B (87.5% of total) | | Mean seq len | 2,068 | | Median seq len | 1,727 | | P90 seq len | 4,200 | | Max seq len | 8,191 | | Skipped | 0 |

>

Huge improvement over the old prompt-only data (324M tokens, mean 355). Now 5.75x more tokens and the loss tokens are 87.5% (vs 3.5% before).

>

Dataset saved locally at /data/dflash/tokenized_completions/ (45 shards) and uploaded to S3 at tokenized-completions/.

Beneath the terse statistics lies a pivotal moment in a complex machine learning pipeline. This message marks the successful conclusion of Phase 1 of a three-phase training process for a DFlash speculative decoding drafter — a lightweight model designed to accelerate inference of the larger Qwen3.6-27B language model. To understand why this 6.5-minute tokenization run deserves close analysis, we must trace the chain of reasoning, discovery, and architectural pivots that led to it.

The Crisis That Preceded the Pivot

The story begins with a devastating discovery. The team had spent considerable effort building a 914K-sample dataset for training a DFlash drafter — a speculative decoding model that predicts multiple future tokens in parallel to speed up inference. Hidden state extraction had been set up, S3 uploads configured, and the pipeline appeared ready. But when the team inspected the tokenized dataset, they found that 87% of samples had a loss_mask sum of exactly 6 tokens — just the boilerplate thinking\n\n response\nOK.<|im_end|>. The model responses were essentially empty.

This was a catastrophic failure of the data generation pipeline. The prompts had been fed to Qwen3.6-27B without enabling "thinking mode" — the chain-of-thought reasoning capability that produces the rich, multi-step reasoning traces the DFlash drafter was meant to learn. Without thinking traces, the completions were trivial, and training on them would produce a drafter that could only predict "OK." The entire hidden state extraction pipeline, the 645 GB of prompt-only hidden states stored in S3, and the weeks of effort were rendered useless.

The team pivoted decisively. They provisioned a 7× B200 NVL node (183 GB each, NVLink mesh), deployed SGLang 0.5.11 with MTP (Multi-Token Prediction) speculative decoding, and regenerated all completions with thinking mode enabled. The B200 generation run produced 902,087 completions with full Qwen3.6-27B thinking traces: 1.64 billion output tokens, 7.25 GB in S3. This was the raw material for the new dataset.

The Storage Impossibility and the Online Architecture

With 902K high-quality completions in hand, the team faced a second crisis. The original plan called for offline hidden state extraction — running the target model (Qwen3.6-27B) over every completion, extracting intermediate hidden states from 5 decoder layers, and storing them for training. The math was brutal: 5 layers × 5120 hidden dimension × BF16 precision × ~2000 average tokens × 902,087 samples ≈ 90 TB of storage. This was not merely impractical; it was impossible with the available infrastructure.

This realization forced a fundamental architectural redesign. Instead of extracting and storing hidden states offline, the team pivoted to an online training architecture where hidden states are extracted on-the-fly during the target model forward pass and fed directly to the drafter, eliminating storage entirely. The design used a 2× data-parallel setup: GPUs 0 and 1 each run a frozen copy of Qwen3.6-27B with hook-based extraction, transferring hidden states over PCIe Gen5 to GPUs 2 and 3 which hold the drafter and optimizer, with manual gradient synchronization between the two streams.

This online architecture made the tokenization step even more critical. The tokenized data — input IDs, loss masks, and attention masks — would be loaded in batches and fed through the target model's forward pass in real time. Every aspect of the tokenization had to be correct: the chat template application, the loss mask generation, and the handling of multi-turn conversations and tool calls.

The Tokenization Pipeline: Design Decisions

The tokenization script (tokenize_completions.py) embodied several key design decisions. First, it applied the Qwen3.6 chat template with special handling for thinking tokens. The completions from the B200 run included reasoning_content fields alongside regular content. The script merged these into a single assistant message with the thinking trace embedded, then applied the chat template to produce the full conversation text.

Second, it generated a loss_mask — a binary vector where 1 indicates tokens that should contribute to the training loss. For DFlash training, only assistant tokens (including the thinking trace) should be predicted; user messages and system prompts are conditioning context. The script set loss_mask to 1 for all assistant tokens and 0 for everything else.

Third, the script handled edge cases: multi-turn conversations where only the last assistant turn was generated (earlier turns were stripped), tool-calling prompts where the model produced JSON function calls, and the occasional degenerate <tool_call> loop where the model expected tool execution feedback that never came.

The tokenization was designed for maximum parallelism. The user explicitly requested high parallelism — first suggesting 32 download workers, then pushing tokenization to 128 workers. The assistant optimized accordingly: downloads used a thread pool with 32 concurrent connections (each thread creating its own boto3 client for thread safety), while tokenization used multiprocessing with 128 workers to saturate the CPU cores. The initial implementation had a serial download loop and imported torch unnecessarily; both were fixed in rapid iterations (messages 7708-7722).

The 6.5-Minute Run: What the Numbers Mean

The final run processed 902,087 completions in 6.5 minutes — approximately 2,313 samples per second. This throughput was achieved by combining 32-way parallel S3 downloads with 128-way parallel tokenization on a CPU-only machine (the tokenizer doesn't require GPU since it only uses the Hugging Face tokenizer, not the model itself).

The statistics reveal the nature of the generated data:

Output Knowledge and Its Significance

This message created critical output knowledge for the project:

  1. Dataset readiness confirmation: The tokenized dataset exists, is validated (zero errors), and is accessible both locally (45 Arrow shards at /data/dflash/tokenized_completions/) and remotely (S3 at tokenized-completions/). This unblocks Phase 2+3 of the training pipeline.
  2. Training signal quality metrics: The 87.5% loss token ratio confirms that the dataset will provide dense training signal. The DFlash drafter's block-diffusion loss and exponential position decay (gamma=4.0) will operate on a rich distribution of token positions.
  3. Sequence length distribution: The mean, median, P90, and max values inform the dynamic batching strategy in the training script (train_dflash_online.py), which uses a TOKEN_BUDGET of 8,192 tokens per batch. The P90 of 4,200 means most samples fit comfortably within a single batch, while the max of 8,191 requires careful handling.
  4. Storage requirements: 45 Arrow shards for 1.866B tokens is manageable (roughly 3-4 GB total, depending on compression). This is a tiny fraction of the 90 TB that offline hidden state extraction would have required, validating the online architecture decision.

The Broader Context: A Pipeline in Motion

This message sits at a specific point in a larger narrative. The project began with environment setup on Ubuntu 24.04 with Blackwell GPUs, moved through multiple model deployments (GLM-5-NVFP4, Qwen3.5-397B, Qwen3.5-122B, Qwen3.6-27B), and eventually focused on building a better DFlash drafter through hidden state extraction and training. The discovery of the empty-response dataset was a major setback that forced a complete regeneration of training data. The B200 generation run produced the raw completions. The tokenization run (this message) converted those completions into a training-ready dataset. The next phase — online training on 4× RTX PRO 6000 Blackwell GPUs — would use this dataset to train the DFlash drafter.

The 6.5-minute tokenization run, while brief, was the culmination of days of work: designing the online architecture, implementing three scripts, fixing import errors, optimizing parallelism, and validating the pipeline. It represents the moment when the data pipeline shifted from "building" to "ready." The message's tone — factual, numeric, comparative — reflects the assistant's role as a precise reporter of engineering outcomes. But the exclamation "Huge improvement" and the detailed comparison to the old dataset reveal the relief and satisfaction of having overcome a significant obstacle.

For a reader unfamiliar with the conversation, this message demonstrates several principles of large-scale ML engineering: the importance of data quality validation (catching the empty-response problem early), the willingness to discard months of work when data is fundamentally flawed, the necessity of architectural pivots when faced with impractical storage requirements, and the power of parallel processing to reduce what could be hours of tokenization to minutes.