Debugging the d2t Tensor: Unraveling a Vocabulary Mapping Mystery in EAGLE-3 Speculative Decoding

Introduction

In the complex world of speculative decoding for large language models, few debugging sessions are as methodical and revealing as the one captured in message 3567 of this opencode session. The assistant, deep in the trenches of deploying a custom-trained EAGLE-3 draft model for the Kimi-K2.5 architecture, had just confronted a baffling failure: the newly trained 1.2B-parameter draft model achieved an acceptance rate of approximately 0.20—effectively zero draft tokens being accepted by the target model. After ruling out weight loading issues, hidden state dimension mismatches, and auxiliary capture configuration problems, the investigation had narrowed to a single, suspicious data structure: the d2t tensor, which maps draft vocabulary indices to target vocabulary indices.

Message 3567 represents a pivotal moment in this debugging odyssey. It is the message where the assistant, having noticed something odd about the d2t tensor in the previous message, pivots to a deeper investigation of the vocabulary mapping between the draft model and the target model. This seemingly narrow technical inquiry would ultimately illuminate a critical aspect of how EAGLE-3 speculative decoding works—and why the trained draft model was failing to produce useful predictions.

The Context Leading to This Message

To understand why message 3567 was written, we must trace the debugging journey that preceded it. The assistant and user had been working for days on deploying an EAGLE-3 speculative decoding system for the Kimi-K2.5 model, a massive multi-modal language model running on a server with 8 GPUs. The pipeline involved training a lightweight draft model (1.2B parameters) to predict the target model's next tokens, using hidden states extracted from intermediate layers of the target model as conditioning features.

The training had been completed on 10,000 samples of synthetic data extracted via SGLang, producing a checkpoint stored in safetensors format. However, when the assistant deployed this checkpoint on the SGLang server and ran benchmarks, the acceptance rate hovered around 0.20—meaning the draft model's predictions were accepted only about 20% of the time, which is essentially the random baseline (since the draft model samples from its own distribution, and random acceptance would be around 20% for a 5-token draft window).

A systematic debugging effort unfolded across messages 3553–3566:

  1. Weight key name mismatch (msg 3553–3554): The speculators library saves the decoder layer as layers.0.* but SGLang's LlamaForCausalLMEagle3 expects midlayer.*. The assistant fixed this by renaming keys, but the acceptance rate barely budged (0.20 → 0.21).
  2. Hidden state dimension analysis (msg 3555–3556): The assistant verified that the fc.weight fusion layer (projecting 21504 → 7168) had reasonable non-zero weights, confirming the model was actually trained.
  3. Auxiliary hidden state capture (msg 3557–3564): The assistant traced how SGLang captures hidden states from three intermediate layers of the target model (layers 3, 31, 59 after SGLang's +1 offset) and verified that the kimi_k25.py delegation correctly passes through to the language model's set_eagle3_layers_to_capture method.
  4. The d2t tensor anomaly (msg 3565–3566): While checking the checkpoint's d2t tensor, the assistant noticed that the first 10 entries were all zero: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0]). This was suspicious because it suggested that draft tokens 0 through 9 all mapped to target token 0, which would mean the draft model couldn't distinguish between these tokens. This is the precipitating observation that leads directly to message 3567.

The Message Itself: A Deeper Probe into the Vocabulary Mapping

Message 3567 opens with the assistant's observation: "That's odd — lots of zeros at the start. Let me check more." This simple statement captures the essence of the debugging mindset: a hypothesis has been formed (the d2t tensor might be corrupted or misunderstood), and now it must be tested with more data.

The assistant executes a Python script on the remote server via SSH that loads the original d2t.pt file from the vocabulary mapping directory (/data/eagle3/synth_10k_sglang/vocab_mapping/d2t.pt), rather than the checkpoint's safetensors file. This is a crucial distinction: the d2t.pt file is the source vocabulary mapping created during the data preparation phase, while the safetensors file contains the trained version of the same mapping. By checking the source, the assistant can determine whether the problem originated in data preparation or in training.

The script performs several diagnostic queries:

  1. Non-zero count: (d2t != 0).sum() — finds that 31,876 out of 32,000 entries are non-zero, meaning only 124 entries are zero.
  2. Unique values: d2t.unique().shape[0] — reveals only 20,954 unique values across 32,000 entries, indicating significant redundancy in the mapping (many draft tokens map to the same target token).
  3. Slice inspection: Examining ranges d2t[100:120], d2t[1000:1010], and d2t[31990:] to understand the distribution of zero and non-zero entries.
  4. t2d tensor analysis: Loading the companion t2d.pt tensor, which is a boolean mask of shape [163840] (the target model's vocabulary size), where True indicates that a target token is included in the draft vocabulary. The assistant finds that exactly 32,000 entries are True, confirming the draft vocabulary size.
  5. Draft token ID extraction: Using t2d.nonzero().squeeze() to get the actual target token IDs that are included in the draft vocabulary, then printing the first and last 10. The results are illuminating:
d2t non-zero entries: 31876 / 32000
d2t unique values: 20954
d2t[100:120]: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
d2t[1000:1010]: tensor([787, 787, 787, 788, 792, 792, 792, 792, 793, 794])
d2t[31990:]: tensor([131446, 131464, 131465, 131529, 131530, 131530, 131586, 131589, 131608, 131608])
t2d shape: torch.Size([163840]), dtype: torch.bool
t2d sum (num draft tokens): 32000
First 10 target tokens in draft vocab: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Last 10 ...

The message cuts off at "Last 10 ..." because the output was truncated, but the key findings are already visible.

The Role of the d2t Tensor in EAGLE-3

To appreciate the significance of this investigation, we must understand what the d2t tensor does in the EAGLE-3 architecture. EAGLE-3 uses a technique called "vocabulary alignment" where the draft model has its own vocabulary (typically a subset of the target model's vocabulary) and must map its predictions back to the target model's vocabulary space.

The d2t (draft-to-target) tensor is a mapping of shape [draft_vocab_size] where d2t[i] gives the target vocabulary ID corresponding to draft token i. During inference, when the draft model predicts a token, the system uses d2t to convert it to the target model's token ID for verification. The companion t2d (target-to-draft) tensor is a boolean mask indicating which target tokens are represented in the draft vocabulary.

In SGLang's EAGLE-3 implementation, the d2t tensor is used to compute hot_token_id:

if "d2t" in name:
    self.hot_token_id = loaded_weight + torch.arange(loaded_weight.shape[0])

This formula treats d2t as differences (offsets) rather than absolute indices, adding the draft token's position to produce the target token ID. However, the assistant had previously noted (in msg 3565) that the checkpoint's d2t tensor contained absolute indices (values up to 131,608), not differences. This discrepancy between how the tensor was saved and how it was interpreted could be a critical bug.

What the Investigation Revealed

The investigation in message 3567 reveals several important facts about the vocabulary mapping:

1. The zero entries are limited to the first ~124 tokens. With 31,876 non-zero entries out of 32,000, only the first 124 draft tokens map to target token 0. This is likely because tokens 0–123 in the target vocabulary are special tokens (like padding, unknown, or control tokens) that are mapped to a single draft token. This is actually normal behavior—the draft model doesn't need to predict special tokens with fine granularity.

2. The mapping has significant redundancy. With 32,000 draft tokens mapping to only 20,954 unique target tokens, many target tokens are represented by multiple draft tokens. This is a design choice in EAGLE-3: the draft vocabulary is typically a superset of the most frequent target tokens, with multiple draft tokens per target token to allow the draft model to express different probabilities for the same target token through different draft token paths.

3. The first 10 target tokens in the draft vocabulary are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. This confirms that the draft vocabulary includes the earliest tokens in the target vocabulary, which are typically special tokens and control characters.

4. The t2d mask has exactly 32,000 True entries. This confirms that the draft vocabulary size is 32,000, matching the expected size.

Assumptions Made in This Message

The assistant operates under several assumptions in message 3567:

Correct assumption: The source d2t.pt file is the authoritative version. By loading the original vocabulary mapping file rather than the checkpoint's safetensors, the assistant correctly assumes that any corruption would be visible by comparing the two. The source file represents the intended mapping, while the checkpoint's version is what was actually trained.

Correct assumption: The zero entries at the start of d2t are likely special tokens. The assistant's comment "The first N might be special tokens" is a reasonable hypothesis, and the data supports it—only the first ~124 entries are zero, which is consistent with the number of special tokens in many tokenizer vocabularies.

Implicit assumption: The d2t mapping is the same format in both the source and the checkpoint. The assistant doesn't explicitly verify that the source d2t.pt and the checkpoint's d2t tensor contain identical values. This is a minor oversight, but the investigation is still in progress—the assistant is gathering data, not drawing conclusions.

Implicit assumption: The vocabulary mapping is a plausible source of the zero acceptance rate. This is a reasonable debugging heuristic. If the draft model's vocabulary mapping is incorrect, then even a perfectly trained draft model would produce garbage predictions when mapped to the target vocabulary. The fact that the acceptance rate is exactly at the random baseline (0.20 for a 5-token window) is consistent with the draft model's outputs being essentially random after vocabulary mapping.

Input Knowledge Required

To fully understand message 3567, one needs knowledge of:

  1. EAGLE-3 architecture: Understanding that EAGLE-3 uses a draft model with a separate vocabulary that must be mapped to the target model's vocabulary via the d2t tensor.
  2. SGLang's EAGLE-3 implementation: Knowing that SGLang computes hot_token_id from d2t using loaded_weight + torch.arange(loaded_weight.shape[0]), which treats d2t as differences rather than absolute indices.
  3. Vocabulary mapping in speculative decoding: Understanding that draft models typically use a subset of the target vocabulary, with multiple draft tokens potentially mapping to the same target token to allow for probabilistic sampling.
  4. The training pipeline: Knowing that the d2t.pt and t2d.pt files were created during the data preparation phase (vocabulary alignment) and that the d2t tensor in the checkpoint is the trained version of this mapping.
  5. PyTorch tensor operations: Understanding nonzero(), squeeze(), unique(), and boolean indexing to interpret the diagnostic queries.
  6. The broader debugging context: Knowing that the acceptance rate is ~0.20, that weight loading has been verified, and that hidden state dimensions appear correct—all of which makes the vocabulary mapping a plausible remaining suspect.

Output Knowledge Created

Message 3567 produces several valuable pieces of knowledge:

  1. The d2t mapping is not trivially broken. With 31,876 non-zero entries, the mapping is mostly populated. The zeros at the start are likely special tokens, which is normal.
  2. The mapping has significant redundancy (32,000 → 20,954 unique values). This is expected for EAGLE-3's vocabulary alignment strategy, but it's useful to confirm.
  3. The t2d mask is correctly shaped and populated. With exactly 32,000 True entries out of 163,840, the draft vocabulary size matches expectations.
  4. The first 10 target tokens in the draft vocabulary are [0–9]. This confirms that the draft vocabulary includes the earliest tokens, which are typically special tokens.
  5. The d2t tensor values span the full target vocabulary range. Values go up to 131,608, which is within the target vocabulary size of 163,840. However, the message does not resolve the core question of whether the d2t mapping is the cause of the zero acceptance rate. The data is consistent with both a correct mapping and an incorrect one—the zeros at the start are suspicious but explainable, and the overall structure looks reasonable. The investigation would need to continue, perhaps by comparing the source d2t.pt with the checkpoint's d2t tensor, or by examining how SGLang interprets the tensor during inference.

The Thinking Process Visible in the Message

The assistant's reasoning in message 3567 is a textbook example of systematic debugging:

  1. Observation: "That's odd — lots of zeros at the start." The assistant notices an anomaly in the data and flags it for investigation.
  2. Hypothesis formation: The zeros might indicate a problem with the vocabulary mapping, but they might also be special tokens. The assistant doesn't jump to conclusions—instead, it gathers more data.
  3. Systematic data collection: The script probes multiple aspects of the tensor: non-zero count, unique values, specific slices, and the companion t2d tensor. Each query addresses a different potential concern.
  4. Comparative analysis: By checking both d2t and t2d, the assistant builds a complete picture of the vocabulary mapping. The d2t tells us about the mapping from draft to target, while t2d tells us about which target tokens are included.
  5. Pattern recognition: The assistant notices that the zeros are concentrated at the beginning of the tensor (indices 0–124), which is consistent with special tokens. This pattern recognition helps narrow down whether the zeros are a bug or a feature.
  6. Incremental disclosure: The message doesn't try to solve the entire problem at once. Instead, it adds one more piece of evidence to the growing body of knowledge about why the draft model is failing.

The Broader Significance

Message 3567, while seemingly a narrow technical investigation of a single tensor, exemplifies the kind of meticulous debugging required when deploying speculative decoding systems in production. The EAGLE-3 architecture involves multiple interacting components—the target model, the draft model, the vocabulary mapping, the hidden state capture mechanism, and the verification logic—and any one of them can cause the entire system to fail silently.

The fact that the acceptance rate is exactly at the random baseline (0.20) is a strong signal that the draft model's predictions are not being used at all. This could be because:

Conclusion

Message 3567 is a snapshot of active debugging in progress—a moment when a skilled engineer notices an anomaly, forms a hypothesis, and systematically gathers data to test it. The investigation of the d2t tensor reveals that the vocabulary mapping is structurally sound (mostly non-zero, correctly sized, with expected redundancy), but leaves open the question of whether it is being interpreted correctly by SGLang's inference code. This message doesn't solve the mystery, but it advances the investigation by ruling out one class of problems and providing crucial evidence for the next phase of debugging.

In the broader narrative of the EAGLE-3 deployment, message 3567 is the moment when the assistant shifts focus from the high-level architecture (weight loading, hidden state capture) to the low-level data formats (vocabulary mapping, tensor interpretation). This shift in perspective would ultimately prove essential for diagnosing the root cause of the zero acceptance rate—a root cause that lay not in any single component, but in the mismatched assumptions between the training and inference pipelines.