The Temperature That Breaks Everything: Why temp 0 Is Wrong Almost Always

"Also use temp 0.6, temp 0 is wrong almost always" — User, Message 12775

At first glance, this is a trivial operational note: change a sampling parameter from 0.0 to 0.6. But in the context of a multi-day engineering campaign to deploy DeepSeek-V4-Flash on 8× RTX PRO 6000 Blackwell GPUs, this single sentence represents a fundamental reorientation of the debugging strategy. The assistant had just spent an entire reasoning trace (see [msg 12773]) diagnosing a severe coherence collapse — the model was generating hundreds of repeated </div> tags, producing buggy JavaScript with mismatched element IDs, and even hallucinating entirely different user prompts (a "Butter app README" that was never requested). The assistant's working hypothesis pointed at the three custom components in the inference path: the MMA sparse-decode attention kernel, the capture-safe Triton indexer, and the FP8 KV cache. The plan was to reproduce the degeneration deterministically at temperature 0, then A/B test by disabling custom kernels. The user's correction — delivered in two terse messages — dismantles that entire approach at its foundation.

The Message in Its Full Context

The subject message arrives immediately after two critical exchanges. First, the user had said in [msg 12774]: "Note we really really don't want to quant the cache, not now" — ruling out KV cache quantization as either the cause or the fix. Then the assistant had written a Python test script (test_degen.py) configured at temperature 0 with max tokens 4000, designed to produce deterministic output that would clearly expose whether the custom kernels were corrupting attention. The user's response — "Also use temp 0.6, temp 0 is wrong almost always" — is a quiet but devastating intervention. It says: your entire experimental design is invalid, because the very parameter you chose to isolate the bug is itself a source of the bug you're trying to isolate.

Why Temperature 0 Causes Degeneration

The user's assertion that "temp 0 is wrong almost always" reflects a deep understanding of how autoregressive language models behave under greedy decoding. At temperature 0, the model always selects the single token with the highest probability at each step. This determinism creates a well-known failure mode: once the model enters a probability loop where a particular token (like </div>) has even a slight statistical edge, greedy decoding will lock onto that token and never escape. The model cannot self-correct because there is no stochastic mechanism to break out of the attractor. What looks like an attention or KV cache bug — the endless </div></div></div>... spam — is actually a sampling pathology that any model exhibits at temperature 0, regardless of kernel correctness.

The assistant's reasoning in [msg 12773] reveals the trap: "Temperature 0 will give me deterministic output so I can clearly identify if it's a real attention/KV bug versus sampling noise." This is logically sound — determinism aids reproducibility — but it fatally conflates sampling noise with sampling pathology. Temperature 0 does not eliminate noise; it replaces stochastic noise with deterministic degeneration. The resulting output is not a clean signal; it is a contaminated signal where the artifact of greedy decoding is indistinguishable from the artifact of a broken kernel. The user's correction reframes the entire debugging problem: you cannot use a pathological sampling configuration to diagnose whether the model is pathologically generating tokens.

The Assumptions That Fell

The assistant made several interconnected assumptions that the user's message quietly invalidates. First, that temperature 0 is a neutral debugging tool — a way to remove randomness and see the "true" behavior of the model. The user reveals that temperature 0 is not neutral; it is a distortion that introduces its own failure modes. Second, that degeneration collapse and context confabulation are necessarily symptoms of kernel bugs or KV cache corruption. The user's correction opens the possibility that at least some of the observed degeneration was simply the model operating at an inappropriate temperature. Third, that the debugging process should prioritize determinism over realism — that a reproducible but artificial test condition is more informative than a stochastic but realistic one.

There is also a subtler assumption embedded in the assistant's approach: that the custom MMA kernel and Triton indexer could be validated against a "stock" baseline at temperature 0, and any difference in degeneration patterns would isolate the kernel bug. The user's message implies that both configurations — custom and stock — would degenerate at temperature 0, just in different ways, making the comparison uninformative. The signal-to-noise ratio of the experiment was zero because the noise (greedy degeneration) was guaranteed to drown out any signal (kernel-specific corruption).

Knowledge Required to Understand This Message

To fully grasp the weight of this correction, one needs several layers of context. First, the technical understanding of LLM sampling: that temperature controls the softmax sharpness, with temperature 0 collapsing to argmax selection, and that argmax selection in autoregressive models is provably prone to repetitive loops (the "repetition trap" documented in the literature on exposure bias and teacher forcing). Second, the specific history of this deployment: the custom MMA sparse-decode kernel (described in the segment summary as achieving 2.2–2.9× throughput improvement), the capture-safe Triton indexer with early-exit per page, and the FP8 KV cache that the user had just ruled out as a variable. Third, the debugging context: the assistant had just written a test script and was about to run it when the user intervened. Fourth, the operational reality of the deployment: the model was serving live requests through a PD-disaggregated router with systemd services, and the user had firsthand visibility into the failure modes from production transcripts.

Knowledge Created by This Message

This message creates several pieces of actionable knowledge. Most immediately, it establishes that any reproduction of the degeneration must use temperature 0.6 — the production temperature — rather than temperature 0. This changes the test harness from a deterministic comparator to a statistical one: instead of checking whether output A exactly matches output B, the assistant must now check whether the distribution of outputs at temperature 0.6 is healthy (diverse, coherent, terminating properly) versus degenerate (repetitive, confabulating). This is a harder experimental design problem, but it is the only design that tests the actual failure mode.

Second, the message creates the knowledge that temperature 0 is itself a suspected cause of the observed symptoms. This reframes the debugging priority: before investigating the MMA kernel, the Triton indexer, or the FP8 cache, the first step should be to verify that the model produces coherent output at temperature 0.6 with stock kernels. If it does, then the degeneration at temperature 0 was an artifact of the sampling parameter, not the custom kernels. If it does not, then the kernel investigation proceeds — but with a correct baseline.

Third, the message implicitly establishes a methodological principle for this project: debugging must happen at production settings. The temptation to simplify — to use temperature 0, to reduce context length, to disable stochasticity — is strong, but each simplification introduces its own artifacts. The user's insistence on temperature 0.6 is a stand-in for a broader insistence on ecological validity in the testing process.

The Thinking Process Visible in the Exchange

The assistant's reasoning in [msg 12773] shows a methodical, kernel-focused debugging mindset: isolate variables, make the test deterministic, A/B compare. This is classic systems engineering, and it would be correct in a world where the sampling strategy was independent of the model's coherence. But the user's correction reveals that sampling and coherence are not independent — temperature 0 actively causes the very symptom being investigated. The assistant's blind spot was treating the sampling parameter as an external, neutral control variable rather than as an integral part of the model's behavior.

The user's two messages — first ruling out KV cache quantization, then correcting the temperature — form a pincer movement. The first eliminates one of the three custom components from suspicion. The second eliminates the debugging methodology itself. After these two messages, the assistant cannot proceed with the planned A/B test. The entire approach must be redesigned.

Broader Significance

This message, for all its brevity, encapsulates a lesson that extends far beyond this specific deployment. In the engineering of large language model systems, there is a persistent temptation to debug at "simplified" settings — greedy decoding, short contexts, single requests — because these settings are more tractable and reproducible. But the pathologies that emerge at production settings (temperature > 0, long contexts, concurrent requests) are often qualitatively different from those at simplified settings. A bug that only manifests at temperature 0.6 with 128K context and concurrent requests cannot be reproduced at temperature 0 with 8K context and a single request. Conversely, a bug that does manifest at temperature 0 may be an artifact of the temperature itself, not a real system bug.

The user's correction — "temp 0 is wrong almost always" — is a reminder that in complex systems, the debug configuration must match the production configuration in all relevant dimensions. The assistant learned this the hard way: a carefully crafted test script, a well-reasoned debugging plan, and a promising hypothesis about kernel bugs, all invalidated by a single sentence about a sampling parameter. The temperature that was supposed to reveal the bug was itself the bug.