The Temperature Default: A Surgical Fix for DeepSeek-V4's Coherence Collapse

Introduction

In the course of deploying DeepSeek-V4-Flash on a cluster of 8× RTX PRO 6000 Blackwell GPUs, the assistant and user encountered a subtle but critical quality issue: the model would produce degenerate, repetitive outputs when clients sent requests without an explicit temperature value. The root cause was not a bug in the model weights, nor a flaw in the custom MMA attention kernels that had been painstakingly optimized over the preceding engineering campaign. It was something far simpler—and far more instructive about the hidden assumptions that production inference systems make about sampling parameters.

Message [msg 12788] captures the moment of verification: the assistant has just modified the model's generation_config.json file to set a default temperature of 0.6 and top_p of 0.95, restarted the prefill and decode servers that form the prefill-decode (PD) disaggregation architecture, and is now running a live test to confirm the fix works. This single message, appearing mundane at first glance, represents the culmination of a deep debugging journey through sglang's sampling parameter resolution pipeline, the OpenAI API protocol layer, and the interplay between server-side defaults and client-side omissions.

The Context: Why Temperature Matters

The story begins with a discovery made several messages earlier ([msg 12780]–[msg 12782]). The user had observed that the deployed DeepSeek-V4-Flash model was producing incoherent output—repetitive text, truncated responses, and what appeared to be a "degeneration" pattern where the model would loop on short phrases. The assistant initially investigated whether the fp8 KV cache quantization was causing quality degradation, but a test at temperature 0.6 revealed the truth: the model produced perfectly coherent output (valid HTML, zero degeneration, correct element IDs) when sampling was enabled, but degenerated at temperature 0. The issue was greedy decoding, not quantization.

DeepSeek's family of models (V3, R1, V4-Flash) are known to require a non-zero temperature for coherent generation. The model's training and alignment assume stochastic sampling; at temperature 0, the logit distribution collapses to the single most likely token at each position, which for these models produces a degenerate mode where repetition and truncation dominate. This is a documented characteristic, not a bug—but it becomes a deployment problem when client harnesses omit the temperature parameter, because the inference server must decide what default to apply.

The user's question in [msg 12782] was direct: "Can we set default temperature for calls which don't set it?" This seemingly simple request triggered a multi-step investigation into how sglang resolves sampling parameters, which classes own the default values, and which configuration knob would actually take effect for the OpenAI-compatible API layer.

The Investigation: Tracing the Sampling Parameter Pipeline

The assistant's reasoning in the messages preceding [msg 12788] reveals a careful, methodical approach to understanding sglang's parameter resolution. The first step was to examine the model's generation_config.json ([msg 12783]), which contained generic transformers defaults: temperature: 1.0, top_p: 1.0. These are the defaults that HuggingFace's transformers library assigns when a model doesn't specify its own sampling preferences—they are not DeepSeek's recommended values (which are 0.6 and 0.95 respectively).

The assistant then traced the code path through serving_chat.py and protocol.py to understand how the OpenAI API layer handles omitted temperature. The critical finding was that ChatCompletionRequest.temperature defaults to 1.0 as a Python float, not None ([msg 12785]). This means that when a client omits the temperature field, the request object still carries temperature=1.0 into the sampling parameter resolution logic. The to_sampling_params() method checks whether self.temperature is None to decide whether to fall back to the model's generation config—but since the default is 1.0, not None, the fallback path is never triggered for temperature.

However, the assistant discovered that there are multiple request classes in the protocol layer, and the actual ChatCompletionRequest class (around line 680) defines temperature: Optional[float] = None. This is the class used for chat completions, and its None default means that omitted temperature values do trigger the fallback to default_params, which is populated from the model's generation_config.json when sampling_defaults="model" is configured. The resolution chain is:

If the request explicitly sets temperature, use that value
If temperature is None, check default_params (from generation_config)
If still not found, fall back to _DEFAULT_SAMPLING_PARAMS["temperature"] which is 0.7 The problem was that step 2 was resolving to 1.0 (the model's generation_config default), which is higher than DeepSeek's recommended 0.6 but not catastrophic. The real issue was that the user's harness was explicitly sending temperature=0, which bypassed the entire fallback chain and forced greedy decoding. The assistant correctly identified that the cleanest fix was to edit generation_config.json directly ([msg 12787]), setting temperature: 0.6 and top_p: 0.95. This ensures that any request that omits temperature (leaving it None) will resolve to 0.6 through the default_params path, while requests that explicitly set temperature (including 0) will still honor their own value. The assistant also considered using --preferred-sampling-params but correctly determined that this flag would be overridden by the resolved temperature value before it reached the sampling logic.

Message 12788: The Verification

Message [msg 12788] is the verification step. It contains three distinct phases of testing, each designed to confirm a different aspect of the fix:

Phase 1: Server Readiness

The assistant runs a polling loop that checks the systemd journal for both the prefill and decode servers, waiting for the "fired up and ready" log message. This is necessary because the servers were restarted in the previous message ([msg 12787]) and may take time to load the model weights, initialize the KV cache, and register with the router. The loop runs for up to 280 seconds (14 iterations × 20 seconds), polling every 20 seconds.

The output shows the prefill server ready at 40 seconds and the decode server ready at 60 seconds. This asymmetry is expected: the prefill server (TP4 on GPUs 0–3) has a smaller model shard to load, while the decode server (TP4 on GPUs 4–7) must initialize the full KV cache and attention kernels. The "READY" signal at 60 seconds confirms both servers are operational.

Phase 2: The Stochasticity Test

The core verification is a Python script sent via SSH that sends two identical chat completion requests without specifying a temperature. The prompt is a simple creative writing task: "Write one short creative sentence about the ocean. No preamble." The script calls gen() twice and compares the outputs.

The results are revealing:

Run 1: "The ocean sings a song of salt and stars."
Run 2: "The ocean's dark heart pulses with ancient, silent storms." Both outputs are coherent, creative, and non-repetitive. They are also different—the script confirms DIFFER (=> temp>0 default applied): True. This is the critical validation: if the temperature were 0 (greedy), both runs would produce identical output. If the temperature were 1.0 (the old default), the outputs would also differ but with higher entropy. The fact that they differ confirms that temperature > 0 is being applied, and the qualitative coherence confirms that the value is reasonable (around 0.6, not the old 1.0 which can produce overly random output). The choice of prompt is deliberate: "No preamble" instructs the model to output only the sentence itself, making the comparison clean. The 4000-token max_tokens ensures the model has sufficient generation budget to complete a coherent thought. The use of repr() with a 120-character slice captures enough of each response to assess quality while keeping the output readable in the terminal.

Phase 3: Log Verification

The final command checks the server logs for confirmation that the generation config was loaded:

Jun 18 11:26:35 dflash-train bash[168055]: [2026-06-18 11:26:35] Using default chat sampling params from model generation config...

This log message, emitted by serving_chat.py at line 188, confirms that the servers picked up the modified generation_config.json during startup. The assistant had traced this exact code path in [msg 12787] to understand how default_sampling_params gets populated, and the log confirms the mechanism works as expected.

Assumptions and Their Validity

The assistant made several assumptions in this message, all of which proved correct:

That the servers would reload the generation config on restart: This is a safe assumption for any production inference system—configuration files are read at startup, not hot-reloaded. The log verification confirms this.
That the OpenAI API layer would pass through omitted temperature as None: This was verified in the code investigation ([msg 12785]–[msg 12786]) where the assistant confirmed that ChatCompletionRequest.temperature is Optional[float] = None. The test results confirm this assumption.
That two identical requests without temperature would produce different outputs: This assumes the model's RNG is seeded differently for each request (or not seeded at all), which is the standard behavior for production serving. The differing outputs confirm this.
That the PD-disaggregation router (port 30001) would correctly forward requests to the decode server: The PD architecture splits prefill and decode across different GPU sets, with a router coordinating between them. The successful responses confirm the router is functioning.
That temperature 0.6 is the correct default for DeepSeek-V4-Flash: This was based on DeepSeek's published recommendations for V3 and R1, which the assistant extrapolated to V4-Flash. The coherent output validates this assumption, though the test doesn't prove 0.6 is optimal—only that it's non-degenerate.

Input Knowledge Required

To fully understand this message, one needs:

Understanding of sglang's PD disaggregation architecture: The prefill and decode servers run as separate systemd services, each with their own model shard (TP4), and a router on port 30001 coordinates between them. The assistant polls both services' journals independently.
Knowledge of the OpenAI ChatCompletion API: The test script sends a standard POST /v1/chat/completions request with model, messages, and max_tokens fields, omitting temperature. The response format with choices[0].message.content is standard OpenAI protocol.
Understanding of stochastic sampling in LLMs: The core insight—that temperature > 0 produces stochastic outputs while temperature 0 produces deterministic greedy decoding—is fundamental to the test's design.
Familiarity with systemd journalctl: The assistant uses journalctl -u sglang-dsv4-decode --since "3 min ago" to filter recent logs, and grep -c "fired up and ready" to count readiness messages. This is a standard pattern for checking service health in Linux deployments.
Knowledge of the generation_config.json mechanism: The assistant had to trace through sglang's source code to understand that default_sampling_params is populated from the model's generation_config.json and used as the fallback for omitted sampling parameters.

Output Knowledge Created

This message produces several important outputs:

Empirical confirmation that the generation_config edit works: The test proves that modifying generation_config.json is the correct mechanism for setting default sampling parameters in sglang's OpenAI-compatible API layer.
A validated test methodology: The "two identical requests, compare outputs" pattern is a simple but effective way to verify that temperature > 0 is being applied. This methodology can be reused for any future sampling parameter changes.
Confidence in the PD-disaggregation deployment: The successful requests through the router confirm that the prefill and decode servers are communicating correctly, the KV cache transfer is working, and the systemd services are stable.
A baseline for quality: The coherent, creative outputs ("The ocean sings a song of salt and stars") demonstrate that the model is functioning correctly at the new default temperature, providing a quality baseline for future optimization work.
Documentation of the resolution path: The assistant's reasoning, the code investigation, and the verification test together form a complete record of how a seemingly simple configuration change required deep understanding of sglang's internals.

The Thinking Process

The assistant's reasoning in this message is notable for its precision and methodical structure. The thinking is organized into a clear plan:

Wait for readiness: The assistant explicitly states the need to wait for both servers to be ready before testing. This shows an understanding of the PD architecture's asynchronous startup.
Verify the log picks up the 0.6 default: Rather than assuming the configuration was loaded, the assistant plans to check the journal for the "default chat sampling params" log message. This is a verification step that confirms the mechanism works at the code level.
Verify omitted-temp requests are stochastic: The core test is designed to distinguish between temperature=0 (greedy, deterministic) and temperature>0 (stochastic). The assistant explicitly frames the expected outcome: "omitted-temp requests are now stochastic (temp 0.6) not greedy." The reasoning also shows awareness of the distinction between "omitted" and "explicitly set" parameters. The assistant notes that the fix only affects requests that omit temperature—requests that explicitly set temperature=0 will still get greedy decoding. This is by design: the user asked for a default for unset calls, not a forced override. The bash scripting in the message is sophisticated: the polling loop uses journalctl with a time window (--since "3 min ago"), counts readiness messages with grep -c, and uses parameter expansion (${rd:-?}) to handle empty output. The Python heredoc uses urllib.request (the standard library HTTP client) to avoid dependency on requests, which may not be installed in the production environment. The use of repr() for output ensures that special characters are visible in the terminal.

Conclusion

Message [msg 12788] is a masterclass in verification-driven engineering. The assistant doesn't just apply a configuration change and move on—it designs a test that validates the fix at multiple levels: server readiness, stochastic output behavior, and log confirmation. The test is minimal but sufficient: two identical requests, compared for difference, with a creative writing prompt that makes quality assessment intuitive.

The broader significance of this message lies in what it reveals about production inference systems. A temperature default is a trivial configuration parameter in isolation, but getting it to actually take effect required tracing through five layers of code: the OpenAI protocol definition, the to_sampling_params() resolver, the default_params population from generation_config, the sampling_defaults flag, and the PD-disaggregation router. The assistant's systematic investigation—checking the generation_config, reading the protocol code, tracing the resolution logic, and testing empirically—is a model of how to debug configuration issues in complex distributed systems.

The fix itself is elegant: by editing generation_config.json rather than adding command-line flags or patching code, the assistant leverages sglang's existing parameter resolution architecture. The change is persistent across server restarts, compatible with the model's native configuration format, and invisible to clients. It's the kind of fix that, once applied, seems obvious—but getting there required deep knowledge of the system's internals and a willingness to trace code paths until the right knob was found.