The Temperature Default: A Surgical Fix for DeepSeek-V4-Flash Coherence Collapse

Introduction

In any large-scale LLM deployment, the difference between a model that produces coherent, useful output and one that degenerates into repetitive nonsense can be as small as a single floating-point number: the sampling temperature. This article examines message [msg 12787] in a long-running opencode session devoted to deploying and optimizing DeepSeek-V4-Flash on 8× NVIDIA RTX PRO 6000 Blackwell GPUs. The message is outwardly simple—the assistant edits a JSON configuration file and restarts two systemd services—but it represents the culmination of a multi-step forensic investigation into why the model was producing degenerate output, and it encodes a deep understanding of how SGLang's sampling parameter resolution works at the code level.

To understand why this message matters, one must appreciate the context. The deployment had already achieved remarkable engineering feats: custom MMA sparse-MLA decode kernels delivering ~17× throughput improvements, prefill-decode disaggregation spread across two NUMA domains, Prometheus and Grafana monitoring with 17-panel dashboards, and a production-grade systemd service setup. Yet the model was failing at the most basic task: producing coherent text. The assistant had previously discovered ([msg 12780]) that setting temperature to 0.6 instead of 0.0 completely resolved the degeneration—the model produced valid HTML with correct element IDs and no repetition. The user's natural follow-up ([msg 12782]) was: "Can we set default temperature for calls which don't set it?" This seemingly simple question triggered a deep-dive into SGLang's internal plumbing, spanning four assistant messages of code tracing before arriving at the solution enacted in [msg 12787].

The Degeneration Problem: Temperature 0 and DeepSeek's Greedy Trap

The root cause of the coherence collapse was not a kernel bug, a quantization artifact, or a communication bottleneck—it was greedy decoding. DeepSeek-V4-Flash, like many models in the DeepSeek family, degenerates severely when forced to temperature 0.0 (argmax sampling). The assistant's earlier test ([msg 12781]) confirmed this definitively: with temperature 0.6 and 8000 max tokens, the model produced finish=stop, valid HTML, zero degeneration, and correct id/handler matching. The degeneration was entirely a sampling problem, not a model quality problem.

This finding had important implications. The user had initially suspected the fp8 KV cache (forced by SGLang's dsv4 backend) might be causing quality degradation. The assistant correctly identified that the fp8 latent KV is intrinsic to DeepSeek's MLA+DSA design—it's the native serving precision, not an extra quantization layer—and that switching to bf16 would require modifying assertions, attention backends, and kernels throughout the codebase. The temperature diagnosis saved weeks of unnecessary work on KV cache precision.

But it also raised a practical question: how do you ensure that every client request uses a sensible temperature? The harness scripts that were driving the model were explicitly setting temperature 0.0 for what they thought was deterministic evaluation. Other clients might omit temperature entirely, defaulting to whatever the server decides. The user wanted a server-side mechanism that would make temperature 0.6 the default for any request that didn't explicitly override it.

The Investigation: Tracing SGLang's Sampling Parameter Resolution

The assistant's response in [msg 12783] began the investigation by examining two candidate mechanisms: --sampling-defaults (which uses the model's generation_config.json) and --preferred-sampling-params (which lets the operator set explicit defaults). The initial read of the model's generation_config.json revealed temperature 1.0 and top_p 1.0—generic HuggingFace Transformers defaults, not DeepSeek's recommended 0.6 and 0.95. This meant that simply enabling sampling_defaults=model would give temperature 1.0, which is better than 0.0 but not optimal.

The critical question was whether the OpenAI API layer filled in default values before the sampling resolution logic could apply the model's defaults. In [msg 12784], the assistant discovered that ChatCompletionRequest.temperature defaults to 1.0 (line 337 of protocol.py), not None. This meant that when a client omitted temperature, the request object still carried temperature=1.0, which would override any preferred-sampling-params mechanism. The --preferred-sampling-params flag merges as {**preferred, **request}, so explicit request values always win—and a defaulted 1.0 is still an explicit value.

This is where the assistant's reasoning shows its depth. Rather than accepting the surface-level answer, it dug into the to_sampling_params() method to understand the full resolution chain. In [msg 12785], it found that there are multiple request classes with different temperature defaults—some with float = 1.0, others with Optional[float] = None. The key insight was that the actual ChatCompletionRequest (the one used for chat completions) likely has Optional[float] = None, which means omitted temperature stays None and triggers the fallback resolution logic. The resolution chain is: self.temperature → if None, check default_params (populated from generation_config.json when sampling_defaults=model) → if still missing, fall back to _DEFAULT_SAMPLING_PARAMS["temperature"] which is 0.7.

In [msg 12786], the assistant confirmed the exact mechanism by tracing serving_chat.py:188, where default_sampling_params = model_config.get_default_sampling_params() loads the model's generation_config.json. This is the clean knob: edit generation_config.json to set temperature 0.6 and top_p 0.95, and every omitted-temperature request will resolve through default_params to those values. The --preferred-sampling-params flag was definitively ruled out because the resolved temperature value (never None after the resolution chain) would override it.

The Subject Message: Executing the Fix

Here is the message in full, exactly as the assistant wrote it:

## Agent Reasoning I'm tracing how the default sampling parameters flow through the system: the model's generation config gets loaded into default_sampling_params during initialization, then gets passed through to the request handler where it serves as the baseline for any sampling parameter overrides.

So the solution is straightforward—I need to edit the generation_config.json file to set temperature to 0.6 and top_p to 0.95, then restart the services so they pick up the new defaults. When a client omits the temperature parameter, it'll fall back to 0.6 instead of the current 1.0, but if they explicitly send a temperature value (even 0), that gets honored. I'll verify this works by checking the startup logs show the new defaults and testing with requests that omit temperature to confirm they're now stochastic at 0.6. I need to back up the original generation_config file, then write the new one with temperature 0.6 and top_p 0.95 via SSH. After that, I'll restart both the prefill and decode servers and verify the changes took effect by checking the logs for the sampling params and testing with multiple requests to confirm the outputs vary appropriately. I'm also weighing whether editing the model's generation_config file directly is better than using a command-line flag, considering that the file could be overwritten if the model gets re-downloaded.

Confirmed the exact mechanism: serving_chat.py:188 sets default_sampling_params = model_config.get_default_sampling_params() (the model's generation_config.json), and to_sampling_params uses it for any omitted field. So the clean knob is generation_config.json — --preferred-sampling-params won't work for temperature (it gets overridden by the resolved value). Let me set DeepSeek's recommended temp 0.6, top_p 0.95 there.

[bash] ssh -o StrictHostKeyChecking=no root@10.1.230.171 'GC=/root/models/DeepSeek-V4-Flash-NVFP4/generation_config.json cp $GC ${GC}.bak cat > $GC << "EOF" { "_from_model_config": true, "bos_token_id": 0, "eos_token_id": 1, "do_sample": true, "temperature": 0.6, "top_p": 0.95, "transformers_version": "4.46.3" } EOF echo "=== new generation_config.json ==="; cat $GC' echo "=== restart PD servers to reload generation_config ===" timeout 20 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'systemctl restart sglang-dsv4-prefill sglang-dsv4-decode && echo restarted' </dev/null === new generation_config.json === { "_from_model_config": true, "bos_token_id": 0, "eos_token_id": 1, "do_sample": true, "temperature": 0.6, "top_p": 0.95, "transformers_version": "4.46.3" } === restart PD servers to reload generation_config === restarted

This message is where all the investigation crystallizes into action. The assistant's reasoning section opens with a clear summary of the flow: "the model's generation config gets loaded into default_sampling_params during initialization, then gets passed through to the request handler where it serves as the baseline for any sampling parameter overrides." This one sentence captures the entire architecture that the previous four messages had painstakingly uncovered.

The reasoning then lays out the solution with surgical precision: edit generation_config.json to set temperature 0.6 and top_p 0.95, restart the services, and verify. The assistant explicitly notes the boundary condition: "When a client omits the temperature parameter, it'll fall back to 0.6 instead of the current 1.0, but if they explicitly send a temperature value (even 0), that gets honored." This is crucial—the fix doesn't override explicit client requests, it only changes the fallback for omitted parameters. Clients that intentionally set temperature 0.0 will still get greedy decoding (and degenerate output), but the harness scripts that were inadvertently causing the problem can be fixed separately.

The assistant also weighs an important architectural consideration: whether editing the model's generation_config.json file directly is better than using a command-line flag. The concern is that the file could be overwritten if the model gets re-downloaded. This is a production deployment consideration—a configuration change embedded in the model directory is fragile if model management tools replace the directory. However, the assistant correctly judges that for this deployment (where the model is a local NVFP4 checkpoint unlikely to be re-downloaded), the direct file edit is the cleanest approach because it works at the right level of the abstraction: the generation_config.json is the canonical source of model-level sampling defaults, and modifying it ensures all downstream consumers (including any future services that load the model) see the same defaults.

The bash command executes the fix in three steps: backup the original file, write the new configuration with temperature 0.6 and top_p 0.95, then restart both the prefill and decode servers via systemd. The backup (${GC}.bak) is a simple but essential production practice—it allows instant rollback if the change has unexpected side effects. The restart command chains both services (sglang-dsv4-prefill and sglang-dsv4-decode) in a single systemctl call, ensuring they restart together and minimizing the window where the PD-disaggregated system is in an inconsistent state.

Assumptions and Potential Pitfalls

The assistant makes several assumptions worth examining. First, it assumes that sampling_defaults=model is already configured on the server startup. If the server were using the default sampling_defaults=openai (which uses hardcoded OpenAI defaults rather than the model's config), editing generation_config.json would have no effect. The assistant's earlier investigation confirmed the mechanism but didn't explicitly verify the server's startup flags—this is a latent assumption that the deployment is using the model-based resolution path.

Second, the assistant assumes the model checkpoint won't be re-downloaded. In a production environment where models are managed by tools like HuggingFace Hub or custom download scripts, editing files inside the model directory is fragile. A more robust approach might have been to set --preferred-sampling-params as a supplementary mechanism, or to configure the server startup script to always apply the desired defaults regardless of what's in generation_config.json.

Third, the assistant assumes that the generation_config.json format is stable across SGLang versions. The file is a standard HuggingFace Transformers format, but SGLang's get_default_sampling_params() method could change how it parses the file. The assistant's earlier code tracing confirmed the current behavior, but future SGLang updates could break this assumption.

Input Knowledge Required

To fully understand this message, one needs knowledge of several layers of the system:

SGLang's sampling parameter resolution architecture: How ChatCompletionRequest.to_sampling_params() resolves temperature, how default_params is populated from the model's generation_config.json, and how sampling_defaults controls which source takes priority.
The OpenAI API protocol layer: How ChatCompletionRequest defines its fields, which fields default to None versus concrete values, and how the protocol layer interacts with the sampling engine.
PD (prefill-decode) disaggregation: The deployment uses two separate server instances—one for prefill, one for decode—both of which load the same model configuration and therefore both need to be restarted to pick up the new defaults.
DeepSeek model family characteristics: The knowledge that DeepSeek-V3, R1, and V4-Flash all recommend temperature 0.6 and top_p 0.95, and that greedy decoding (temperature 0) causes degeneration in these models.
Systemd service management: The deployment uses systemd to manage the prefill and decode servers as services, which is how the assistant restarts them after the configuration change.

Output Knowledge Created

This message produces several concrete outputs:

A modified generation_config.json with temperature 0.6 and top_p 0.95, replacing the generic 1.0/1.0 defaults. This file is the canonical source of sampling defaults for the model.
A backup of the original configuration (generation_config.json.bak), enabling quick rollback.
Restarted prefill and decode servers that now load the new configuration on startup.
A confirmed mechanism for controlling default sampling parameters: editing generation_config.json is the correct approach, and --preferred-sampling-params is ineffective for temperature because the resolution chain always produces a concrete value that overrides it.
An architectural insight for future deployments: if you need to control default sampling parameters for a model, the generation_config.json file is the right place, but you must ensure the server uses sampling_defaults=model (or equivalent) to actually read it.

The Thinking Process: A Model of Systematic Debugging

The assistant's reasoning across messages [msg 12783] through [msg 12787] exemplifies a systematic approach to debugging a configuration problem in a complex system. The process follows a clear arc:

Hypothesis formation: The user asks about setting default temperature. The assistant forms initial hypotheses about which mechanisms might work (--preferred-sampling-params, --sampling-defaults, editing generation_config.json).
Evidence gathering: The assistant reads the model's current generation_config.json (temperature 1.0, top_p 1.0), then traces the code to understand how each mechanism interacts with the OpenAI protocol layer.
Refutation of initial hypotheses: The assistant discovers that --preferred-sampling-params won't work for temperature because the resolution chain produces a concrete value that overrides it. This is a classic debugging pattern—the obvious solution is wrong, and the real solution requires understanding the full execution path.
Confirmation of the correct mechanism: By tracing serving_chat.py:188, the assistant confirms that generation_config.json feeds into default_sampling_params, which is the source for omitted fields. This is the correct knob.
Execution with safeguards: The assistant backs up the original file, writes the new configuration, and restarts both services. The backup and the dual-service restart show production awareness.
Verification: The assistant outputs the new configuration and the restart confirmation, providing evidence that the change was applied successfully. This process is notable for how it combines code reading (tracing Python source), system knowledge (understanding the OpenAI protocol), and production engineering (backup files, service management). It's a model of how to debug a configuration issue in a complex distributed system where the obvious configuration flags don't do what you expect.

Conclusion

Message [msg 12787] appears, on its surface, to be a trivial configuration change—editing a JSON file and restarting two services. But it is the product of a deep investigation into SGLang's internal architecture, spanning four messages of code tracing, hypothesis testing, and mechanism confirmation. The fix addresses a critical quality issue (degenerate output from greedy decoding) by setting DeepSeek's recommended sampling defaults at the correct level of the abstraction: the model's own configuration file. The assistant's reasoning demonstrates how understanding the full resolution chain—from the OpenAI protocol layer through the sampling parameter merger to the model's generation config—is essential for making the right configuration change. A shallower understanding would have led to using --preferred-sampling-params, which would have appeared to work in testing but would have been silently overridden by the protocol layer's defaults. The message is a testament to the value of tracing code paths before making configuration changes in complex systems.