The Temperature Default Problem: Tracing Sampling Parameters Through SGLang's OpenAI API Layer

In the course of deploying a production inference server for the DeepSeek-V4-Flash model on 8× RTX PRO 6000 Blackwell GPUs, a subtle but critical quality issue emerged: when clients sent requests without specifying a temperature parameter, or explicitly set temperature to 0, the model produced degenerate output — repetitive, incoherent text that rendered the service unusable. The root cause had been identified in earlier messages: greedy decoding (temperature=0) caused the model to collapse into repetition, while the recommended temperature of 0.6 produced coherent, high-quality output. But fixing this at the client level was impractical — the assistant needed to make the server itself apply a sensible default.

Message [msg 12784] captures the moment when the assistant pivots from confirming the temperature fix to understanding how to make it stick at the server level. It is a detective story about tracing the flow of a single parameter through a complex codebase, and it reveals the hidden complexity behind what seems like a simple configuration change.

The Context: A Quality Crisis Resolved

The story leading up to this message is one of escalating diagnosis. The deployment had been plagued by what looked like model degeneration — outputs filled with repeated spaces and closing tags, suggesting a fundamental quality problem. The user had initially suspected the FP8 KV cache quantization, which DeepSeek-V4-Flash forces intrinsically through its MLA (Multi-Head Latent Attention) architecture. But the assistant's investigation told a different story.

In [msg 12778] and [msg 12779], the assistant systematically ruled out the cache quantization hypothesis. By testing temperature 0.6 against temperature 0, it demonstrated that the degeneration was a sampling artifact, not a precision artifact. At temperature 0.6 with adequate token budgets, the model produced full, coherent HTML pages with valid structure, matching IDs and handlers, and zero repetition. At temperature 0, the model either produced nothing (consuming its entire budget on reasoning) or degenerated into repetitive patterns. The FP8 KV cache, while a legitimate concern for precision-sensitive applications, was not the culprit.

This left a clear action item: the server needed to default to temperature 0.6 for requests that didn't specify one, and ideally override requests that explicitly set temperature 0. The user's question in [msg 12782] — "Can we set default temperature for calls which don't set it?" — was the natural next step.

The Initial Investigation: Two Mechanisms, One Unknown

The assistant's first response to this question, in [msg 12783], identified two candidate mechanisms in SGLang:

  1. --sampling-defaults: A server flag that can use the model's generation_config.json as the source of default sampling parameters.
  2. --preferred-sampling-params: A flag that sets explicit server-side defaults, documented to merge as {**preferred, **request} so that explicit request parameters override the preferred defaults. The assistant checked the model's generation_config.json and found it contained generic transformers defaults: temperature: 1.0, top_p: 1.0. These were not DeepSeek's recommended values (0.6 and 0.95), but a generic fallback from the HuggingFace transformers library. Using sampling_defaults=model would give temperature 1.0 — better than greedy decoding but still not ideal. The assistant also searched the codebase for how preferred_sampling_params and sampling_defaults are handled, finding references in tokenizer_manager.py but the output was truncated. This left a critical question unanswered: does the OpenAI API layer fill in temperature=1.0 as a default when the client omits it, before the request reaches the sampling logic?

Message [msg 12784]: The Deep Dive

This is where the subject message begins. The assistant's reasoning shows a sophisticated understanding of the problem's architecture. Let me quote the reasoning in full:

Looking at the generation_config.json, the model's default sampling parameters are temperature 1.0 and top_p 1.0 with sampling enabled, which is actually a generic transformers default rather than DeepSeek's specific recommendation. Using sampling_defaults=model would apply these settings, giving us temperature 1.0—higher than ideal but at least not greedy decoding.

>

The key issue is understanding how the preferred sampling parameters merge with request-level overrides. The code merges them so that explicit request parameters override the preferred defaults, meaning the preferred settings only apply when a request doesn't specify its own sampling parameters.

>

But there's a critical dependency: if the OpenAI serving layer automatically fills in temperature=1.0 as a default when the client omits it, then by the time the request reaches the tokenizer manager, the temperature is already set, and the preferred sampling parameters get overridden anyway. So whether preferred_sampling_params actually works for temperature depends on whether the OpenAI layer leaves temperature unset when the client omits it, or fills it with a default value.

This reasoning reveals a deep understanding of how defaults propagate through layered systems. The assistant correctly identifies that there are two default mechanisms at play: the OpenAI API layer's defaults (set in the Pydantic model definition) and the server's own preferred sampling parameters. The question is which one wins when a client omits temperature entirely.

The assistant considers two approaches:

  1. Edit generation_config.json directly to set temperature=0.6 and top_p=0.95, then use sampling_defaults=model. This would make the model's config the authoritative source for defaults, bypassing the OpenAI layer's defaults.
  2. Use --preferred-sampling-params with the same values, relying on the merge logic to apply them when request parameters are absent. But approach 2 has a fatal flaw if the OpenAI layer fills in temperature=1.0 before the merge happens. The assistant recognizes this: "if the OpenAI serving layer automatically fills in temperature=1.0 as a default when the client omits it, then by the time the request reaches the tokenizer manager, the temperature is already set, and the preferred sampling parameters get overridden anyway." This is the kind of insight that comes from experience with layered API frameworks. The OpenAI-compatible layer in SGLang uses Pydantic models to parse incoming requests, and Pydantic has its own default mechanism. If temperature: float = 1.0 is the Pydantic default, then a request without temperature gets parsed as temperature=1.0 before any server-side logic sees it. The preferred_sampling_params merge would then see an explicit temperature=1.0 and not override it, because the merge is {**preferred, **request} — request wins.

The Bash Command: Reading the Source

The assistant's next action is to run a bash command that searches the SGLang source code for the relevant default definitions. The command is:

timeout 20 ssh -o StrictHostKeyChecking=no root@10.1.230.171 'cd /root/sglang-dsv4 && 
echo "=== ChatCompletionRequest temperature default ==="
grep -nE "temperature.*=|top_p.*=" python/sglang/srt/entrypoints/openai/protocol.py | grep -iE "float|none|=.*[0-9]" | head
echo "=== sampling_defaults resolution (model vs openai) ==="
grep -rnE "sampling_defaults" python/sglang/srt/entrypoints/openai/protocol.py python/sglang/srt/entrypoints/openai/serving_chat.py | head
echo "=== how temperature default is decided ==="
grep -nE "sampling_defaults|temperature" python/sglang/srt/entrypoints/openai/protocol.py | grep -iE "sampling_defaults|0\.7|1\.0|default" | head' </dev/null

This command is carefully structured to answer three specific questions:

  1. What is the Pydantic default for temperature in ChatCompletionRequest? The answer is found at line 337: temperature: float = 1.0. This confirms the assistant's fear — the OpenAI layer defaults temperature to 1.0 when omitted.
  2. How does sampling_defaults interact with the OpenAI protocol? The grep for sampling_defaults in protocol.py and serving_chat.py returns empty results, suggesting this mechanism is handled elsewhere (likely in the tokenizer manager, as hinted in the previous message).
  3. How is the temperature default ultimately decided? The grep shows line 337 with temperature: float = 1.0 for ChatCompletionRequest, but also reveals that other request types (lines 680, 1381, 1579) use Optional[float] = None, meaning they leave temperature unset when omitted. Line 1733 shows temperature: float = 0.0 for yet another endpoint. This last finding is particularly interesting. The fact that some request types use Optional[float] = None while others use float = 1.0 means the behavior depends on which API endpoint the client uses. The standard /v1/chat/completions endpoint (ChatCompletionRequest) defaults to 1.0, which would defeat preferred_sampling_params. But other endpoints might leave it unset, allowing the server's preferred defaults to apply.

The Thinking Process: A Model of Diagnostic Reasoning

What makes this message particularly valuable as a case study is the assistant's reasoning process. It demonstrates several hallmarks of expert debugging:

1. Layered Default Analysis

The assistant doesn't just look at one configuration point. It traces the temperature parameter through multiple layers: the client request → the OpenAI Pydantic parser → the tokenizer manager's sampling logic → the actual model inference. At each layer, a default may be applied, and the assistant correctly identifies that the first layer to set a non-None value will prevent downstream defaults from taking effect.

2. Hypothetical Reasoning

The assistant considers multiple scenarios and their implications: "if the OpenAI serving layer automatically fills in temperature=1.0 as a default when the client omits it, then by the time the request reaches the tokenizer manager, the temperature is already set." This is a testable hypothesis, and the bash command is designed to verify it.

3. Pragmatic Escalation

When the assistant realizes that preferred_sampling_params might not work due to the OpenAI layer's defaults, it escalates to a more invasive but more reliable approach: modifying generation_config.json directly. This shows an understanding that sometimes the cleanest configuration mechanism isn't the one that actually works, and you need to go deeper.

4. Empirical Verification

Rather than relying solely on code reading, the assistant plans to test the configuration empirically: "modify generation_config.json to set temperature to 0.6 and top_p to 0.95, add --preferred-sampling-params with the same values as extra insurance, restart both servers, then send requests without temperature specified twice to check if they produce different outputs." This is a classic debugging loop: form a hypothesis, implement a fix, test it, observe the result.

Assumptions and Potential Mistakes

The assistant makes several assumptions in this message that are worth examining:

Assumption 1: The OpenAI layer's default is the blocking issue

The assistant assumes that temperature: float = 1.0 in the Pydantic model means the temperature is always set to 1.0 when omitted. This is correct for standard Pydantic behavior — a field with a default value will always have that value if not provided. However, the assistant hasn't verified whether SGLang's code path actually uses the Pydantic-parsed value directly or whether there's additional processing that could distinguish between "explicitly set to 1.0" and "omitted, defaulted to 1.0." Some frameworks use sentinel values or None to distinguish these cases, even when the Pydantic model has a default.

Assumption 2: sampling_defaults=model reads generation_config.json

The assistant assumes that setting sampling_defaults=model will cause the server to read the model's generation_config.json and use its values. This is a reasonable assumption given the flag's name, but the assistant hasn't verified the actual code path. The grep for sampling_defaults in the OpenAI protocol files returned empty, suggesting the mechanism might be handled differently than expected.

Assumption 3: DeepSeek's recommended values apply to V4-Flash

The assistant assumes that DeepSeek-V3 and R1's recommended sampling parameters (temperature 0.6, top_p 0.95) apply to V4-Flash as well. This is a reasonable extrapolation — models from the same family typically share optimal sampling configurations — but it's not verified. The model's own generation_config.json specifies temperature 1.0, which could indicate that V4-Flash has different optimal sampling behavior.

Assumption 4: The fix is server-side only

The assistant focuses entirely on server-side configuration changes, assuming that all clients will benefit from the default change. This ignores the possibility that some clients might explicitly set temperature to values that override the server defaults in undesirable ways. The assistant acknowledges this briefly: "calls that explicitly set it to 0 would need fixing in the harness itself." But the primary fix is server-side.

Input Knowledge Required

To fully understand this message, the reader needs:

  1. Understanding of OpenAI API conventions: The /v1/chat/completions endpoint accepts a temperature parameter that controls sampling randomness. Values range from 0 (greedy, deterministic) to 2 (highly random). The OpenAI specification defaults temperature to 1.0 when not specified.
  2. Knowledge of Pydantic defaults: The Python Pydantic library, used by SGLang for request parsing, sets field values to their declared defaults when not provided. A field declared as temperature: float = 1.0 will always be 1.0 if the client omits it.
  3. Understanding of SGLang's architecture: SGLang has a layered architecture where OpenAI-compatible request parsing happens in protocol.py, then the parsed request is passed to the tokenizer manager for sampling logic, and finally to the model backend for inference.
  4. Knowledge of DeepSeek-V4's sampling behavior: The model degenerates at temperature 0 (greedy decoding) and produces coherent output at temperature 0.6. This was established through empirical testing in earlier messages.
  5. Familiarity with the deployment context: The server runs on 8× RTX PRO 6000 Blackwell GPUs with a custom SGLang build that includes DeepSeek-V4 hooks forcing FP8 KV cache quantization.

Output Knowledge Created

This message produces several important pieces of knowledge:

  1. The OpenAI layer defaults temperature to 1.0: Line 337 of protocol.py confirms temperature: float = 1.0 for ChatCompletionRequest. This means any client that omits temperature gets 1.0, not the server's preferred default.
  2. Different endpoints have different default behaviors: Lines 680, 1381, and 1579 show Optional[float] = None for other request types, meaning they leave temperature unset when omitted. Line 1733 shows temperature: float = 0.0 for yet another endpoint. The behavior is endpoint-specific.
  3. sampling_defaults is not handled in the OpenAI protocol layer: The grep for sampling_defaults in protocol.py and serving_chat.py returned empty, confirming this mechanism is handled at a lower level (likely in the tokenizer manager).
  4. The model's generation_config.json contains generic defaults: Temperature 1.0 and top_p 1.0 are HuggingFace transformers defaults, not DeepSeek-specific recommendations. This means sampling_defaults=model would not give the desired behavior without modifying the config file.
  5. A two-pronged approach is needed: The assistant identifies that both generation_config.json modification and --preferred-sampling-params may be necessary, with the former being the more reliable mechanism for the ChatCompletion endpoint.

The Broader Significance

This message exemplifies a class of problems that are increasingly common in the LLM deployment space: the tension between API compatibility layers and model-specific optimal configurations. OpenAI's API specification was designed for GPT models and has baked-in assumptions about default parameters. When serving a different model family (DeepSeek-V4) through an OpenAI-compatible layer, these assumptions can silently degrade quality.

The temperature default problem is particularly insidious because it doesn't manifest as an error — the server returns valid responses, and the client gets what it asked for (temperature 1.0). But the output quality is suboptimal, and the degradation is invisible to monitoring unless someone specifically checks for it. This is why the assistant's careful tracing of the default propagation path is so valuable: it reveals a hidden quality gap that would otherwise go unnoticed.

The solution — modifying generation_config.json and using sampling_defaults=model — is a pragmatic workaround that exploits a gap in the default propagation. By making the model config the authoritative source, the assistant bypasses the OpenAI layer's defaults entirely. But this is a fragile fix: any future SGLang update that changes how sampling_defaults resolves could break it. A more robust solution would require changes to the OpenAI protocol layer itself, perhaps by using Optional[float] = None for temperature and letting the server's preferred defaults fill it in.

Conclusion

Message [msg 12784] captures a pivotal moment in the deployment of a production LLM inference server. The assistant has identified the root cause of a quality issue (greedy decoding at temperature 0), confirmed the fix (temperature 0.6), and is now working to make that fix permanent and automatic. The message shows expert-level diagnostic reasoning: tracing a parameter through multiple software layers, identifying where defaults are applied, and formulating a testable hypothesis about the blocking issue.

The bash command's output reveals the critical finding: the OpenAI API layer defaults temperature to 1.0 when omitted, which would defeat the server's preferred sampling parameters. This discovery forces the assistant to escalate from a clean configuration change to a more invasive approach — modifying the model's generation_config.json directly.

What makes this message particularly instructive is the assistant's reasoning process. It doesn't just apply a configuration flag and move on; it thinks through the entire propagation path, identifies potential failure modes, and designs an empirical test to verify its understanding. This is the kind of deep, systematic thinking that separates a reliable deployment from a fragile one. The temperature default problem, once understood, becomes a cautionary tale about the hidden assumptions in API compatibility layers — and a testament to the value of tracing parameters all the way down.