The Moment of Truth: A First Test Query That Reveals a Silent Crash

ssh 10.1.230.175 'curl -s --max-time 120 http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d "{\"model\": \"glm-5\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in one sentence.\"}], \"max_tokens\": 50}" 2>&1'

At first glance, message 160 in this opencode session looks unremarkable: a simple curl command sending a chat completion request to a locally running model server. The prompt is trivial — "Say hello in one sentence." — and the parameters are minimal. Yet this message is a fulcrum point in a much larger narrative. It represents the culmination of hours of intensive infrastructure work, the first real test of a freshly deployed 400-billion-parameter Mixture-of-Experts model running across eight NVIDIA Blackwell GPUs, and the precise moment at which a silent, devastating failure is first encountered. Understanding why this message was written, what it assumed, and what it ultimately reveals requires stepping back into the full arc of the deployment effort that preceded it.

The Long Road to a Simple Curl Command

To appreciate message 160, one must understand the journey that led to it. The assistant had spent the better part of a session (Segment 0) setting up a complete machine learning environment from scratch on Ubuntu 24.04: installing NVIDIA drivers version 590.48.01, configuring CUDA Toolkit 13.1, creating a Python virtual environment with uv, and resolving a notoriously difficult flash-attn build issue that required reducing parallel compilation jobs from 128 to 20 to avoid memory exhaustion. The machine was then upgraded to eight RTX PRO 6000 Blackwell GPUs, each with 96 GB of VRAM — a staggering 768 GB total.

In Segment 1, the focus shifted to deploying the GLM-5-NVFP4 model, a 400-billion-parameter quantized MoE model, using SGLang as the serving framework. This required building SGLang from its main branch to incorporate a critical SM120 shared memory fix (PR #14311) — without which the model would not run on Blackwell hardware at all. Transformers had to be upgraded to version 5.2.0 to support the model's custom glm_moe_dsa architecture. The first launch attempt crashed with an out-of-memory error during CUDA graph capture because --mem-fraction-static 0.95 reserved too much memory for the KV cache, leaving no room for runtime buffers. The assistant diagnosed this, killed the process, and relaunched with --mem-fraction-static 0.88 — a seemingly small change that made all the difference.

By message 158, the server had started successfully. The logs showed KV cache allocation of 425,664 tokens across all eight GPUs, CUDA graph capture completing for batch sizes up to 64, and the HTTP server listening on port 8000. Message 159 was the very first test query — a curl asking "Hello! What model are you?" with max_tokens=100. Message 160, the subject of this article, is the second test query, sent immediately after.

What the Message Actually Does

The command in message 160 is straightforward but worth examining closely. It uses curl with the -s (silent) flag to suppress progress output, sets a generous 120-second timeout (--max-time 120), and sends a POST request to the SGLang server's /v1/chat/completions endpoint — the standard OpenAI-compatible API. The request body specifies the model name (glm-5), a single user message asking for a one-sentence greeting, and a max_tokens limit of 50. The 2>&1 redirect ensures that any error output is captured alongside standard output.

The choice of prompt is deliberate. "Say hello in one sentence." is about as simple a request as one can make to a language model. It requires no reasoning, no tool use, no complex generation — just a straightforward, short response. This is a classic "canary" prompt: if the model cannot handle this, it cannot handle anything. The 50-token limit ensures the response will be brief, minimizing both latency and the chance of running into generation issues. The 120-second timeout is generous, accounting for the possibility that the first request after server startup might be slow due to CUDA kernel warmup or JIT compilation.

The Assumptions Embedded in This Message

Every line of this message encodes assumptions about the state of the world. The assistant assumes that the server process is still alive and healthy — a reasonable assumption given that message 158 showed the server starting successfully, and message 159 (the first test query) had not yet returned a failure. The assistant assumes that the model weights have been loaded correctly across all eight tensor-parallelism ranks, that the CUDA graphs are properly captured, that the KV cache is functional, and that the quantization format (NVFP4) is compatible with the serving stack. It assumes that the network stack on the remote machine is functional, that port 8000 is accessible, and that the HTTP endpoint will respond within two minutes.

More subtly, the assistant assumes that the model will generate correctly — that the forward pass through the quantized MoE layers will produce valid probability distributions, that the sampling logic will select reasonable tokens, and that the decode phase will not trigger any device-side errors. This assumption, as the subsequent messages reveal, is tragically incorrect.

What This Message Actually Discovers

The result of this curl command is not shown in message 160 itself — the assistant must wait for the tool call to complete. But message 161 opens with "Hmm, empty output," and message 162 reveals the truth: the server has crashed. The health endpoint returns "Connection refused," and the log inspection shows a device-side assert triggered error caused by NaN/Inf values in the probability tensor during decode. The model loads successfully, captures CUDA graphs, and starts serving — but the very first generation attempt produces numerical garbage that crashes the entire process.

This is a particularly insidious class of failure. The server starts, reports healthy, and accepts requests. Only when a request triggers the actual generation pipeline does the crash occur. A naive monitoring system checking only the health endpoint would report the server as operational. The crash is silent from the client's perspective — the curl command simply hangs until the timeout, returning nothing.

The Broader Debugging Context

Message 160 sits at a critical juncture in the debugging arc. Prior to this message, the assistant had been focused on getting the server to start — resolving OOM errors, tuning memory fractions, and waiting for model downloads. Message 160 represents the transition from "can the server start?" to "can the server actually generate text?" The answer, it turns out, is no — and the debugging that follows (spanning dozens of subsequent messages) becomes a deep investigation into attention backend compatibility, DeepGemm scale format mismatches, and RoPE parameter issues.

The assistant's next actions reveal the thinking process: checking the server health, inspecting logs, and discovering the NaN crash. The debugging trajectory that follows — switching attention backends between flashinfer, triton, and flashmla_sparse; forcing different FP8 GEMM backends; disabling CUDA graphs; consulting a local research repository (FINDINGS.md) that documents the same DeepGemm scale format issue on Blackwell — all of this is set in motion by the silent failure that message 160 uncovers.

Input Knowledge and Output Knowledge

To understand this message fully, one needs knowledge of the OpenAI chat completions API format, the SGLang serving framework, basic curl usage, and the concept of tensor parallelism for distributed model inference. One also needs awareness that deploying quantized models on new GPU architectures (Blackwell SM120) involves significant risk of numerical incompatibilities — a fact that the assistant's subsequent debugging confirms.

The output knowledge created by this message is profoundly negative but valuable: it establishes that the GLM-5-NVFP4 model, despite loading successfully and passing all initialization checks, crashes during the very first generation step when served with the current configuration. This negative result is itself a form of progress — it rules out a wide range of possible issues (network problems, download failures, OOM errors, configuration syntax) and narrows the search to the generation pipeline itself. The message also implicitly validates the earlier debugging work: the server does start, the model does load, the memory fraction is correctly tuned — the problem lies elsewhere.

Conclusion

Message 160 is a moment of apparent simplicity that masks deep complexity. A seven-line curl command, born from hours of infrastructure work, carrying the weight of unspoken assumptions about numerical stability, kernel compatibility, and hardware-software integration. Its empty response is not a failure of the command but a revelation of a deeper problem — one that will occupy the assistant for the remainder of the session. In the narrative of this deployment effort, message 160 is the hinge: the point at which the question shifts from "will it start?" to "why does it crash?"