The First Benchmark Attempt: A 401 Unauthorized and the Hidden Complexity of Model Serving

The Message

ssh root@10.1.230.174 "/root/ml-env/bin/python3 -m sglang.bench_serving \
  --backend sglang-oai --model glm-5 --base-url http://localhost:8000 \
  --dataset-name random --num-prompts 32 --random-input-len 256 \
  --random-output-len 128 --request-rate -1 2>&1"

The response was immediate and unambiguous:

Traceback (most recent call last):
  File "/root/ml-env/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 657, in hf_raise_for_status
    response.raise_for_status()
  File "/root/ml-env/lib/python3.12/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://huggingface.co/glm-5/resolve/main/config.json'

This message, indexed as message 643 in a long and complex coding session spanning the deployment of a GLM-5-NVFP4 model across 8 NVIDIA Blackwell GPUs, captures a seemingly trivial moment: a benchmark command that failed because of a mismatched model name. But within this small failure lies a rich story about the assumptions we make when deploying machine learning systems, the hidden complexity of tooling interfaces, and the reasoning process that leads from a working server to a broken benchmark.

Why This Message Was Written: The Urgency to Measure

To understand why the assistant wrote this message, we must trace the narrative arc that led to it. The session had been a grueling multi-hour effort to deploy the GLM-5-NVFP4 model — a large Mixture-of-Experts (MoE) language model quantized to 4-bit floating point — on a cluster of 8 RTX PRO 6000 Blackwell GPUs. The journey had been fraught with obstacles: CUDA initialization failures resolved by disabling HMM in the nvidia_uvm kernel module, missing ninja-build for FlashInfer JIT compilation, and a critical upgrade of the transformers library from 4.57.1 to 5.2.0 to support the glm_moe_dsa model architecture.

By message 640, the SGLang server was finally running. The assistant had confirmed this with visible excitement — "The server is fired up and ready to roll!" — noting that the KV cache could hold approximately 495,000 tokens and that each GPU had 5.19 GB of free memory after loading the model weights. In messages 641 and 642, the assistant verified that the server was actually generating coherent text, testing with a simple "What is 2+2?" prompt and receiving a reasoning-heavy response that demonstrated the model's thinking capabilities.

With a working server, the natural next step was benchmarking. The entire purpose of this deployment effort was to achieve high throughput — the user had set targets of 1,000+ tokens per second total throughput and 100+ tokens per second for single-stream requests. The assistant needed to measure where the system stood. Message 643 represents the very first attempt to gather those numbers, the moment when the assistant transitions from "does it work?" to "how well does it work?"

The Assumption That Broke the Benchmark

The assistant's reasoning in this message reveals a subtle but critical assumption: that the --model parameter in the SGLang benchmark tool (sglang.bench_serving) would accept the served model name — in this case, glm-5 — just as the chat completion API did. After all, the server had been launched with --served-model-name glm-5, and the curl commands in messages 640 and 641 successfully used "model": "glm-5" in their request bodies.

But the benchmark tool interprets --model differently. It uses this value not just as a label for the API request, but as a HuggingFace repository identifier to download the tokenizer and model configuration. The tool needs to know the actual tokenizer to prepare the benchmark prompts — it must encode the random input text and decode the generated output tokens. When the assistant passed glm-5, the tool dutifully tried to fetch https://huggingface.co/glm-5/resolve/main/config.json, which returned a 401 Unauthorized because no such repository exists.

This is a classic case of parameter overloading — a single flag (--model) serving two different purposes depending on the context. In the server launch command, --model refers to the HuggingFace repository (e.g., lukealonso/GLM-5-NVFP4), while --served-model-name is the API-facing name. But in the benchmark tool, --model conflates both roles: it serves as the API model name and as the tokenizer source. The assistant had internalized the distinction between these two concepts during the server launch phase but failed to recognize that the benchmark tool collapsed them back into one.

Input Knowledge and Output Knowledge

This message operates at a boundary between two knowledge states. The input knowledge required to understand it includes: familiarity with the SGLang serving stack, understanding of HuggingFace's model repository system, awareness of the distinction between served model names and HuggingFace model IDs, and knowledge of the bench_serving tool's command-line interface. The assistant had demonstrated all of this knowledge in prior messages — it knew how to launch the server, how to query it via curl, and how to inspect the benchmark tool's help output (message 642).

The output knowledge created by this message is the critical insight that --model in the benchmark tool must point to a valid HuggingFace tokenizer. This knowledge is immediately actionable: in the very next message (644), the assistant adds --tokenizer lukealonso/GLM-5-NVFP4 to the command, which resolves the issue and allows the benchmark to proceed. The error message itself was the teacher — the 401 Unauthorized revealed exactly which URL the tool was trying to access, making the fix obvious in retrospect.

Mistakes and Incorrect Assumptions

Beyond the primary assumption about --model, the message reveals a secondary mistake: the use of --request-rate -1. The assistant intended this to mean "unlimited request rate" (send all requests as fast as possible), but the benchmark tool in this version of SGLang did not support negative values for this parameter. This is evident from the subsequent messages (644-645), where the assistant first tries without --request-rate and then switches to --request-rate inf. The -1 convention is common in some benchmarking tools (notably the vLLM benchmark) to mean "no rate limiting," but SGLang's implementation expected inf or a positive number.

This double failure — both --model and --request-rate being wrong — makes the message particularly instructive. It shows how a single command can fail for multiple independent reasons, and how the first error (the 401) masks the second one (the invalid rate parameter). The assistant's debugging process in the following messages reveals this layered debugging pattern: fix the visible error first, then discover the hidden one.

The Thinking Process Visible in the Message

While the message itself contains only the command and its error output, the reasoning behind it is visible through the surrounding context. In message 642, the assistant explicitly states its intent: "The model is working and generating coherent reasoning! Now let me run a proper benchmark." The assistant then checks the benchmark tool's help output to understand the available parameters. This is a deliberate, methodical approach — verify the tool exists and understand its interface before running it.

The choice of benchmark parameters reveals additional reasoning. The assistant selects --dataset-name random with --num-prompts 32, --random-input-len 256, and --random-output-len 128. These are conservative choices: 32 prompts with 256 input tokens and 128 output tokens each. This suggests the assistant wanted a quick, lightweight benchmark to get initial numbers without overwhelming the freshly launched server. The 256/128 split is also meaningful — it tests the common case of a moderately long prompt with a short-to-medium generation, which is representative of many real-world usage patterns.

The use of --request-rate -1 (intending unlimited) indicates the assistant wanted to measure the server's maximum throughput under load, rather than its performance under a controlled rate. This is a stress-test mindset: push the system to its limits and see where it breaks.

The Broader Context: Why This Moment Matters

This message sits at a pivotal moment in the session. The assistant had just spent hours fighting with CUDA drivers, kernel modules, package versions, and JIT compilation errors to get the server running. Message 643 represents the first step into the evaluation phase — the phase where all that setup effort is finally put to the test. The failure is frustrating but minor, and it is resolved within two subsequent messages.

But the deeper significance lies in what this error reveals about the complexity of modern ML serving stacks. A single command involves at least four distinct systems: the SSH transport layer, the Python runtime, the SGLang benchmark tool, and the HuggingFace Hub API. Each of these systems has its own conventions, error handling, and failure modes. The 401 error is not a bug in any of them — it is a natural consequence of passing a non-existent repository name. The assistant's mistake was in not recognizing that the benchmark tool needed a HuggingFace identifier, not a server-side model name.

This kind of "interface mismatch" error is endemic to complex distributed systems. The server API and the benchmark tool speak different dialects of the same language: one uses model to mean "the name the server knows this model by," while the other uses it to mean "the HuggingFace repository where this model's tokenizer lives." Bridging this gap requires either documentation (which the assistant consulted via --help) or experience (which the assistant gained through this very error).

Conclusion

Message 643 is, on its surface, a trivial failure: a benchmark command that returned a 401 error. But examined closely, it reveals the intricate web of assumptions, knowledge boundaries, and interface conventions that engineers navigate when deploying machine learning systems. The assistant's reasoning — grounded in the successful curl tests and the help output — was sound but incomplete. The error served as a corrective signal, refining the assistant's understanding of the benchmark tool's parameter semantics.

The message also marks a transition point in the session: from infrastructure wrestling to performance measurement. The server was alive, the model was generating, and the only thing standing between the assistant and the throughput numbers was a single misunderstood parameter. The fix, applied in message 644, was trivial: add --tokenizer lukealonso/GLM-5-NVFP4. But the lesson — that --model in the benchmark tool is not the same as --model in the server launcher — is a lasting insight about the importance of precise parameter semantics in complex toolchains.