The Smoke Test: Verifying a 547GB Model's First Breath

The Message

ssh root@10.1.230.174 'curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"/shared/kimi-k2.5-int4\", \"messages\": [{\"role\": \"user\", \"content\": \"What is the capital of France? One sentence.\"}], \"max_tokens\": 256, \"temperature\": 0.6}" 2>/dev/null | python3 -c "import sys,json; r=json.load(sys.stdin); c=r[\"choices\"][0][\"message\"]; print(\"REASONING:\", len(c.get(\"reasoning\",\"\") or \"\"), \"chars\"); print(\"CONTENT:\", c[\"content\"]); print(\"USAGE:\", r[\"usage\"])"'

REASONING: 364 chars
CONTENT:  The capital of France is Paris.
USAGE: {'prompt_tokens': 18, 'total_tokens': 98, 'completion_tokens': 80, 'prompt_tokens_details': None}

At first glance, this looks like a trivial interaction — ask a model the capital of France, get a correct answer, move on. But in the context of an intensive multi-session machine learning deployment effort spanning days, this message represents a critical milestone: the first successful inference from a freshly loaded 547-billion-parameter model that had just finished a 36-minute weight-loading ordeal across eight NVIDIA Blackwell GPUs. This smoke test was the moment of truth where weeks of debugging, patching, rebuilding, and tuning either paid off or collapsed into yet another cryptic error message.

Why This Message Was Written: The Moment of Truth

The subject message sits at a pivotal juncture in the conversation. The user had just instructed the assistant to deploy and benchmark the native INT4 version of Kimi-K2.5 (moonshotai/Kimi-K2.5), a 1-trillion-parameter Mixture-of-Experts model using Multi-Head Latent Attention (MLA) architecture. This was the third major model deployment in rapid succession: the team had already wrestled with the NVFP4 variant of Kimi-K2.5 (which achieved ~60 tok/s but was bottlenecked by PCIe allreduce), pivoted to MiniMax-M2.5 (which hit an impressive ~4,000 tok/s at high concurrency with TP=8+EP), and was now circling back to Kimi-K2.5 in its native INT4 format.

The context immediately preceding this message (see [msg 2360]) shows the assistant confirming that all eight GPUs are loaded with 96,733 MiB each out of 97,887 MiB total — nearly 99% GPU memory utilization. The model had consumed every available byte of VRAM across the entire cluster of eight RTX PRO 6000 Blackwell GPUs. The 36-minute load time, the 547GB of safetensor shards, the careful orchestration of tensor parallelism across eight devices — all of it culminated in this single curl command.

The motivation was straightforward but high-stakes: verify that the model loaded correctly and produces coherent output before proceeding to benchmarking. The user's previous instruction ([msg 2359]) was explicit: "Run benchmarks and try to get to single stream >40~50; If not there try NCCL LL alg and other safe-ish tricks." But benchmarks are meaningless if the model is producing garbage output — a problem the team had encountered before with the GLM-5 NVFP4 deployment, where incoherent output led to a deep debugging session involving Triton MLA attention backend bugs and GGUF dequantization shard ordering issues (see [msg 2360] context in segment 16).## The Reasoning Behind the Smoke Test Design

The choice of prompt — "What is the capital of France? One sentence." — is deliberately simple and diagnostic. It's not a coincidence that this is one of the most canonical test prompts in the LLM deployment playbook. The question tests several things simultaneously:

  1. Basic factual recall: The model must retrieve a well-known fact from its weights. If the model produces anything other than "Paris" (or a reasonable variant), something is fundamentally wrong with the weight loading, quantization, or architecture support.
  2. Instruction following: The "One sentence." constraint tests whether the model respects output formatting instructions. The NVFP4 variant had previously demonstrated coherent output, so this serves as a regression check.
  3. Reasoning extraction: The assistant's post-processing pipeline explicitly prints the length of the model's "reasoning" field (364 characters). Kimi-K2.5 is a reasoning model — it generates internal chain-of-thought before producing its final answer. The presence of 364 characters of reasoning confirms that the model's reasoning capabilities are intact, not just its final output.
  4. Token accounting: The usage statistics (18 prompt tokens, 80 completion tokens, 98 total) provide a quick sanity check. 18 prompt tokens for "What is the capital of France? One sentence." is reasonable for the model's tokenizer, and 80 completion tokens for a short answer with reasoning is within expected bounds. The temperature setting of 0.6 is also deliberate — high enough to allow some variation but low enough to produce deterministic, reproducible results. This is a common choice for smoke tests where you want to verify coherence without introducing sampling noise.

What This Message Reveals About the Deployment State

The successful response — "The capital of France is Paris." — tells us several things about the state of the system:

The model loaded correctly. After the 36-minute weight-loading process, all 64 safetensor shards were properly read, deserialized, and distributed across eight GPUs using tensor parallelism. The INT4 quantization scheme (compressed-tensors, group_size=32, symmetric) was applied correctly, with only the MoE routed expert weights quantized while attention layers and shared experts remained in full precision.

The MLA architecture works on Blackwell GPUs. The NVFP4 deployment had required a custom Triton MLA sparse attention backend for SM120 (Blackwell's compute architecture). The INT4 variant uses the same MLA architecture, and this smoke test confirms that the attention mechanism is functioning correctly — a non-trivial achievement given the earlier debugging saga.

The vLLM integration is clean. The model uses a custom architecture ("KimiK25ForConditionalGeneration") with custom model code loaded via --trust-remote-code. The fact that it serves correctly through the OpenAI-compatible API endpoint means the vLLM server's model registry, weight loading, and forward pass are all properly integrated.

No leftover patches are interfering. Earlier in the session, the assistant had performed a clean vLLM reinstall to remove stale GLM-5 debug patches. This smoke test confirms that the reinstall was successful and no residual artifacts are corrupting inference.

The Assumptions Embedded in This Message

Every smoke test carries assumptions, and this one is no exception:

The model is assumed to be coherent from a single prompt. A single question about France's capital is a necessary but not sufficient test for model quality. The model could produce correct output for this trivial query while still having subtle issues — for example, degraded performance on complex reasoning tasks due to quantization artifacts, or incorrect behavior on longer sequences. The assistant implicitly assumes that if this basic test passes, the model is worth benchmarking further.

The network and API infrastructure are assumed reliable. The curl command traverses the full stack: HTTP client → vLLM API server → model inference → response serialization. A failure at any layer would produce an error. The successful 200 response validates the entire serving pipeline.

The reasoning field is assumed to be meaningful. The assistant prints the reasoning length (364 chars) but doesn't inspect its content. This assumes that if reasoning is present and non-trivial in length, it's likely coherent. In practice, this is a reasonable heuristic — a model producing garbage output would typically show either no reasoning or obviously repetitive/confused reasoning.

Mistakes and Incorrect Assumptions

One subtle issue with this smoke test is that it doesn't verify multi-turn conversation or tool calling — both features that the user explicitly enabled via --tool-call-parser kimi_k2 and --enable-auto-tool-choice. The model could pass this single-turn test but fail on tool-calling scenarios, which are critical for the intended use case (the user is building toward a coding agent deployment).

Additionally, the test uses a very short prompt (18 tokens). The model was configured with --max-model-len 131072, and long-context behavior is a completely different regime. Quantization artifacts that are invisible at short context lengths can become catastrophic as the sequence grows, particularly for INT4 quantization with group_size=32. The smoke test doesn't probe this at all.

The temperature of 0.6 also masks potential determinism issues. If the model had a subtle bug that only manifests with greedy sampling (temperature=0), this test wouldn't catch it. The NVFP4 deployment had previously suffered from "incoherent output" that was only visible after multiple test prompts — a single successful response can create false confidence.

Input Knowledge Required

To fully understand this message, one needs:

Knowledge of the deployment history: The 36-minute load time, the 547GB model size, the eight-GPU tensor parallelism configuration, and the earlier debugging of GLM-5 and NVFP4 variants. Without this context, the message looks like a trivial API call.

Understanding of MLA architecture: Kimi-K2.5 uses Multi-Head Latent Attention (MLA), which requires specialized Triton kernels on Blackwell GPUs. The fact that this works at all is a technical achievement after the earlier custom backend implementation.

Familiarity with vLLM's serving architecture: The curl command targets the OpenAI-compatible API endpoint, meaning the model is served through vLLM's API server with continuous batching, PagedAttention, and tensor parallelism.

Knowledge of INT4 quantization schemes: The model uses compressed-tensors INT4 with group_size=32, only quantizing MoE expert weights. This selective quantization is visible in the model's config.json and affects both memory usage and inference quality.

Output Knowledge Created

This message creates several pieces of actionable knowledge:

Confirmation of successful deployment: The team now knows that Kimi-K2.5 INT4 loads and serves correctly on their hardware. This justifies the 36-minute load time and the 547GB of disk space consumed.

Baseline for benchmarking: The 80-token completion for a simple query provides a reference point. When benchmarking begins, any significant deviation in tokens-per-second or output quality can be attributed to the benchmark conditions rather than fundamental model issues.

Validation of the deployment pipeline: The entire workflow — model download, vLLM configuration, API server launch, weight loading, and inference — is validated end-to-end. This pipeline can be reused for future model deployments.

A green light for optimization: With coherence confirmed, the assistant can proceed to the user's next instruction: benchmarking and attempting to reach >40-50 tok/s single-stream throughput, potentially with NCCL LL algorithm tuning and other optimizations.

The Broader Significance

In the grand narrative of this coding session, message [msg 2361] is the quiet moment before the storm. It's the deep breath after a 36-minute weight-loading marathon, the check engine light that stays off, the first heartbeat of a 547-billion-neuron brain waking up across eight GPUs. The assistant doesn't celebrate — there's no "Success!" or "It works!" — just a clean, structured output dump followed by the inevitable next step. The benchmark results that follow this message will reveal whether this massive model can meet the user's performance targets, or whether yet another round of NCCL tuning and optimization tricks will be needed. But for this one message, everything works, and that is enough.