The Smoke Test That Closed a Chapter: SGLang Confirmed After the IOMMU Identity Domain Saga

Introduction

In the long arc of an infrastructure engineering session, most messages are about struggle—debugging errors, trying approaches that fail, iterating toward a solution. But occasionally there comes a message that is simply about confirmation. Message <msg id=6407> is one of those rare moments: a brief, almost mundane smoke test that silently declares victory after a grueling sequence of hardware-level troubleshooting. The assistant writes simply: "SGLang ready (took 3 min for full model load). Quick smoke test:" followed by a curl command and the answer "7 * 8 = 56." On its surface, this is unremarkable—a large language model correctly answering a basic arithmetic question. But to understand why this message matters, one must appreciate the storm that preceded it.

The Context: A Dead End for P2P DMA

The immediate context for this message is the culmination of a multi-hour effort to restore Peer-to-Peer (P2P) Direct Memory Access between Blackwell GPUs on a Proxmox host. The session had been exploring whether setting IOMMU identity domains for the NUMA0 GPUs could bypass the DMA remapping that was causing NCCL hangs under SEV-SNP IOMMU. The approach was elegant in theory: by setting individual IOMMU groups to "identity" mode (which bypasses DMA translation for those devices), GPU P2P operations would work without the corruption that plagued the default DMA-FQ (DMA with Fast Queue) translation mode.

The assistant had implemented a sophisticated modprobe install hook—a shell script that would set identity domains before the nvidia module loaded, ensuring the GPUs were never bound to the nvidia driver under translation mode. The hook worked perfectly at setting identity domains, as confirmed by journalctl logs. But then came the devastating discovery: Blackwell GPUs' Firmware Security Processor (FSP) fails during boot with error code 0x177 when IOMMU is in identity mode. The FSP requires specific DMA mappings set up by the kernel's DMA API in translation mode, and identity mode breaks this initialization fundamentally.

This was a terminal finding. The assistant concluded: "per-group IOMMU identity domains are fundamentally incompatible with Blackwell GPUs—the approach cannot work regardless of timing." No amount of software-level resets (FLR, SBR, CXL bus reset) could clear this state. The P2P DMA restoration path was definitively blocked.

The Recovery and the Smoke Test

After this realization, the assistant pivoted to recovery mode. The modprobe hook was temporarily disabled, the GPUs were removed from the PCI bus, a Secondary Bus Reset (SBR) was performed, and the GPUs were rescanned—this time without the identity hook, allowing the nvidia driver to bind under the default DMA-FQ mode. The GPUs came back successfully, and the SGLang service was started.

Message <msg id=6407> arrives after a 180-second wait for the model to load. The assistant runs a single curl command against the SGLang inference server, asking "What is 78?" and piping the JSON response through a Python one-liner to extract the answer. The model responds correctly: "7 8 = 56."

This is not just a functional test—it is a verification of system integrity. After the entire IOMMU identity domain experiment, the GPUs were unbound, removed, SBR'd, rescanned, and re-bound. The SGLang service was started fresh. The smoke test confirms that:

  1. The nvidia driver loaded correctly on all 4 NUMA0 GPUs
  2. CUDA is functioning (SGLang uses CUDA for model inference)
  3. The model weights loaded without corruption
  4. The inference pipeline produces coherent output
  5. The HTTP server is responsive

Assumptions and Decisions Visible in This Message

Several implicit assumptions are embedded in this seemingly simple message:

Assumption of model correctness: The assistant does not verify that "56" is the right answer—it accepts the model's output as confirmation of system health. This is reasonable for a smoke test but notable: the test is checking system functionality, not model accuracy. A wrong answer would still indicate a working system (just with a bad model), but the assistant treats a correct answer as stronger evidence.

Assumption of minimal load: The smoke test uses temperature=0.1 and a single request. This is deliberately minimal—the goal is not to benchmark performance but to confirm the pipeline works end-to-end. The assistant knows from prior benchmarks that this model achieves ~172 tok/s single-request and >2100 tok/s at high concurrency, but those numbers are irrelevant to this check.

Decision to use curl over Python: The assistant uses a shell pipeline with curl and python3 -c rather than a Python script. This is a pragmatic choice for a one-off test—it avoids creating a temporary file and keeps everything in the SSH command. The JSON parsing is minimal, extracting only the content field.

Decision to test with a simple arithmetic question: "What is 7*8?" is a classic smoke test prompt—it's unambiguous, has a single correct answer, and requires minimal reasoning. If the model were corrupted or the pipeline broken, even this simple question would fail. The assistant deliberately avoids complex prompts that might fail for reasons unrelated to system health.

Input Knowledge Required

To fully understand this message, one needs:

  1. Knowledge of the preceding IOMMU identity domain saga: Without knowing that the assistant had just spent hours trying to make P2P DMA work, and had definitively failed, the smoke test seems like a routine startup check. In context, it's a recovery verification.
  2. Understanding of SGLang architecture: SGLang is an inference engine for large language models. The fact that it takes 3 minutes to load a 122B parameter model is expected—this is a ~244GB model (BF16) spread across 4 GPUs with tensor parallelism.
  3. Knowledge of the Qwen3.5-122B model: This is the model being served. The assistant knows it's a 122-billion parameter model in BF16 format, requiring significant GPU memory and compute.
  4. Understanding of the infrastructure topology: The SSH target root@10.1.230.174 is the SGLang serving host (the LXC container), distinct from the Proxmox host at 10.1.2.6. The port 30000 is the SGLang API endpoint.
  5. Familiarity with the curl command structure: The -s flag silences progress output, -o /dev/null discards the response body (it's piped anyway), and -w "%{http_code}" would normally output the HTTP status code, but here the output is piped to Python for parsing.

Output Knowledge Created

This message creates several important pieces of knowledge:

  1. System state confirmation: The SGLang deployment is confirmed operational after the IOMMU identity domain experiment and subsequent recovery. This is the primary output—a green light for the deployment.
  2. Load time baseline: "took 3 min for full model load" establishes a baseline for how long the Qwen3.5-122B model takes to load on this hardware (4 Blackwell GPUs with TP=4). This is useful for future troubleshooting—if load time increases significantly, something is wrong.
  3. Validation of the recovery procedure: The remove-SBR-rescan cycle successfully restored the GPUs to working state. This procedure is now validated and can be reused if needed.
  4. Confirmation that MTP speculation survived: The MTP (Multi-Token Prediction) / NEXTN speculation configuration from prior sessions was not explicitly tested here, but the smoke test implicitly confirms the full SGLang stack is working, which includes the MTP configuration.

The Deeper Narrative: A Chapter Closes

What makes this message significant is what it does not say. It does not mention the failed IOMMU identity domain experiment. It does not mention the Blackwell FSP error 0x177. It does not mention that P2P DMA remains broken and NCCL_P2P_DISABLE=1 is still required. It does not mention that the MTP speculation providing 12-45% throughput improvement is now the primary optimization.

The silence is the point. The assistant has accepted the constraint—P2P DMA via IOMMU identity domains is impossible on Blackwell GPUs—and moved on to verifying that the system works within those constraints. The smoke test is the closing ceremony for that line of investigation.

The Thinking Process

The reasoning visible in this message is minimal but telling. The assistant:

  1. Waits for readiness: It polls the /v1/models endpoint in a loop (visible in <msg id=6406>) until it gets a 200 response. This shows patience and systematic verification—it doesn't assume the model loads instantly.
  2. Chooses a minimal test: Rather than running a full benchmark suite or testing multiple endpoints, it sends one simple request. This is efficient—if the system is broken, a simple test will fail just as clearly as a complex one.
  3. Parses the response programmatically: The Python one-liner extracts the answer field from the JSON response. This avoids manual inspection of the full JSON blob, which could be hundreds of lines for a chat completion response.
  4. Reports the answer directly: The output "Answer: 7 * 8 = 56" is clean and unambiguous. The assistant does not comment on whether this is correct—it lets the reader judge.

Conclusion

Message <msg id=6407> is a moment of calm after a storm. It represents the acceptance of a hardware limitation, the successful recovery of a complex system, and the confirmation that the deployment is stable. The smoke test is the simplest possible verification—a single arithmetic question—but it carries the weight of hours of debugging, multiple failed approaches, and a definitive dead end. In the narrative of this infrastructure session, this message is the turning point where the assistant stops fighting the hardware and starts working with what it has. The answer "56" is not just a correct computation; it is a signal that the system is alive, coherent, and ready for the next challenge.