A Quiet Verification: Reading the SGLang Service Logs After the IOMMU Identity Domain Reversal

Introduction

In the midst of a sprawling, multi-session odyssey to deploy large language models across eight NVIDIA Blackwell GPUs on an Ubuntu 24.04 host, the assistant issues what appears to be the most mundane of commands: a simple journalctl query to peek at the SGLang service logs. Message [msg 6340] consists of exactly one bash command and three lines of output, yet this brief verification step sits at a critical inflection point in the conversation. It is the moment after a major architectural reversal — the abandonment of the IOMMU identity domain approach for GPU P2P DMA — and the beginning of a return to operational stability. The message's true significance lies not in what it says, but in what it confirms: the SGLang inference service has restarted cleanly, the GPUs are accessible, and the system is ready to move forward with the optimizations that actually work.

The Message in Full

The subject message reads:

[assistant] [bash] ssh root@10.1.230.174 'journalctl -u sglang-qwen -n 5 --no-pager 2>&1'
Mar 09 15:47:43 llm-two sglang-qwen[14363]:   warnings.warn(
Mar 09 15:47:44 llm-two sglang-qwen[14363]: [2026-03-09 15:47:44] WARNING server_args.py:2020: Disabling overlap schedule since mamba no_buffer is not compatible with overlap schedule, try to use --disable-radix-cache if overlap schedule is necessary
Mar 09 15:47:44 llm-two sglang-qwen[14363]: [2026-03-09 15:47:44] server_args=ServerArgs(model_path='/shared/models/Qwen3.5-122B-A10B', tokenizer_path='/shared/models/Qwen3.5-122B-A10B', t...

The command is straightforward: SSH into the container host at 10.1.230.174 and fetch the last five lines of the systemd journal for the sglang-qwen service. The output reveals two things: a warning about an incompatibility between the overlap schedule and the Mamba no-buffer configuration, and a truncated dump of the ServerArgs object showing the model and tokenizer paths.

Context: Why This Message Was Written

To understand why the assistant ran this particular command at this particular moment, we must trace the events of the preceding messages. The conversation had been deep in a rabbit hole of IOMMU configuration, attempting to restore GPU P2P DMA capability under an AMD IOMMU that was running in full translation mode (DMA-FQ) to support an SEV-SNP virtual machine. The assistant had tried multiple approaches: runtime switching of IOMMU group types via the sysfs interface ([msg 6323]), Secondary Bus Reset (SBR) cycles to re-enumerate GPUs ([msg 6324]), kernel boot parameter analysis ([msg 6325]), and even an investigation of the nvidia driver's own DmaRemapPeerMmio parameter ([msg 6326], [msg 6327]).

The P2P testing in [msg 6332] and [msg 6333] revealed a frustrating partial result: DmaRemapPeerMmio=1 was already enabled and working for some GPU peer pairs but not others, producing a directional IOMMU mapping pattern that suggested a driver bug or incomplete implementation. The IO_PAGE_FAULTs logged in dmesg confirmed that the nvidia driver was not creating complete IOMMU mappings for all peer BAR ranges.

Faced with this dead end, the assistant pivoted in [msg 6334] to a pragmatic approach: create a boot-time systemd service that would set IOMMU identity domains for the NUMA0 GPUs before the nvidia driver loaded. This was the last hope for P2P DMA — if the IOMMU groups could be switched to identity mode at boot, before the nvidia driver probed the devices, the Blackwell GPUs would never encounter the translation-mode IOMMU that was causing the faults. The service was created in [msg 6335] and [msg 6336], and the old udev rules were cleaned up in [msg 6337].

Then, in [msg 6339], the assistant restarted the SGLang service with the current working configuration (P2P disabled via NCCL_P2P_DISABLE=1, relying on SHM transport for inter-GPU communication). The service started successfully and reported as active.

Message [msg 6340] is the immediate follow-up to that restart. The assistant is performing a post-start health check, verifying that the service not only started (which was confirmed in [msg 6339]) but is running without critical errors. This is standard operational practice: after any service restart, check the logs to catch warnings or errors that might indicate a misconfiguration or runtime problem.

The Warning: Overlap Schedule and Mamba No-Buffer Incompatibility

The most substantive piece of information in the message is the warning from server_args.py:2020:

Disabling overlap schedule since mamba no_buffer is not compatible with overlap schedule, try to use --disable-radix-cache if overlap schedule is necessary

This warning reveals an important detail about the SGLang configuration. The "overlap schedule" is a SGLang optimization that overlaps GPU computation with communication during inference, effectively pipelining the forward pass to hide latency. It is particularly beneficial for tensor-parallel deployments where model layers are split across multiple GPUs — exactly the scenario here with Qwen3.5-122B-A10B running with TP=4 across four Blackwell GPUs.

The "mamba no_buffer" refers to the Mamba state space model backend operating in no-buffer mode. Mamba is a class of sequence models that uses state spaces instead of attention, and the no-buffer variant avoids intermediate buffering for potentially better memory efficiency. However, this mode is incompatible with the overlap scheduler, forcing SGLang to disable the overlap optimization.

The warning also suggests an alternative: if overlap scheduling is critical, one can use --disable-radix-cache instead. The radix cache is SGLang's prefix caching mechanism, which reuses KV cache entries across requests with common prefixes. Disabling it would free up the overlap scheduler but at the cost of losing prefix caching benefits.

This warning is not necessarily a problem — SGLang gracefully degrades by disabling the incompatible feature. But it is a signal that the current configuration is not optimal. The assistant would need to decide whether to adjust the server arguments to enable overlap scheduling (by either changing the Mamba backend or disabling the radix cache) or accept the performance trade-off.

Assumptions and Knowledge Required

To fully interpret this message, one needs substantial domain knowledge. First, an understanding of the SGLang inference engine and its server architecture: the ServerArgs object encapsulates all configuration parameters for the inference service, including model path, tokenizer path, tensor parallelism, and various optimization flags. The truncated output shows only the model and tokenizer paths, but the full object would contain dozens of parameters.

Second, familiarity with the Mamba architecture and its interaction with SGLang's scheduling system. The warning specifically mentions "mamba no_buffer," which implies the model is being served with a Mamba-compatible backend (Qwen3.5-122B-A10B is a hybrid architecture combining attention layers with Mamba-style state space layers, hence the "A10B" designation — 10 billion active parameters from a 122B total).

Third, knowledge of the system topology: the service is running inside an LXC container on the host 10.1.230.174, which has access to four NUMA0 Blackwell GPUs via nvidia driver pass-through. The other four NUMA1 GPUs are bound to vfio-pci for the SEV-SNP VM.

The assistant assumes that the service restart was successful (confirmed by [msg 6339]) and that the logs will reflect a clean startup. This assumption is validated — there are no crash traces or fatal errors in the output, only a configuration warning.

Output Knowledge Created

This message produces several pieces of actionable knowledge:

  1. Service health confirmed: The SGLang service is running and serving the Qwen3.5-122B-A10B model from /shared/models/. The startup completed without fatal errors.
  2. Configuration incompatibility identified: The overlap schedule is disabled due to the mamba no-buffer backend. This is a performance-relevant finding — the assistant now knows that the current configuration is not using the overlap optimization, which could be leaving throughput on the table.
  3. Alternative path documented: The warning itself provides the remediation option (--disable-radix-cache), giving the assistant a clear next step if overlap scheduling becomes important for performance tuning.
  4. Baseline established: With the service running in this configuration, any subsequent performance benchmarks will be measured against this baseline. If the assistant later enables overlap scheduling, the before/after comparison will be meaningful.

The Thinking Process Visible in the Message

While the message itself contains no explicit reasoning (it is a pure tool call with output), the reasoning is embedded in the timing and selection of this command. The assistant is operating in a structured pattern: after any configuration change or service restart, verify the logs. This is visible throughout the conversation — every service start is followed by a journalctl or systemctl is-active check.

The choice of -n 5 (last five lines) is deliberate: the assistant wants to see the tail end of the startup sequence, which typically contains the most important messages (the final initialization status, any warnings, and the server args dump). The --no-pager flag ensures the output is captured cleanly without interactive paging.

The fact that the assistant does not immediately act on the warning is also informative. At this point in the conversation, the priority is stability — the system has just been through an extensive IOMMU experiment that ended in reversal. The assistant is taking a "first, do no harm" approach: get the service running, verify it, and then consider optimizations. The warning about overlap scheduling is noted but not acted upon because the immediate goal is to confirm that the IOMMU identity domain reversal did not break the inference service.

The Broader Significance

Message [msg 6340] marks the transition from the IOMMU exploration phase to the operational stabilization phase. The IOMMU identity domain approach had been definitively blocked by the Blackwell FSP's requirement for DMA translation mode during initialization — a discovery that would be confirmed after the next reboot in the chunk summary. The assistant had pivoted to creating a boot-time service, but even that would ultimately fail because the Blackwell firmware cannot initialize under identity IOMMU regardless of timing.

In this light, the SGLang service restart and log check represent a return to the working configuration: P2P DMA disabled, SHM transport for NCCL, and the model serving on four GPUs with TP=4. The MTP (Multi-Token Prediction) speculation optimization, enabled in a previous session, would continue to provide 12-45% throughput improvement at low concurrency — a significant win that survived the reboot.

The warning about overlap scheduling, while minor in isolation, foreshadows the ongoing performance tuning work. The assistant would later benchmark the service, explore MTP speculation configurations, and eventually achieve over 2100 tok/s at high concurrency. But for now, the goal is simply to confirm that the system is stable after the IOMMU experiment. Message [msg 6340] accomplishes that goal with quiet efficiency.