Mapping the Terrain: Reconnaissance Before GPU Passthrough on Proxmox

The Message

ssh root@10.1.2.6 'cat /etc/kernel/cmdline 2>/dev/null || cat /proc/cmdline; echo "---"; qm list 2>/dev/null; echo "---"; pct list 2>/dev/null; echo "---"; cat /etc/default/grub 2>/dev/null | grep CMDLINE'

With the result:

root=ZFS=rpool/ROOT/pve-1 iommu=pt amd_pstate=active processor.max_cstate=1 nmi_watchdog=0 boot=zfs
---
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID       
       128 llm-one              stopped    450000          1203.50 0         
       131 ml-pipelines         running    204800            53.50 241083    
---
VMID       Status     Lock         Name                
129        stopped                 llm-two             
---
GRUB_CMDLINE_LINUX_DEFAULT="quiet"

Context and Motivation

This message sits at a pivotal inflection point in a complex ML infrastructure session. The conversation had just finished deploying the Kimi-K2.5 INT4 model into a hardened production configuration on an 8-GPU server running inside a Proxmox LXC container. The deployment included a systemd service, hierarchical KV cache with 358 GB of pinned host memory, EAGLE-3 speculative decoding with spec_v2 overlap, and proper tool-call and reasoning parsers. Everything was running smoothly.

Then the user pivoted abruptly: "Doing a tangent experiment on the proxmox node — 10.1.2.6, 10.1.16.221 guest vm; Because it's now a VM and not a container do we need to unload nvidia driver on host? Blacklist and reboot? How can we make this sort of switch reasonably painless?"

The user wanted to take the same 8 Blackwell GPUs that were serving the Kimi-K2.5 model and pass them through to a full virtual machine instead of the LXC container. This is a fundamentally different virtualization paradigm: LXC containers share the host kernel and can directly access host devices via the nvidia driver, while VMs need hardware passthrough via VFIO, which requires the host to release the device from its native driver.

The assistant's response before this message was a clarifying question — "All 8 GPUs to VM?" — which the user confirmed. The assistant then began investigating the host state with the reconnaissance command in message 5755.

Why This Message Was Written

This message is a pure reconnaissance probe. The assistant needed to understand the current state of the Proxmox host before it could formulate a plan for GPU passthrough. The reasoning is straightforward but critical: you cannot give advice about changing a system's configuration until you know what the system currently looks like.

The assistant was operating with several unknowns:

  1. Is IOMMU already enabled? GPU passthrough requires IOMMU (I/O Memory Management Unit) support in the kernel, typically enabled via the iommu=pt or iommu=on boot parameter. Without it, VFIO passthrough is impossible.
  2. What VMs and containers already exist? The host might already have VMs configured with GPU passthrough, or containers that currently own the GPUs. The assistant needed to know if there were existing configurations that would conflict with the new VM.
  3. Is the GRUB configuration set up for passthrough? Many Proxmox setups require specific kernel parameters in GRUB_CMDLINE_LINUX_DEFAULT to enable VFIO and IOMMU. The assistant needed to check whether these were already in place or would need to be added.
  4. What is the current kernel command line? The running kernel's boot parameters (from /proc/cmdline) tell the story of what's actually active, as opposed to what's configured in GRUB (which may not have been applied if the system hasn't rebooted). The message bundles four distinct probes into a single SSH command, using shell separators (echo "---") to delineate outputs. This is a deliberate design choice: rather than making four separate SSH connections (each incurring TCP handshake, SSH authentication, and latency), the assistant consolidates them into one. This is a pattern seen throughout the session — the assistant frequently batches independent commands into a single SSH invocation to minimize round-trip overhead.

The Assumptions Embedded in This Message

Every reconnaissance operation carries assumptions, and this one is no exception. The assistant assumed that the Proxmox host was accessible via SSH at 10.1.2.6 with the same key-based authentication used throughout the session. It assumed that the relevant configuration files existed at standard paths (/etc/kernel/cmdline, /etc/default/grub). It assumed that qm and pct (the Proxmox management tools for QEMU VMs and LXC containers respectively) were installed and functional.

More subtly, the assistant assumed that the kernel command line was the right place to look for IOMMU status. This is correct for Proxmox — the iommu=pt parameter is typically passed via the kernel command line, and its presence in /proc/cmdline confirms it's active. However, there's an implicit assumption that if IOMMU is enabled, the rest of the VFIO infrastructure (vfio-pci driver, IOMMU groups, PCI passthrough mappings) is also in place. As we'll see in subsequent messages, this assumption was partially correct — IOMMU was enabled, but the VFIO mappings needed further investigation.

Input Knowledge Required

To understand this message fully, one needs knowledge of:

Output Knowledge Created

The message produced several critical pieces of information:

  1. IOMMU is already enabled: The kernel command line contains iommu=pt, which means the hardware-level infrastructure for GPU passthrough is already active. This is a green light — no kernel parameter changes or reboots are needed for IOMMU.
  2. VM 128 (llm-one) exists but is stopped: This 450 GB RAM VM with 1.2 TB boot disk is configured but not running. It could potentially be repurposed for the tangent experiment, or a new VM could be created.
  3. VM 131 (ml-pipelines) is running: This 204 GB RAM VM is active and, as we learn in subsequent messages, already has all 8 GPUs passed through via the pro6000 mapping. This is a crucial discovery — it means the GPU passthrough infrastructure is already proven to work on this host.
  4. LXC container 129 (llm-two) is stopped: The container that was previously running the Kimi-K2.5 model is no longer active, which means the GPUs are not currently bound to the nvidia driver on the host. This simplifies the passthrough process considerably.
  5. GRUB_CMDLINE_LINUX_DEFAULT is minimal: The GRUB configuration only contains "quiet", meaning no special boot parameters are configured there. The IOMMU enablement must be coming from the kernel command line directly (likely via /etc/kernel/cmdline, which is the Proxmox bootloader-specific configuration).

The Thinking Process Visible in the Message

The structure of the command reveals the assistant's reasoning process. It's not just a random collection of checks — each probe answers a specific question in a logical sequence:

First probe: kernel command line. This is the most important check. Without IOMMU, GPU passthrough is a non-starter. By checking this first, the assistant establishes whether the fundamental hardware virtualization support exists. If it didn't, the entire plan would need to change (requiring a host reboot with modified kernel parameters).

Second probe: VM list. After confirming IOMMU, the assistant checks what VMs exist. This serves two purposes: it identifies potential existing passthrough configurations (VM 131's hostpci entries) and it identifies available VM IDs for a new VM (128 is stopped, 129 is a container, 131 is running).

Third probe: container list. This checks whether the LXC container that was using the GPUs is still active. If container 129 were running, the GPUs would be bound to the nvidia driver, and the assistant would need to plan for stopping the container and potentially unloading the nvidia kernel modules before VFIO could claim the devices.

Fourth probe: GRUB configuration. This cross-references the active kernel command line with the persistent GRUB configuration. The discrepancy between /proc/cmdline (which has iommu=pt and many other parameters) and /etc/default/grub (which only has "quiet") tells the assistant that the boot parameters are being set elsewhere — likely in Proxmox's custom bootloader configuration at /etc/kernel/cmdline.

The ordering is strategic: the most critical check (IOMMU) comes first, followed by progressively more specific inventory checks. If IOMMU had been missing, the assistant could have stopped there and reported the finding without needing the VM/container listings.

What This Message Does Not Say

It's worth noting what this message does not do. It does not attempt to modify any configuration. It does not unload the nvidia driver. It does not start or stop any VMs or containers. It is purely diagnostic — a "look before you leap" moment.

The assistant is following a fundamental systems engineering principle: gather data before making changes. The message is the first step in a multi-step process that will eventually involve creating or configuring a VM with GPU passthrough, installing the open NVIDIA kernel module (as the Blackwell GPUs require the nvidia-dkms-590-open package for SM120 support), and establishing a workflow for rebinding GPUs between the host driver and VFIO.

Connection to the Broader Session

This reconnaissance message connects to several threads in the larger conversation. The session had spent dozens of messages optimizing the Kimi-K2.5 deployment — fixing EAGLE-3 speculation, tuning NCCL parameters, upgrading CUDA to version 13, patching SGLang for SM120 support, and enabling hierarchical KV cache. All of that work was done inside an LXC container where the GPUs were directly accessible via the nvidia driver.

The pivot to VM passthrough represents a completely different deployment model. In an LXC container, the GPU is accessed through the host's nvidia driver, and CUDA/NCCL operations work transparently. In a VM with VFIO passthrough, the GPU is entirely owned by the guest, which runs its own nvidia driver. This gives the guest more control (it can install different driver versions, reboot independently, etc.) but adds complexity on the host side.

The assistant's discovery that VM 131 already has all 8 GPUs passed through via pro6000 mappings is significant — it means the PCI topology and IOMMU groups are already configured for this exact setup. The tangent experiment can likely reuse this existing infrastructure rather than building it from scratch.

Conclusion

Message 5755 is a textbook example of infrastructure reconnaissance done right. It's concise, targeted, and strategically ordered. The assistant asks four specific questions of the Proxmox host, each designed to eliminate a different category of unknown, and each building on the previous one. The result is a clear picture of the current state: IOMMU is enabled, a VM with full GPU passthrough already exists and is running, the LXC container is stopped, and no GRUB modifications are needed. This information sets the stage for the next steps — investigating the existing passthrough configuration and establishing a workflow for switching GPUs between the LXC container and the VM.

In the messages that follow, the assistant will discover that the proprietary NVIDIA driver in the VM fails to initialize the Blackwell GPUs, leading to the installation of the open kernel module (nvidia-dkms-590-open) and the establishment of a GPU rebinding workflow. But none of that would be possible without first understanding the terrain — which is exactly what this message accomplishes.