The Kernel Cmdline That Unlocked Performance: A Deep Dive Into Boot-Time Tuning for ML Inference

Introduction

In the high-stakes world of large language model inference, every millisecond counts. When the assistant in this opencode session was tasked with squeezing maximum throughput out of a GLM-5-NVFP4 model running on eight RTX PRO 6000 Blackwell GPUs, the investigation led to an unexpected conclusion: the bottleneck wasn't in the model architecture, the GPU kernels, or the communication fabric. It was hiding in the Linux kernel's default configuration — in CPU frequency scaling governors, idle states, and interrupt handling policies that were silently throttling performance.

Message <msg id=1303> captures a pivotal moment in this optimization journey. After a comprehensive system audit revealed critical misconfigurations, the assistant executes the final step before a system reboot: updating the kernel command-line parameters to enable amd_pstate=active, processor.max_cstate=1, and nmi_watchdog=0. This single message, seemingly a routine administrative task, represents the convergence of hours of diagnostic work, the application of deep kernel knowledge, and a calculated bet that boot-time parameters would unlock the performance that runtime tuning alone could not achieve.

The Message: What Was Actually Said

The message reads:

NVIDIA driver 590.48.01 built and installed for the 6.14 kernel. Now let me set up all the persistent configuration:

>

`` echo "=== Step 2: Update kernel cmdline ===" # Current: root=ZFS=rpool/ROOT/pve-1 amd_iommu=on iommu=pt boot=zfs vmlinuz # New: add amd_pstate=active processor.max_cstate=1 nmi_watchdog=0 cat /etc/kernel/cmdline echo "" echo "Writing new cmdline..." echo "root=ZFS=rpool/ROOT/pve-1 iommu=pt amd_pstate=active processor.max_cstate=1 nmi_watchdog=0 boot=zfs" > /etc/kernel/cmdline cat /etc/kernel/cmdline echo "" echo "Refreshing boot entries..." proxmox-boot-tool refresh 2>&1 echo "Done" ``

>

=== Step 2: Update kernel cmdline === root=ZFS=rpool/ROOT/pve-1 amd_iommu=on iommu=pt boot=zfs vmlinuz

>

Writing new cmdline... root=ZFS=rpool/ROOT/pve-1 iommu=pt amd_pstate=active processor.max_cstate=1 nmi_watchdog=0 boot=zfs

>

Refreshing boot entries... Running hook script 'proxmox-auto-removal'.. Running hook script 'zz-proxmox-boot'.. Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace.. Copying and configuring kernels on /dev/disk/by-uuid/5360-A706 Copying k...

The message is deceptively brief. It opens with a status update confirming that the NVIDIA driver (version 590.48.01) has been successfully built and installed for the new 6.14.11 kernel — a critical prerequisite that was anything but straightforward. It then proceeds to the main task: modifying the kernel command line to inject three performance-critical parameters, and refreshing the Proxmox boot configuration so these parameters take effect on the next reboot.

The Road to This Message: Context and Motivation

To understand why <msg id=1303> exists, one must trace the investigation that preceded it. The session had been engaged in a prolonged battle to improve inference throughput for the GLM-5-NVFP4 model. Initial benchmarks showed a staggering gap between theoretical maximum performance and actual results: the model should have been capable of approximately 309 tokens per second in single-stream mode, but the system was delivering only about 10.36 tok/s — a mere 3.4% efficiency ([chunk 10.0]).

This prompted a comprehensive parallel system audit using ten diagnostic agents. The audit uncovered a litany of misconfigurations: the CPU frequency scaling driver was using acpi-cpufreq instead of the superior amd_pstate driver; the kernel was an outdated 6.8.12; NUMA balancing was enabled, causing unnecessary memory migrations; the CPU was allowed to enter deep C-states (power-saving idle states) that introduced wake-up latency; and the PCIe MaxReadReq was stuck at 512 bytes instead of the optimal 4096.

The assistant applied runtime fixes for many of these issues — disabling NUMA balancing, adjusting sysctl parameters, and modifying PCIe settings — and re-ran benchmarks. The results were underwhelming. As <msg id=1293> shows, the tuned system still delivered only 10.30 tok/s at concurrency 1. The runtime fixes had barely moved the needle.

This led to a critical realization: some of the most impactful tuning parameters — particularly the CPU frequency scaling driver (amd_pstate) and the maximum C-state limit — cannot be changed at runtime. They must be set at boot time via kernel command-line parameters. The user's instruction in <msg id=1294> — "Try new kernel and the reboot-requiring fixes, feel free to reboot that proxmox node too" — gave the assistant the green light to pursue the nuclear option: a kernel upgrade combined with persistent boot-time configuration.

The Reasoning Behind Each Parameter

The three parameters added to the kernel command line each target a specific performance bottleneck identified during the audit.

amd_pstate=active is perhaps the most consequential. The AMD CPU in the Proxmox host supports the amd_pstate frequency scaling driver, which provides hardware-managed P-states (performance states) with finer granularity and lower latency than the generic acpi-cpufreq driver. By default, many Linux distributions boot with acpi-cpufreq because it is the safest, most compatible option. But for compute-bound ML workloads, amd_pstate=active allows the CPU to ramp to maximum frequency more aggressively and respond to load changes more quickly. This directly impacts GPU kernel launch latency — the CPU must dispatch work to the GPU, and if the CPU is slow to wake up or ramp up, every CUDA kernel launch incurs a delay.

processor.max_cstate=1 restricts the CPU to C-state 1 (halt state) and prevents deeper C-states (C2, C3, C6, etc.). Deep C-states save power by powering down portions of the CPU core, but they come at a cost: waking from C6 can take tens of microseconds. In an ML inference server where GPU kernels are launched in rapid succession, each kernel launch requires the CPU to be responsive. If the CPU has fallen into a deep C-state between launches, the wake-up latency adds directly to the time-to-first-token and inter-token latency. Setting max_cstate=1 trades a modest power increase for deterministic, low-latency CPU responsiveness.

nmi_watchdog=0 disables the NMI (Non-Maskable Interrupt) watchdog, a kernel feature that detects hung CPUs by periodically firing NMIs. On systems with many GPUs and high PCIe traffic, the NMI watchdog can introduce spurious interrupts that interfere with timing-sensitive operations. Disabling it is a standard recommendation for latency-sensitive workloads.

Notably, the assistant also removed amd_iommu=on from the cmdline. The original configuration included both amd_iommu=on and iommu=pt. The iommu=pt (pass-through) mode is already the more performant option — it allows devices that are assigned to a single guest to bypass IOMMU translation. Keeping both amd_iommu=on and iommu=pt is redundant but harmless. The assistant's removal of amd_iommu=on while keeping iommu=pt suggests a deliberate choice to simplify the cmdline, possibly based on the understanding that iommu=pt implicitly enables IOMMU in pass-through mode. This is a minor but telling detail — it shows the assistant was not blindly appending parameters but actively curating the boot configuration.

Assumptions and Decisions

The assistant made several assumptions in this message, most of which were reasonable but worth examining.

Assumption 1: The kernel upgrade would not break CUDA. The assistant had already verified that the NVIDIA DKMS module built successfully for the 6.14.11 kernel ([msg 1302]). However, building successfully does not guarantee runtime stability. The assistant assumed that the signed kernel module would load correctly and that the CUDA runtime would function identically. As later messages would reveal, this assumption was partially incorrect — a post-reboot CUDA failure occurred due to stale NVIDIA device major numbers in the LXC cgroup configuration ([chunk 10.0]). This was not a kernel or driver issue per se, but a container configuration issue triggered by the new kernel's device numbering.

Assumption 2: amd_pstate=active would be supported by this CPU. The assistant did not explicitly verify that the AMD EPYC processor in the Proxmox host supports the amd_pstate driver. While modern AMD EPYC CPUs (Genoa, Bergamo, Turin) do support it, older generations may not. The assistant's confidence was likely based on the system audit results, which would have shown the CPU model. This assumption proved correct, as the system booted successfully with the new parameter.

Assumption 3: The proxmox-boot-tool refresh command would correctly propagate the new cmdline to all boot entries. Proxmox uses a complex boot system that can involve multiple ESP (EFI System Partition) partitions and fallback entries. The assistant assumed that a single refresh command would update all relevant boot configurations. The output confirms it copied kernels to /dev/disk/by-uuid/5360-A706, suggesting a single ESP, which is typical.

Assumption 4: The benefits of these parameters would outweigh the costs. Disabling deep C-states and the NMI watchdog increases power consumption. Enabling amd_pstate=active may cause the CPU to run at higher frequencies more consistently, also increasing power draw. The assistant implicitly assumed that the performance gains from reduced latency would justify the power cost — a reasonable trade-off for an ML inference server where throughput is the primary metric.

The Thinking Process Visible in the Message

The message reveals a structured, methodical thinking process. The assistant begins with a status confirmation ("NVIDIA driver 590.48.01 built and installed for the 6.14 kernel"), which serves both as a progress update and as a verification that the prerequisite work is complete. It then announces the next step ("Now let me set up all the persistent configuration"), framing the cmdline update as part of a larger configuration effort.

The inline comments in the bash script are particularly revealing:

# Current: root=ZFS=rpool/ROOT/pve-1 amd_iommu=on iommu=pt boot=zfs vmlinuz
# New: add amd_pstate=active processor.max_cstate=1 nmi_watchdog=0

The assistant explicitly documents the before-and-after state, showing that it understands the diff between the old and new configurations. This is not a blind overwrite — it is a deliberate transformation where the assistant preserves the essential boot parameters (root filesystem, IOMMU mode, boot method) while injecting the new tuning parameters.

The three-step structure of the script — read current, write new, verify — demonstrates a "verify before and after" pattern typical of careful system administration. The assistant reads the current cmdline, writes the new one, reads it back to confirm, and then refreshes the boot entries. This pattern minimizes the risk of a typo or truncation rendering the system unbootable.

The use of proxmox-boot-tool refresh shows an understanding of Proxmox's boot architecture. On standard Debian systems, updating /etc/kernel/cmdline is sufficient; on Proxmox, the boot entries must be explicitly refreshed to copy the new cmdline to the ESP. The assistant correctly identifies this requirement.

Input Knowledge Required

To fully understand this message, the reader needs knowledge in several domains:

Linux kernel boot parameters: Understanding what amd_pstate, max_cstate, and nmi_watchdog do, and why they matter for performance. This is specialized knowledge that goes beyond general Linux administration.

Proxmox boot infrastructure: Proxmox Virtual Environment uses a unique boot system based on proxmox-boot-tool and ESP partitions. A standard Linux user might not know that /etc/kernel/cmdline is not directly read by the bootloader but is processed by proxmox-boot-tool to generate boot entries.

NVIDIA DKMS and kernel compatibility: The message opens by confirming the NVIDIA driver was built for the new kernel. This requires understanding that NVIDIA's proprietary driver is distributed as a DKMS (Dynamic Kernel Module Support) package that must be recompiled for each kernel version. The assistant had to install kernel headers and manually trigger the DKMS build ([msg 1301], [msg 1302]) because the automatic build was skipped.

CPU frequency scaling drivers: The distinction between acpi-cpufreq and amd_pstate is not widely known. The assistant's decision to switch to amd_pstate=active reflects a deep understanding of AMD CPU architecture and Linux power management.

C-states and their performance impact: The concept of CPU C-states (idle states) and their wake-up latency is critical for understanding why processor.max_cstate=1 matters for inference workloads. Many system administrators leave C-states at their default configuration, unaware of the latency cost.

Output Knowledge Created

This message produces several concrete outputs:

  1. A modified kernel cmdline at /etc/kernel/cmdline containing the new boot parameters. This is a persistent configuration that will survive reboots.
  2. Updated Proxmox boot entries on the ESP partition, ensuring the new kernel and cmdline are used on next boot.
  3. A verified state where the old cmdline was read, the new cmdline was written, and the boot entries were refreshed. The assistant captured the output of each step, creating an audit trail.
  4. A dependency satisfied: The NVIDIA DKMS module for kernel 6.14.11 was confirmed built and installed, which was a prerequisite for the reboot to succeed with GPU functionality.
  5. Documentation of the change: The inline comments in the bash script serve as documentation of what was changed and why. This is valuable for future debugging or for another administrator who might encounter the system.

The Broader Significance

Message <msg id=1303> is a textbook example of the "last mile" problem in systems optimization. The assistant had already applied every runtime tuning technique available — sysctl parameters, PCIe configuration, NUMA balancing, CPU governor changes — and seen minimal improvement. The breakthrough required recognizing that the most impactful knobs were locked behind a reboot.

This pattern is common in high-performance computing: the default kernel configuration is designed for general-purpose workloads and power efficiency, not for maximum throughput. The parameters that truly transform performance — frequency scaling drivers, idle state limits, interrupt handling policies — are baked into the kernel command line and cannot be changed without restarting the system.

The message also illustrates the importance of understanding the full boot chain. On a Proxmox system, the kernel cmdline is not simply read by GRUB; it is processed by proxmox-boot-tool, which generates boot entries on the ESP. A naive administrator might update /etc/kernel/cmdline and reboot, only to find the changes had no effect because the boot entries were not refreshed. The assistant's use of proxmox-boot-tool refresh demonstrates platform-specific knowledge that was essential for the change to take effect.

Conclusion

Message <msg id=1303> is a pivotal moment in a larger optimization saga. It represents the transition from runtime tuning — which had proven insufficient — to persistent, boot-time configuration that would fundamentally change the system's performance characteristics. The assistant's methodical approach — verifying prerequisites, documenting changes, understanding the platform's boot architecture, and curating rather than blindly appending parameters — reflects a deep systems engineering discipline.

The message also serves as a reminder that in complex systems, the most impactful optimizations are often the ones that require a reboot. The parameters changed here — amd_pstate=active, processor.max_cstate=1, nmi_watchdog=0 — are invisible to most monitoring tools and cannot be adjusted on a running system. They represent the kind of foundational tuning that separates a well-optimized ML inference server from one that is merely functional.

In the subsequent messages, the reboot would reveal new challenges — CUDA initialization failures, device numbering issues — but the foundation laid in <msg id=1303> was correct. The kernel upgrade and boot parameters were necessary steps that, once combined with fixes for the post-reboot issues, would eventually lead to significant throughput improvements. The message stands as a testament to the principle that sometimes, to go fast, you must first stop — and reboot.