Locking in Performance: The Systemd Service That Makes GPU Tuning Survive Reboot

In the high-stakes world of large language model inference optimization, system tuning is never truly "done" — it must be made to survive the next reboot. Message [msg 1305] captures this exact moment: the assistant, having spent hours upgrading kernels, tweaking sysctls, and benchmarking the GLM-5-NVFP4 model across eight RTX PRO 6000 Blackwell GPUs, now faces the mundane but critical task of making one particular tuning parameter persist across driver reloads. The result is a systemd service that sets PCIe MaxReadReq to 4096 bytes on every GPU at boot — a small piece of infrastructure that embodies the entire philosophy of production-grade performance engineering.

The Context: A System at the Edge of Performance

To understand why this message exists, one must trace the narrative arc of the preceding hours. The session had been a marathon of ML infrastructure optimization. The team had already computed theoretical maximum single-stream performance for the GLM-5-NVFP4 model at 309 tok/s, only to measure a disappointing 10.36 tok/s — a staggering 3.4% efficiency that screamed "something is fundamentally wrong." A comprehensive parallel audit via ten agents had uncovered a laundry list of system misconfigurations: a suboptimal CPU governor (acpi-cpufreq instead of amd_pstate), an outdated kernel (6.8.12), enabled NUMA balancing, deep CPU C-states, and a PCIe MaxReadReq stuck at 512 bytes instead of the optimal 4096.

The user had given the green light in [msg 1294]: "Try new kernel and the reboot-requiring fixes, feel free to reboot that proxmox node too." This launched a cascade of infrastructure work. The assistant installed the 6.14.11 Proxmox kernel, rebuilt the NVIDIA 590.48.01 DKMS driver for the new kernel, updated the kernel command line with amd_pstate=active and processor.max_cstate=1, and wrote persistent sysctl configuration to /etc/sysctl.d/99-gpu-compute.conf. Each of these changes was individually necessary, but none would survive a reboot without deliberate effort — and the MaxReadReq fix was the most fragile of all.

The MaxReadReq Problem: Why Runtime Fixes Are Not Enough

The PCIe MaxReadReq (Maximum Read Request Size) parameter controls how many bytes a PCIe device can request in a single read transaction. For GPU workloads, a larger MaxReadReq (4096 bytes vs. the default 512) allows the GPU to fetch data from host memory in larger chunks, reducing the number of PCIe transactions and improving bandwidth utilization. The assistant had previously verified that setting MaxReadReq to 4096 improved H2D bandwidth, but there was a catch: this setting resets to the hardware default (512 bytes) every time the NVIDIA driver reloads.

This is a well-known frustration in GPU compute environments. The PCIe configuration space is writable by software, but the NVIDIA driver reinitializes the device during load, overwriting any previous setpci modifications. A runtime fix applied via a script is ephemeral — it vanishes on the next reboot or driver reload. The only way to make it stick is to apply it after the driver has loaded, every single time.

Anatomy of the Systemd Service

The assistant's solution in [msg 1305] is a systemd oneshot service with careful dependency ordering:

[Unit]
Description=Set GPU PCIe MaxReadReq to 4096
After=nvidia-persistenced.service
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c "for bdf in 01:00.0 11:00.0 61:00.0 71:00.0 81:00.0 91:00.0 e1:00.0 f1:00.0; do setpci -s \$bdf CAP_EXP+0x08.w=5937; done; for rp in 00:01.1 10:01.1 60:01.1 70:01.1 80:01.1 90:01.1 e0:01.1 f0:01.1; do setpci -s \$rp CAP_EXP+0x08.w=5937; done; echo MaxReadReq set to 4096 on all GPUs"

[Install]
WantedBy=multi-user.target

The design reveals several deliberate decisions:

Dependency ordering: The service declares After=nvidia-persistenced.service and After=multi-user.target. The nvidia-persistenced service initializes the NVIDIA driver and GPU state. By waiting for it, the assistant ensures the PCIe configuration space is fully accessible and that the driver has finished its own initialization (which would otherwise overwrite the MaxReadReq). The multi-user.target dependency provides an additional safety net, ensuring the system is fully booted before the tuning runs.

Oneshot with RemainAfterExit: The Type=oneshot means systemd runs the ExecStart command once and considers the unit "started." RemainAfterExit=yes tells systemd to consider the unit as still active even after the ExecStart process exits — this prevents systemd from thinking the service has failed and attempting to restart it.

The setpci commands: The setpci utility reads and writes PCI configuration space. The argument CAP_EXP+0x08.w=5937 targets the PCI Express Capabilities register at offset 0x08 from the base of the PCI Express capability structure. The value 5937 is a bitwise encoding: in the Device Control register (which is what CAP_EXP+0x08 points to), bits [7:5] control MaxReadReq. The value 0x37 (binary 00110111) sets bits 5 and 6 (MaxReadReq = 4096) while preserving other settings.

Eight GPUs, eight root ports: The service iterates over two sets of BDF (Bus:Device.Function) addresses: eight GPU endpoints (01:00.0, 11:00.0, etc.) and eight corresponding root ports (00:01.1, 10:01.1, etc.). The root ports are the upstream PCIe ports connecting each GPU to the CPU's PCIe root complex. Setting MaxReadReq on both ends ensures the entire data path is configured consistently — though in practice, the GPU endpoint setting is the critical one for GPU-initiated reads.

Assumptions and Potential Pitfalls

The service makes several assumptions worth examining:

The BDF addresses are static: The service hardcodes eight specific PCIe addresses. This assumes the GPU topology won't change — no card reseating, no BIOS reconfiguration, no additional PCIe devices that shift bus numbers. In a production system this is usually safe, but any hardware change would silently break the service.

setpci is available: The service depends on setpci being installed (part of the pciutils package). On a minimal server build, this might not be present. The assistant doesn't add a dependency check or install step.

The register value is correct for all GPUs: The value 5937 assumes a specific PCIe capability structure layout. While this is standardized, different GPU generations or firmware versions could theoretically use different capability offsets. The assistant is implicitly trusting that the RTX PRO 6000 Blackwell GPUs all respond identically.

No error handling: The ExecStart command is a single bash command with no error checking. If any setpci call fails (e.g., because a GPU is in reset or the BDF doesn't exist), the service still reports success with the final echo. A more robust implementation might check exit codes or validate the setting afterward.

The Broader Tuning Philosophy

This message represents more than just a systemd unit file. It embodies a critical lesson in performance engineering: a tuning that doesn't survive reboot is not a tuning at all. The assistant had already verified that MaxReadReq=4096 improved H2D bandwidth to ~53 GB/s (as shown in <msg id=1287-1288>), but that verification was done with a runtime setpci command that would vanish on reboot. The systemd service closes the loop, transforming an ephemeral experiment into a permanent configuration.

This pattern recurs throughout the session. The kernel cmdline changes (amd_pstate=active, processor.max_cstate=1) were made persistent by writing to /etc/kernel/cmdline and running proxmox-boot-tool refresh. The sysctl settings were made persistent by writing to /etc/sysctl.d/99-gpu-compute.conf. Each fix follows the same philosophy: identify the parameter, test it at runtime, then encode it into the system's persistent configuration layer.

Output Knowledge Created

This message produces concrete, actionable infrastructure:

  1. A systemd unit file (/etc/systemd/system/gpu-pcie-tuning.service) that can be deployed on any system with the same GPU topology.
  2. A documented set of BDF addresses for eight RTX PRO 6000 Blackwell GPUs and their root ports, providing a reference for future PCIe debugging.
  3. A validated PCIe register value (5937 for CAP_EXP+0x08.w) that correctly sets MaxReadReq to 4096 on these specific GPUs.
  4. A dependency chain showing the correct ordering relative to nvidia-persistenced.service. The input knowledge required to create this includes: familiarity with PCIe configuration space, knowledge of the setpci tool and its register encoding, understanding of systemd unit file syntax and dependency ordering, the specific BDF topology of the target system (obtained via lspci), and awareness that the NVIDIA driver resets PCIe configuration on load.

Conclusion

Message [msg 1305] is a quiet but essential moment in a larger optimization journey. It doesn't introduce a new algorithm or discover a breakthrough performance technique — it does something arguably more important for production systems: it makes existing gains permanent. The systemd service for GPU PCIe MaxReadReq is the kind of infrastructure that separates a one-time benchmark run from a reliably performant deployment. In the ongoing quest to bridge the gap between 10 tok/s and 309 tok/s theoretical maximum, every persistent configuration matters, and this message ensures that at least one tuning parameter will survive the next reboot.