The Sysctl That Anchored a Reboot: Persistent Kernel Tuning for Blackwell GPU Inference

Introduction

In the middle of a marathon optimization session for GLM-5-NVFP4 inference on an 8-GPU Blackwell system, a seemingly unremarkable message appears. The assistant writes a file:

cat > /etc/sysctl.d/99-gpu-compute.conf << EOF
# GPU compute performance tuning
kernel.numa_balancing = 0
kernel.sched_autogroup_enabled = 0
vm.swappiness = 10
vm.dirty_ratio = 5
vm.dirty_background_ratio = 2
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 1048576
net.core.wmem_default = 1048576
net.core.netdev_max_backlog = 5000
kernel.perf_event_paranoid = 1
EOF

Ten lines of kernel parameters, a heredoc, and a verification cat. On its surface, this is one of the most mundane operations in systems administration. Yet this message sits at a critical inflection point in the conversation: it is the bridge between ephemeral tuning applied at runtime and permanent tuning that will survive a kernel upgrade and system reboot. Understanding why this message exists, what each parameter means, and what assumptions it encodes reveals the depth of the optimization effort that preceded it.

The Broader Context: From 10 tok/s to Theoretical Limits

To appreciate this message, one must understand the journey that led to it. The session had been wrestling with a stark performance gap: the GLM-5-NVFP4 model, a massive Mixture-of-Experts transformer running on eight NVIDIA RTX PRO 6000 Blackwell GPUs, was achieving only about 10.36 tokens per second in single-stream inference. The theoretical maximum, computed from first principles using the model's arithmetic intensity and the GPU's FP4 tensor core throughput, was approximately 309 tok/s. The system was operating at a mere 3.4% efficiency.

A comprehensive parallel system audit ([msg 1295] context) had uncovered multiple misconfigurations: the CPU governor was using acpi-cpufreq instead of the modern amd_pstate driver; the kernel was an outdated 6.8.12; NUMA balancing was enabled, causing unnecessary page migrations; deep CPU C-states were introducing latency on wake; and the PCIe MaxReadReq was stuck at 512 bytes instead of the optimal 4096. These findings had been documented and runtime fixes had been applied, but they were all transient — a single reboot would wipe them out.

The user's instruction was clear: "Try new kernel and the reboot-requiring fixes, feel free to reboot that proxmox node too" ([msg 1294]). This directive set in motion a careful sequence of operations: install the 6.14.11 Proxmox kernel, install matching kernel headers, rebuild the NVIDIA DKMS driver (590.48.01) for the new kernel, update the kernel command line with amd_pstate=active and processor.max_cstate=1, and — critically — make all runtime tunings persistent. The subject message is the sysctl persistence step.

Anatomy of the Message

The message executes a single bash command on the remote Proxmox host (root@10.1.2.6). It writes a configuration file to /etc/sysctl.d/99-gpu-compute.conf and then prints its contents to confirm. The 99- prefix ensures it loads late in the sysctl initialization order, after other defaults have been applied, so these GPU-specific values take precedence.

The parameters fall into three categories:

NUMA and scheduling (lines 3-4): kernel.numa_balancing = 0 disables automatic NUMA page migration, which can cause performance degradation for GPU workloads that have stable memory access patterns. kernel.sched_autogroup_enabled = 0 disables the autogroup feature in the Completely Fair Scheduler, preventing the kernel from automatically grouping tasks by TTY session — for a dedicated inference server, this grouping adds overhead without benefit.

Memory management (lines 5-7): vm.swappiness = 10 reduces the kernel's tendency to swap, keeping inference data in RAM. vm.dirty_ratio = 5 and vm.dirty_background_ratio = 2 aggressively limit the amount of dirty page cache, ensuring that write-heavy operations (like logging or checkpointing) don't build up large backlogs that could trigger latency spikes.

Network buffers (lines 8-11): The four net.core.* parameters increase socket buffer sizes. While the inference workload primarily uses localhost connections (the benchmark tool communicates with the SGLang server via HTTP on 127.0.0.1:8000), larger buffers help absorb bursty request patterns and reduce the chance of dropped packets under high concurrency.

Security/perf (line 12): kernel.perf_event_paranoid = 1 allows unprivileged users to access perf event counters, which is useful for profiling without requiring root.

The Persistence Principle

The critical insight about this message is that it is not about discovering these parameters — they had already been applied at runtime via echo value &gt; /proc/sys/... commands. The message is about institutionalizing them. Before a reboot, runtime tunings are fragile; after a reboot, they are gone. Writing to /etc/sysctl.d/ transforms temporary experiments into permanent policy.

This distinction matters deeply in the context of the optimization workflow. The assistant had spent significant effort auditing the system, identifying bottlenecks, and testing fixes. Each runtime tuning was a hypothesis validated by measurement. The sysctl file represents the commitment: "These hypotheses are confirmed, and we want them applied on every boot from now on." Without this step, the kernel upgrade and reboot would have erased all progress, and the post-reboot benchmarks would have reflected the same untuned system as before.

Assumptions and Knowledge Required

The message makes several assumptions that are worth examining:

  1. The sysctl mechanism is the right tool. On Proxmox/Debian systems, /etc/sysctl.d/ is the standard mechanism for persistent kernel parameter configuration. This is correct and idiomatic.
  2. These values are universally beneficial for GPU compute. The assistant assumes that the tuning parameters identified for this specific workload (GLM-5-NVFP4 inference) will not harm other workloads or system stability. This is a reasonable assumption for a dedicated ML inference server.
  3. The values are safe across kernel versions. The parameters chosen are well-established and present in both the old 6.8 kernel and the new 6.14 kernel. The assistant did not check for parameter availability in the new kernel — a minor risk, but a safe one for these common parameters.
  4. No reboot is needed to apply them. Sysctl files are read by the systemd-sysctl service at boot, but can also be applied immediately with sysctl --system. The assistant does not apply them immediately in this message, presumably because a reboot is imminent anyway. One notable omission: the file does not include kernel.numa_balancing's sibling parameter kernel.numa_balancing_ratelimit_ms, which controls the rate at which NUMA balancing can migrate pages. Setting numa_balancing = 0 disables it entirely, so this is not needed.

The Thinking Process Visible in the Sequence

Looking at the surrounding messages, a clear methodology emerges. Message [msg 1303] completes the kernel cmdline update. Message [msg 1304] (the subject) writes the sysctl file. Message [msg 1305] creates a systemd service for PCIe MaxReadReq tuning. Message [msg 1306] verifies the boot order. Each step is a dependency for the next: you install the kernel first, then configure how it boots, then ensure tunings survive the boot, then handle hardware-specific settings that sysctl cannot address (PCIe registers require setpci), and finally verify the boot configuration.

The assistant is working through a checklist, and the sysctl message is the step that converts software tunings from runtime to persistent. It is the difference between "works now" and "works forever."

Conclusion

The message that writes /etc/sysctl.d/99-gpu-compute.conf is, on its face, a trivial systems administration operation. But in the context of a deep optimization effort — one that spans kernel upgrades, DKMS rebuilds, NUMA topology analysis, and FP4 GEMM kernel profiling — it represents a critical commitment point. It is the moment when temporary fixes become permanent infrastructure. The ten lines of kernel parameters encode hours of diagnostic work: the NUMA audit that revealed cross-socket penalties, the latency analysis that implicated C-state transitions, the network benchmarking that informed buffer sizing. Without this message, the reboot that follows would have reset the system to its untuned state, and the optimization journey would have to begin again from scratch.