The Quoting Trap: How a Systemd Service Led to a Cleaner Script Architecture for GPU Passthrough

In the intricate dance of configuring a high-performance AI inference server across multiple GPUs, the smallest details can cascade into significant problems. Message 6080 of this opencode session captures a pivotal moment where the assistant recognized a subtle quoting conflict in a systemd service file and pivoted to a more robust architectural pattern — extracting an inline shell command into a standalone script. This seemingly minor refactor reveals deep knowledge of systemd's parsing rules, Linux PCI device management, and the importance of designing for maintainability in production infrastructure.

The Message

The assistant wrote:

The single quotes in ExecStart will be a problem — systemd uses them for quoting in ExecStart but the bash -c argument also uses them. Let me use a script file instead, which is cleaner:

>

```bash ssh root@10.1.2.6 'cat > /usr/local/bin/gpu-vfio-split.sh << '\''SCRIPT'\'' #!/bin/bash # Bind NUMA 1 GPUs to vfio-pci for VM passthrough # NUMA 1 PCI addresses: 81:00.0, 91:00.0, e1:00.0, f1:00.0

>

DEVICES="0000:81:00.0 0000:91:00.0 0000:e1:00.0 0000:f1:00.0"

>

# Ensure vfio-pci is loaded modprobe vfio-pci

>

for dev in $DEVICES; do drv=$(basename "$(readlink -f /sys/bus/pci/devices/$dev/driver)" 2>/dev/null)

>

if [ "$drv" = "vfio-pci" ]; then echo "$dev already on vfio-pci" continue fi

>

# Unbind from current driver if [ -n "$drv" ] && [ "$drv" != "." ]; then echo "$dev: unbinding from $drv" echo "$dev" > "/sys/bus/pci/drivers/$drv/unbind" 2>/dev/null fi

>

# Register device ID with vfio-pci (idempotent) echo "10de 2bb5" > /sys/bus/pci/drivers/vfio-pci/new_id 2>/dev/null

>

# Bind to vfio-pci echo "$dev" > /sys/bus/pci/drivers/vfio-pci/bind 2>/dev/null

>

drv_new=$(basename "$(readlink -f /sys/bus/pci/devices/$dev/driver)" 2>/dev/null) if [ "$drv_new" = "vfio-pci" ]; then echo "$dev -> vfio-pci OK" else echo "$dev -> vfio-pci FAILED (driver=$drv_new)" fi done SCRIPT chmod +x /usr/local/bin/gpu-vfio-split.sh echo "script created"' script created ```

Context and Motivation: The GPU Topology Problem

To understand why this message was written, we must step back and examine the broader context. The assistant was managing a Proxmox host (node kpro6) equipped with 8 NVIDIA RTX PRO 6000 Blackwell GPUs, each with 96 GB of VRAM. The infrastructure goal was to split these 8 GPUs between two workloads: 4 GPUs (on NUMA node 0) would remain bound to the NVIDIA driver for use by an LXC container running SGLang for inference serving, while the other 4 GPUs (on NUMA node 1) needed to be bound to the vfio-pci driver for passthrough to a SEV-SNP encrypted VM.

This split is non-trivial because all 8 GPUs share the same PCI device ID (10de:2bb5). The standard mechanism for binding devices to vfio-pci at boot — the vfio-pci.ids= kernel parameter — would bind all matching devices, which is the opposite of what was needed. Instead, the assistant needed a selective approach: bind only the 4 NUMA 1 GPUs to vfio-pci, leaving the NUMA 0 GPUs for the NVIDIA driver.

The solution required a boot-time script that would unbind the target GPUs from whatever driver claimed them (likely the NVIDIA driver, which loads early) and rebind them to vfio-pci. This had to happen before nvidia-persistenced.service started, to prevent the NVIDIA driver from fully initializing those GPUs.

The Original Approach and Its Flaw

In the immediately preceding messages ([msg 6078] and [msg 6079]), the assistant had created a systemd oneshot service called gpu-vfio-split.service. The service used an ExecStart directive with an inline bash command:

ExecStart=/bin/bash -c 'for dev in 0000:81:00.0 ...; do ... done'

This approach embedded the entire shell script as a quoted string inside the systemd unit file. While this pattern is common and often works for simple commands, it becomes fragile when the shell script itself contains single quotes, variable expansions, and complex control flow. Systemd's ExecStart= has its own parsing rules: it interprets quotes, escapes, and variable references before passing the string to the shell. Nesting shell quoting inside systemd quoting creates a minefield of escaping issues.

When the assistant verified the file in [msg 6079], the quoting was already mangled — the nested single quotes inside the bash -c argument conflicted with systemd's quoting, producing a service file that would fail at boot or behave unpredictably. The assistant recognized this problem immediately upon reviewing the output.

The Decision: Extract to a Script File

The core decision in this message was to abandon the inline approach and create a standalone script at /usr/local/bin/gpu-vfio-split.sh. This is a textbook example of the "Extract Method" refactoring applied to infrastructure code. The reasoning is sound on multiple levels:

1. Eliminating quoting conflicts. By moving the shell logic into a dedicated script file, the systemd service only needs to reference the script path (ExecStart=/usr/local/bin/gpu-vfio-split.sh), with zero quoting required. The script itself is written in plain bash with standard quoting, free from the constraints of systemd's parser.

2. Testability. A standalone script can be executed manually for testing, debugging, and validation. The assistant could run /usr/local/bin/gpu-vfio-split.sh directly on the host to verify the unbind/bind logic works, without needing to invoke systemd or reboot. This dramatically reduces the feedback loop for troubleshooting.

3. Maintainability. Scripts in /usr/local/bin/ are easier to edit, review, and version-control than inline commands embedded in unit files. Future operators (or the assistant itself) can modify the script without needing to regenerate the systemd unit file.

4. Robustness. The extracted script includes error checking, status reporting, and idempotency guards that would have been unwieldy in an inline ExecStart. It checks whether each device is already bound to vfio-pci before attempting rebinding, handles the case where a device has no current driver, and reports success or failure for each device.

The Script Design: A Closer Look

The script the assistant wrote is a model of clarity and robustness for this kind of low-level device management. Let's examine its design decisions:

Idempotency. The first check in the loop is if [ &#34;$drv&#34; = &#34;vfio-pci&#34; ], which skips devices already correctly bound. This is critical because the script may run on every boot, and blindly unbinding/rebinding devices that are already in the correct state could cause unnecessary PCI bus resets or race conditions.

Graceful unbinding. The script only unbinds a device from its current driver if a driver is actually attached (if [ -n &#34;$drv&#34; ] &amp;&amp; [ &#34;$drv&#34; != &#34;.&#34; ]). The &#34;.&#34; check handles the case where readlink -f returns a dot (indicating no symbolic link), which can happen if the device has no driver at all. This prevents writing to a non-existent unbind path.

Driver registration. The line echo &#34;10de 2bb5&#34; &gt; /sys/bus/pci/drivers/vfio-pci/new_id registers the GPU's vendor:device ID with vfio-pci. The assistant notes this is "idempotent" — writing the same ID multiple times is harmless. This step is necessary because vfio-pci needs to know which device IDs it should claim; without it, the bind operation would fail.

Module loading. The script explicitly calls modprobe vfio-pci to ensure the driver module is loaded before attempting to bind devices. This is a safety measure — on some kernel configurations, vfio-pci may not be auto-loaded, and the bind would silently fail.

Verification. After each bind attempt, the script re-reads the driver symlink and reports whether the operation succeeded or failed, including the actual driver name on failure. This diagnostic output would appear in the systemd journal, making boot-time failures traceable.

Assumptions Embedded in the Message

The assistant made several assumptions, most of which are well-justified but worth examining:

That /usr/local/bin/ exists and is in PATH. On standard Linux installations, this directory exists and is included in the default PATH. The assistant used the full path in the script creation command, so even if it's not in PATH, the systemd service can reference it directly.

That the PCI addresses are stable. The four PCI addresses (0000:81:00.0, 0000:91:00.0, 0000:e1:00.0, 0000:f1:00.0) were determined earlier in the session by examining the PCI topology and NUMA node assignments. The assistant assumes these addresses will not change across reboots, which is generally true for discrete PCIe devices in fixed slots, but could be violated by BIOS reconfiguration, PCIe hotplug, or certain firmware settings.

That the device ID 10de:2bb5 is correct. This was confirmed earlier in the session via lspci -n. The assistant assumes this ID is stable and unique to the RTX PRO 6000 Blackwell GPUs.

That vfio-pci is available. The assistant assumes the kernel has the vfio-pci module available (either built-in or as a loadable module). On Proxmox hosts, this is typically true as VFIO is a core component of the virtualization stack.

That sysfs driver binding works. The mechanism of writing to /sys/bus/pci/drivers/&lt;driver&gt;/unbind and /sys/bus/pci/drivers/&lt;driver&gt;/bind is a standard Linux kernel interface, but it can fail if the device is in use, if the driver refuses to release it, or if the PCI subsystem is in an inconsistent state.

The SSH Heredoc Trick

One subtle but noteworthy aspect of this message is the SSH command itself. The assistant used a clever quoting trick to pass a heredoc through SSH:

ssh root@10.1.2.6 'cat > /usr/local/bin/gpu-vfio-split.sh << '\''SCRIPT'\''

The &#39;\&#39;&#39;SCRIPT&#39;\&#39;&#39; pattern is a way to embed a single quote inside a single-quoted string in bash. The sequence &#39;\&#39;&#39; breaks the single-quoted string, adds an escaped single quote, and resumes the single-quoted string. This produces the literal word SCRIPT as the heredoc delimiter, which is then terminated on the remote side. This is an advanced shell quoting technique that demonstrates deep familiarity with bash's quoting rules.

Input Knowledge Required

To fully understand this message, a reader needs knowledge of:

Output Knowledge Created

This message produced:

The Thinking Process

The assistant's reasoning is visible in the structure of the message. It began by identifying the problem: "The single quotes in ExecStart will be a problem — systemd uses them for quoting in ExecStart but the bash -c argument also uses them." This shows a clear understanding of the root cause, not just a surface-level observation that "the file looks wrong."

The solution — "Let me use a script file instead, which is cleaner" — reflects a design philosophy of separating concerns. The systemd service should be a thin wrapper that triggers an action; the action itself should be encapsulated in a testable, maintainable script. This is the same principle that drives the separation of application code from configuration management.

The script itself was written with care: comments explain each section, variable names are descriptive, error paths are handled, and diagnostic output is preserved. The assistant even included the chmod +x step to ensure the script is executable. This attention to detail suggests the assistant was thinking about the full lifecycle of the script — not just creating it, but ensuring it would work on boot, be debuggable if it failed, and be understandable to anyone who reads it later.

Conclusion

Message 6080 is a masterclass in infrastructure refactoring at the micro scale. What could have been a quick fix — just adjust the quoting in the ExecStart line — was instead a deliberate architectural improvement that eliminated an entire class of quoting bugs, improved testability, and produced a more maintainable system. The assistant recognized that the inline approach had fundamental fragility and chose to invest a few extra minutes in a cleaner design. This kind of judgment — knowing when to patch a problem and when to refactor the approach — is the hallmark of experienced systems engineering.

The message also illustrates a broader lesson: in complex systems, the interface between different configuration layers (systemd, bash, kernel sysfs, SSH) is where bugs breed. By reducing the number of layers that must coordinate on a single line of code — moving from "systemd quoting wrapping bash quoting wrapping shell logic" to "systemd references a file that contains shell logic" — the assistant eliminated the interface entirely. The result is simpler, more robust, and easier to debug.