The Verification That Changed Nothing: How a P2P Benchmark Confirmed System Tuning Hadn't Broken GPU Communication

Introduction

In the middle of an intensive optimization campaign targeting the GLM-5-NVFP4 large language model on an 8-GPU RTX PRO 6000 Blackwell system, a seemingly mundane message arrived. On its surface, message [msg 1287] is just a bash command that runs a Python benchmark script over SSH and prints a table of GPU peer-to-peer (P2P) bandwidth numbers. But this message sits at a critical inflection point in the session: it is the gatekeeper between system tuning and production benchmarking. Before the assistant could rerun the costly inference benchmark suite, it needed to prove that the aggressive kernel and PCIe tuning just applied had not broken the fundamental GPU communication fabric. This article examines that single message in depth, unpacking the reasoning, assumptions, and knowledge embedded in what appears to be a routine verification step.

The Message Itself

The assistant executes the following command on the remote inference server:

ssh root@10.1.230.174 'source /root/ml-env/bin/activate && python3 /tmp/p2p_bench.py'

The script produces this output:

GPU P2P Bandwidth Test - 256MB transfers, CUDA events timing
After tuning: MaxReadReq=4096, NUMA balancing off, C2 disabled
=================================================================
Pair              BW GB/s     Lat ms         Type
-----------------------------------------------------------------
GPU0->GPU1           49.5       5.05    SAME-NUMA
GPU1->GPU2           50.2       4.98    SAME-NUMA
GPU2->GPU3           49.5       5.05    SAME-NUMA
GPU0->GPU3           49.8       5.03    SAME...

The output is truncated in the conversation data, but the next message ([msg 1288]) reveals the full picture: same-NUMA pairs achieve ~50 GB/s, cross-NUMA pairs ~37 GB/s, and host-to-GPU transfers ~53 GB/s. These numbers are, as the assistant notes, "consistent with pre-tuning measurements."

Why This Message Was Written: The Gatekeeper Role

The message exists because of a deliberate four-step plan established by the user in [msg 1274]: "write down in system-improve.md, apply the changes runtime only for now, check that p2p works with some manual micro bench, then rerun inference benchmark." This sequence reveals a sophisticated understanding of risk management in systems engineering.

The assistant had just applied nine runtime system tuning changes to the Proxmox host (<msgs id=1278-1280>). These were not trivial adjustments. They included:

• PCIe MaxReadReq modification: Using setpci to change the MaxReadReq register on all eight GPUs and their root ports from 512 bytes to 4096 bytes. This is a hardware register write that directly modifies PCIe device configuration space.
• NUMA balancing disabled: A kernel parameter change that stops the automatic migration of memory pages between NUMA nodes.
• C2 deep sleep state disabled: Preventing CPUs from entering deeper idle states, which can introduce latency when GPUs need CPU interaction.
• NMI watchdog disabled: Removing a periodic interrupt that can perturb timing-sensitive GPU workloads.
• Scheduler autogroup disabled: Preventing the kernel from automatically grouping related processes, which can interfere with GPU dispatch patterns.
• Swappiness and dirty ratio reduction: Tuning virtual memory behavior to reduce background I/O pressure.
• Network buffer expansion: Increasing socket buffer sizes for potential inter-node communication. Each of these changes touches a different subsystem—PCIe configuration space, the CPU scheduler, the virtual memory manager, the network stack, and the interrupt system. Any one of them could potentially destabilize GPU communication. The P2P benchmark is the canary in the coal mine: if GPU-to-GPU copies still work at expected bandwidth, the system is fundamentally intact. If they don't, the assistant would need to roll back changes before proceeding. The decision to verify P2P before the inference benchmark is economically rational. A full inference benchmark of the GLM-5-NVFP4 model takes significant time and compute resources. Running it only to discover that GPU communication is broken would waste both. The P2P micro-benchmark, by contrast, completes in seconds and provides a direct signal about the health of the GPU interconnect fabric.

How the Benchmark Was Constructed: A Tale of Shell Escaping

The path to this message was not smooth. The assistant first attempted to run the benchmark as an inline Python script passed via -c to the Python interpreter ([msg 1284]). This failed with a zsh error: zsh:1: bad pattern: (GB/s):>10} | {Latency. The problem was that the Python f-string format specifiers—containing characters like >, (, and )—were being interpreted by zsh as glob patterns before the string was passed to Python.

The assistant then tried a heredoc approach ([msg 1285]), writing the script via cat > /tmp/p2p_bench.py << 'PYEOF'. This also failed with the same zsh error, because the heredoc content still contained the problematic f-strings.

Finally, in [msg 1286], the assistant switched to a different strategy: using scp /dev/stdin to pipe the script content directly to a file on the remote machine. This approach bypassed zsh's interpretation entirely because the script content was provided via stdin redirection (<< 'PYEOF'), and crucially, the assistant also rewrote the script to avoid f-strings entirely, using old-style % formatting instead. This dual fix—changing both the transport mechanism and the string formatting—finally succeeded.

This debugging sequence reveals an important aspect of the assistant's thinking: when a shell escaping issue arises, the assistant correctly identifies the root cause (zsh interpreting f-string curly braces and format specifiers) and applies two independent mitigations. The rewrite to avoid f-strings is particularly telling—it shows the assistant recognizing that even if the transport issue is fixed, the underlying Python code should be made more robust against shell interpretation.

Assumptions Embedded in the Benchmark

The benchmark script makes several assumptions that are worth examining:

Assumption 1: 256MB transfers are representative. The script uses 256MB buffers for all measurements. This is large enough to amortize PCIe transaction overhead and measure sustained bandwidth, but small enough to fit comfortably in GPU memory (8 GPUs × 256MB = 2GB, well within the 24GB+ per GPU). The assumption is that the bandwidth characteristics at this size are representative of the model's communication patterns. For a transformer model doing tensor parallelism, the activations passed between GPUs are typically on the order of megabytes to tens of megabytes per layer, so 256MB is a reasonable upper bound.

Assumption 2: CUDA events provide accurate timing. The script uses torch.cuda.Event with enable_timing=True to measure elapsed time. This is the standard approach for GPU timing, as it records timestamps on the GPU itself rather than relying on CPU-side time.time() calls, which would include PCIe round-trip latency and scheduler jitter. The assumption is that CUDA event timing is accurate and that the GPU clock is stable.

Assumption 3: 50 iterations are sufficient. The script runs 50 iterations of each copy operation and averages the result. This is a reasonable trade-off between statistical stability and runtime. With 50 iterations at ~5ms per iteration, each measurement takes about 250ms, and the full benchmark completes in seconds.

Assumption 4: The specific GPU pairs tested cover the relevant topology. The script tests same-NUMA pairs (0→1, 1→2, 2→3, 4→5, 5→6, 6→7), cross-NUMA pairs (0→4, 0→7, 3→4, 3→7), and host-to-GPU transfers (0, 4). This covers the major communication patterns: intra-socket, inter-socket, and host-device. The assumption is that if these representative pairs work, the full mesh is healthy.

Assumption 5: The benchmark script was correctly transferred. After the shell escaping saga, the assistant used scp to transfer the file. The assumption is that the file arrived intact and that the Python code is syntactically correct. This is verified implicitly by the fact that the script ran and produced output.

The Input Knowledge Required

To fully understand this message, a reader needs knowledge in several domains:

GPU topology and NUMA: The system has 8 GPUs spread across two CPU sockets (NUMA nodes). GPUs 0-3 are on NUMA node 0, GPUs 4-7 on NUMA node 1. Communication within a NUMA node goes through the same CPU socket's PCIe root complex, while cross-NUMA communication traverses the inter-socket interconnect (typically Infinity Fabric or UPI). This explains why same-NUMA bandwidth (~50 GB/s) is higher than cross-NUMA (~37 GB/s).

PCIe configuration registers: The MaxReadReq change targets the PCIe Device Control register at offset CAP_EXP+0x08. The value 5937 encodes MaxReadReq=4096 bytes in the appropriate bit field. Understanding that this is a hardware register write, not a software configuration, is crucial—it means the change persists until the next reboot or PCIe reset.

CUDA P2P semantics: GPU-to-GPU copies on the same PCIe fabric use direct peer-to-peer transfers without involving the CPU or system memory. The bandwidth is limited by the PCIe topology: GPUs under the same root port share upstream bandwidth, while GPUs on different root ports can communicate at full link speed.

The distinction between MaxReadReq and MaxPayload: This is a subtle but important point that the assistant addresses in [msg 1288]. PCIe MaxReadReq controls the maximum size of a read request (used when a GPU reads from host memory), while MaxPayload controls the maximum size of a write (used when a GPU writes to another GPU). P2P copies are GPU-initiated writes, so they are limited by MaxPayload (still at 256 bytes), not MaxReadReq (now at 4096 bytes). This explains why the benchmark shows no improvement from the tuning.

The Output Knowledge Created

This message produces several pieces of actionable knowledge:

Confirmation of system integrity: All eight GPUs communicate correctly after the tuning changes. No PCIe configuration was corrupted, no GPU was lost, and the interconnect fabric is healthy.

Baseline bandwidth measurements: Same-NUMA P2P at ~50 GB/s, cross-NUMA at ~37 GB/s, H2D/D2H at ~53 GB/s. These serve as a reference point for future tuning efforts. If subsequent changes degrade these numbers, the assistant will know something is wrong.

Negative result on MaxReadReq effectiveness: The PCIe MaxReadReq change did not improve P2P bandwidth. This is itself valuable knowledge—it means the assistant should not expect this particular tuning to help with tensor-parallel communication. The assistant correctly diagnoses why this is the case (P2P uses writes, not reads), demonstrating a deep understanding of PCIe transaction types.

Green light for inference benchmarking: With the P2P check passed, the assistant can proceed to the next step: rerunning the full inference benchmark suite. This is the primary purpose of the message—it unblocks the critical path.

Mistakes and Incorrect Assumptions

The most notable mistake is the initial assumption that MaxReadReq=4096 would improve P2P bandwidth. The assistant does not explicitly state this assumption, but the entire tuning sequence was motivated by a desire to improve GPU communication performance. The benchmark results show no improvement, and the assistant correctly identifies the reason in the follow-up message. This is not a failure—it is a successful hypothesis test. The assistant formed a hypothesis (MaxReadReq matters for P2P), tested it (via the benchmark), and refined its understanding (P2P uses writes, not reads).

A more practical mistake was the shell escaping issue in <msgs id=1284-1285>. The assistant attempted to pass a Python script containing f-strings through a shell that interprets curly braces and format specifiers. This is a common pitfall when working with remote execution through SSH. The assistant's debugging process—trying inline execution, then heredoc, then scp with code modification—shows a systematic approach to problem-solving, even if the initial attempts failed.

The Thinking Process Visible in the Message

While the message itself is just a bash command and its output, the surrounding context reveals the assistant's reasoning. The assistant is executing a structured plan with clear dependencies:

1. Document findings (completed in [msg 1276])
2. Apply runtime fixes (completed in <msgs id=1278-1280>)
3. Verify P2P (this message)
4. Rerun inference benchmark (next step) The assistant is methodically checking off items in its todo list, updating statuses as it goes. The P2P verification is treated as a critical checkpoint—a necessary condition for proceeding to the expensive benchmark. The assistant does not skip this step or assume the tuning is safe; it demands empirical evidence. The choice of benchmark parameters (256MB, 50 iterations, specific GPU pairs) reflects a deliberate design: large enough to be meaningful, fast enough to be practical, and comprehensive enough to cover all relevant communication patterns. The assistant is balancing thoroughness against time—a key skill in systems engineering.

Conclusion

Message [msg 1287] is a textbook example of a verification gate in a complex systems optimization workflow. It is not glamorous—it does not discover a new optimization or achieve a record throughput number. But it serves an essential function: it provides empirical evidence that the system is still healthy after invasive tuning, and it generates a negative result (MaxReadReq doesn't help P2P) that prevents wasted effort on a dead-end optimization path. In the broader narrative of the GLM-5-NVFP4 optimization campaign, this message represents the disciplined engineering practice that separates systematic optimization from random tweaking. The assistant did not just apply changes and hope for the best; it verified, measured, and understood the results before moving forward.