The TP=4 Transition: Adapting SGLang to a Split GPU Topology

In the middle of a complex infrastructure reconfiguration spanning Proxmox virtualization, GPU passthrough, and distributed inference serving, a single short message marks the critical pivot point where hardware reality meets software configuration. The subject message at index 6067 is deceptively simple:

Need to change --tp 8 to --tp 4. Let me also read the local copy and update both:

This is followed by a [read] tool invocation that retrieves the local copy of the SGLang systemd service file from the path /home/theuser/glm-kimi-sm120-rtx6000bw/sglang-qwen.service, showing the first 13 lines of a unit file describing a service for "SGLang Qwen3.5-397B-A17B NVFP4" running under CUDA 13.0.

On its surface, this appears to be a trivial edit — changing a single integer from 8 to 4 in a command-line flag. But this message sits at the confluence of a much larger story: the physical re-partitioning of eight NVIDIA RTX PRO 6000 Blackwell GPUs across two virtualized environments, the careful management of NUMA topology, and the cascading consequences that a simple topology change can trigger in a distributed deep learning serving stack.

The Context: Why This Message Exists

To understand why the assistant writes "Need to change --tp 8 to --tp 4," one must look at the preceding 25 messages in the conversation. The assistant has just completed a delicate surgical operation on the Proxmox host (kpro6): splitting the 8× RTX PRO 6000 Blackwell GPUs into two groups of four, based on their NUMA node affinity.

The original configuration bound all 8 GPUs to the nvidia driver and passed all of them through to an LXC container (ID 129) running SGLang. The SGLang service was configured with --tp 8, meaning tensor parallelism across all 8 GPUs — the model (Qwen3.5-397B-A17B NVFP4) was sharded across every available GPU.

The new topology, established over messages [msg 6048] through [msg 6058], does the following:

• NUMA 0 GPUs (PCI addresses 01:00.0, 11:00.0, 61:00.0, 71:00.0) remain bound to the nvidia driver and are passed through to LXC 129.
• NUMA 1 GPUs (PCI addresses 81:00.0, 91:00.0, e1:00.0, f1:00.0) are unbonded from nvidia and rebound to vfio-pci for passthrough to a separate SEV-SNP VM. The assistant verified the result in [msg 6064] by running nvidia-smi inside the LXC container, confirming exactly 4 GPUs are visible. The todo list is updated to reflect completion of the GPU binding and LXC config update steps. Now, the assistant must update the SGLang service to match the new hardware reality — hence this message.

The Reasoning: A Mechanical but Critical Decision

The decision to change --tp 8 to --tp 4 is, at one level, entirely mechanical. The tensor parallelism parameter must equal the number of GPUs available to the inference server. If you have 4 GPUs, you set --tp 4. There is no alternative configuration that would allow TP=8 on 4 GPUs — SGLang would either crash during initialization (unable to find the missing GPUs) or hang indefinitely waiting for NCCL ranks that don't exist.

However, the assistant's reasoning also contains an implicit judgment: that the model can still run effectively on 4 GPUs. The Qwen3.5-397B-A17B NVFP4 is a Mixture-of-Experts model with approximately 397 billion total parameters but only about 17 billion active parameters per token, stored in NVFP4 (4-bit floating point) quantization. The original deployment at TP=8 was likely chosen to maximize KV cache capacity and throughput, not because the model couldn't fit on fewer GPUs. The assistant's assumption that TP=4 will work is reasonable given the aggressive quantization and MoE architecture.

The assistant also makes a deliberate operational choice: "Let me also read the local copy and update both." This reveals an awareness that the service file exists in two places — a local copy on the workstation at /home/theuser/glm-kimi-sm120-rtx6000bw/sglang-qwen.service and a deployed copy on the remote server at /etc/systemd/system/sglang-qwen.service. By reading the local copy first, the assistant can edit it and then push it to the remote server (which it does in the following messages [msg 6068] and [msg 6069]), ensuring both copies stay synchronized. This is a good operational practice that prevents configuration drift.

Assumptions Embedded in the Message

Several assumptions underpin this seemingly simple message:

1. The model fits on 4 GPUs. The assistant assumes that the NVFP4 quantized Qwen3.5-397B-A17B model can be loaded with TP=4 without running out of GPU memory. This is a reasonable assumption given the 4-bit quantization and MoE sparsity, but it's not verified until the server actually starts.
2. No other configuration changes are needed. The assistant assumes that changing --tp is sufficient — that the model path, CUDA environment, NCCL settings, and other parameters remain valid. This assumption will be tested when the server starts.
3. The local and remote copies are in sync. The assistant reads the local copy assuming it reflects the current state of the deployed service. This is likely true since the assistant previously deployed this service, but it's an assumption worth noting.
4. TP=4 will provide adequate performance. The assistant implicitly assumes that the throughput with 4 GPUs is acceptable for the intended use case. Given that the earlier benchmarks with TP=8 achieved impressive throughput (over 2100 tok/s at high concurrency), halving the GPU count will roughly halve throughput, but the assistant apparently considers this acceptable.

What Input Knowledge Is Required

To understand this message, the reader needs knowledge spanning several domains:

• Tensor parallelism (TP): Understanding that --tp controls how model layers are sharded across GPUs, and that TP must equal the number of available GPUs.
• SGLang architecture: Knowing that SGLang uses NCCL for inter-GPU communication and that the --tp flag is passed to the underlying distributed runtime.
• Proxmox virtualization: Understanding that LXC containers can be configured with device passthrough via lxc.mount.entry directives, and that PCI mappings in /etc/pve/mapping/pci.cfg control which physical devices are available.
• NUMA topology: Knowing that GPU-to-GPU communication within a NUMA node is faster than across NUMA nodes, and that the split preserves NUMA locality (all 4 remaining GPUs are on NUMA 0).
• Systemd service management: Understanding that service files must be updated and daemon-reload executed before changes take effect.
• NVFP4 quantization: Knowing that 4-bit floating point quantization dramatically reduces model memory footprint, making it feasible to run a 397B-parameter model on 4 GPUs with 96 GB each.

The Output Knowledge Created

This message itself doesn't produce new knowledge — it's a transitional action. But it initiates the creation of an updated service configuration. The output knowledge flows forward:

• The local service file is edited ([msg 6068]) to change --tp 8 to --tp 4.
• The updated file is copied to the remote server ([msg 6069]) via SCP.
• Systemd is reloaded and the SGLang server is started ([msg 6070]). The real knowledge creation happens when the server starts and either succeeds or fails. In this case, the server starts but later encounters a critical issue: NCCL hangs due to P2P DMA corruption under SEV-SNP IOMMU translation ([msg 6072] onward). This reveals an incorrect assumption — that simply changing TP would be sufficient. The NCCL hang requires adding NCCL_P2P_DISABLE=1 to the environment, a fix that emerges from diagnosing the failure.

The Thinking Process Visible

The assistant's thinking is telegraphically concise: "Need to change --tp 8 to --tp 4." This brevity reflects the clarity of the situation — the cause and effect are unambiguous. The GPU count changed from 8 to 4, so the TP parameter must follow. There's no deliberation about alternatives, no exploration of whether TP=2 or TP=1 might work, no consideration of mixed-precision strategies. The mapping is one-to-one: GPUs available equals TP value.

The second clause — "Let me also read the local copy and update both" — reveals a more nuanced operational awareness. The assistant recognizes that the service file exists in two locations and that maintaining consistency between them is important. This is a lesson learned from experience: configuration drift between local and remote copies leads to confusion and hard-to-diagnose bugs.

Conclusion

Message 6067 is a hinge point in the conversation. Everything before it was about reconfiguring the physical and virtual infrastructure — unbinding GPUs, updating LXC configs, creating PCI mappings. Everything after it is about making the software stack work within the new constraints — updating the service file, starting the server, and then diagnosing and fixing the NCCL P2P corruption that emerges as a consequence of the new topology.

The message itself is short, but it carries the weight of the entire topology change. It's the moment where the assistant acknowledges that the hardware work is done and the software adaptation must begin. The simple change from 8 to 4 in a command-line flag represents the entire rebalancing of compute resources across two virtualized environments — a decision that will have ripple effects through the rest of the session as the assistant discovers that P2P DMA, NCCL transport, and driver compatibility all need attention before the server can serve requests reliably.