The Status Report That Tells the Whole Story: Deploying Qwen3.5-122B on Blackwell GPUs

In the middle of a sprawling, multi-session engineering effort spanning GPU topology reconfiguration, IOMMU debugging, model swaps, and performance benchmarking, a simple question arrives: "Deployed? What config are we running?" The user's query in [msg 6236] is terse, almost skeptical — a request for confirmation and clarity after a long chain of complex operations. The assistant's response in [msg 6237] is a masterclass in technical summarization: concise, precise, and packed with actionable information. This message is not merely an answer to a question; it is the culminating documentation of a deployment that involved splitting 8 Blackwell GPUs between two virtualization domains, diagnosing corrupted GPU-to-GPU DMA through an IOMMU, and ultimately landing on a stable, high-performance inference configuration.

The Context That Demanded This Message

To understand why this message was written, one must appreciate the complexity of the preceding work. The session had been through multiple major pivots: deploying GLM-5-NVFP4, then Kimi-K2.5 INT4, then Qwen3.5-397B NVFP4 on 8 GPUs, and finally Qwen3.5-122B-A10B BF16 on 4 GPUs. Each transition involved rebuilding SGLang from source, patching for SM120 (Blackwell) compatibility, fixing CUDA version mismatches, and wrestling with build systems. The most critical discovery was that the SEV-SNP configuration on the Proxmox host had switched IOMMU from passthrough mode to full translation mode (amd_iommu=on without iommu=pt), which silently corrupted every GPU-to-GPU P2P DMA transfer — a catastrophic failure mode that caused NCCL to hang during distributed initialization.

The assistant had already declared "Deployment Complete" in [msg 6235] with benchmark results and a comparison table. But the user's follow-up question suggests they wanted a more precise, structured answer — not just "it's deployed" but exactly what is running and why. The assistant's response in [msg 6237] delivers exactly that: a config snapshot that serves as both operational documentation and a foundation for future optimization.

What the Message Contains

The message is structured in four clear sections: the running configuration, the service file flags, benchmark results, and potential improvements. Let us examine each in turn.

The Running Configuration lists the model identity (Qwen3.5-122B-A10B BF16, native precision, no quantization), the hardware allocation (4 of the 8 RTX PRO 6000 Blackwell GPUs, tensor-parallelism degree 4), the KV cache configuration (BF16 dtype, 780,820 tokens capacity), and the key software stack choices: triton attention backend, auto MoE backend (since the model is BF16, not FP4), and critically, NCCL with P2P disabled, using Ring+LL over shared memory transport, with custom all-reduce also disabled.

The Service File Flags are quoted verbatim, showing the exact command-line arguments passed to SGLang. This is operational gold — anyone who needs to restart or replicate this deployment has the precise invocation. The flags include --reasoning-parser qwen3 and --tool-call-parser qwen3_coder, indicating this is configured for the model's native reasoning and tool-calling capabilities.

The Benchmark Results present a table of aggregate throughput and per-request throughput at concurrency levels from 1 to 128. The numbers tell a clear story: single-stream performance is 108 tok/s (respectable for a 122B BF16 model on 4 GPUs), scaling to 2,800 tok/s aggregate at C=128. The per-request throughput drops from 110 tok/s at C=1 to 41 tok/s at C=128, showing the expected degradation under high concurrency as the scheduler balances competing requests.

The Potential Improvements section lists three avenues: MTP/NEXTN speculative decoding (requiring --mamba-scheduler-strategy extra_buffer and SGLANG_ENABLE_SPEC_V2=1), fused MoE sum-all-reduce, and testing the flashinfer_cutlass MoE runner backend for BF16.

The Decisions Embedded in This Message

Although the message itself is a status report, it implicitly documents a series of critical decisions made earlier in the session. The most consequential is the choice to disable NCCL P2P entirely. This was not a performance optimization but a necessity: under IOMMU full translation mode, every GPU-to-GPU DMA transfer produced corrupted data. The assistant diagnosed this through IO_PAGE_FAULTs in dmesg and a CUDA P2P copy test that showed every cross-GPU transfer returning incorrect values. The fix — NCCL_P2P_DISABLE=1 — forces NCCL to fall back to shared memory transport, which bypasses the IOMMU entirely. This is a textbook example of working around a platform limitation: the hardware supports P2P (the driver reports canAccessPeer=True), but the virtualization layer breaks it.

Another embedded decision is the choice of TP=4 instead of TP=8. The earlier deployment of Qwen3.5-397B NVFP4 used all 8 GPUs. But the switch to the smaller 122B model, combined with the need to split GPUs between the LXC container and a SEV-SNP VM, meant only 4 GPUs were available for inference. The assistant adapted the tensor parallelism degree accordingly, and the benchmark results show this configuration is still highly performant.

The choice of BF16 native precision over FP4 quantization is also significant. The earlier 397B model used NVFP4 (NVIDIA's FP4 format) to fit the larger model into GPU memory. The 122B model fits comfortably in BF16 across 4 GPUs (234 GB total), avoiding quantization artifacts entirely. The assistant explicitly notes this as a quality advantage.

Assumptions and Their Implications

The message makes several assumptions about its audience. It assumes the reader is familiar with SGLang's architecture — that they understand what --attention-backend triton means, why --disable-custom-all-reduce matters, and what --reasoning-parser qwen3 configures. It assumes familiarity with NCCL transport protocols (Ring, LL, SHM). It assumes the reader knows what "MTP" and "NEXTN" refer to in the context of speculative decoding.

These assumptions are reasonable given the conversation history — the user and assistant have been working together on this deployment for many messages. But for an external reader, the message would benefit from additional context. The assistant does not explain why P2P is disabled, why custom all-reduce is disabled, or why MTP is off. These decisions are documented in earlier messages but not repeated here.

The Thinking Process Visible in the Response

The assistant's reasoning is evident in the structure of the response. It begins with the most fundamental information — what model is running and on what hardware — then progressively adds detail: the software configuration, the performance numbers, and finally the optimization roadmap. This is a logical flow from "what" to "how well" to "what next."

The inclusion of the exact service file flags is a deliberate choice. Rather than paraphrasing the configuration, the assistant quotes the command line verbatim, eliminating any ambiguity. This is the mark of an engineer who knows that operational precision matters — a single wrong flag can change model behavior or cause a crash.

The "Potential improvements" section reveals the assistant's forward-looking thinking. Even as it reports the current state, it is already considering the next optimizations. The note that MTP "should work now that NCCL is fixed" connects back to earlier struggles — speculative decoding had been tested earlier and failed, likely due to the same P2P corruption issue. The assistant is tracking dependencies between fixes and features.

Output Knowledge Created

This message creates a definitive snapshot of the deployment state at a specific point in time. It serves as:

Conclusion

Message [msg 6237] is far more than a simple status update. It is the distillation of dozens of earlier decisions, hours of debugging, and multiple architecture pivots into a clear, actionable summary. It answers the user's question completely while simultaneously serving as documentation, a benchmark baseline, and a roadmap. In the context of a complex ML infrastructure deployment, such moments of clarity are invaluable — they transform a chaotic engineering process into a reproducible, understood system. The message exemplifies the kind of technical communication that turns an AI assistant from a tool into a true engineering partner: precise, honest about limitations, and always looking forward to the next improvement.