Segment 42

This sub-session focused on deploying the Qwen3.5-122B-A10B-FP8 model across two NVIDIA GB10 DGX Spark systems connected via InfiniBand RoCE. The assistant explored both nodes, freed GPU memory by stopping an existing GLM-4.7-Flash container, downloaded the 119GB FP8 model from HuggingFace, and rsynced it to the second Spark. After SGLang's spark image proved incompatible and its multi-node NCCL hung, the assistant pivoted to a dedicated vLLM image (hellohal2064/vllm-qwen3.5-gb10) and built a Ray-based multi-node deployment. Key challenges included forcing Ray to use the IB subnet IP for cross-node communication, disabling the memory monitor OOM killer during CUDA graph capture, and configuring NCCL over NET/IBext_v11 for inter-node tensor parallelism. The final deployment achieved ~27 tok/s single-request throughput with correct reasoning output and tool calling enabled. The session concluded with a clean restart of the services after removing old containers, confirming the model loaded successfully and served queries with proper reasoning field extraction.

Deploy Qwen3.5-122B-FP8 on dual DGX SparkFree GPU memory by stopping old GLM containerDownload and distribute 119GB FP8 model across nodesPivot from SGLang to vLLM for multi-node supportFix Ray multi-node networking with IB subnet IPWork around Ray OOM killer during CUDA graph captureConfigure NCCL over InfiniBand for inter-node TPVerify model serving with correct reasoning output

The Dual DGX Spark Deployment: From Reconnaissance to Production Inference on Blackwell ARM 4859 words

Chunks