Segment 0
This sub-session covered the complete setup of an ML development environment on Ubuntu 24.04, starting with installation of NVIDIA drivers (590.48.01) and CUDA Toolkit 13.1, verification of two RTX PRO 6000 Blackwell GPUs, and creation of a Python virtual environment with PyTorch using uv. A major effort was spent resolving flash-attn installation issues, which required installing a secondary CUDA 12.8 toolkit, reducing parallel compilation jobs (MAX_JOBS) from 128 to 20 to avoid memory exhaustion, and later rebuilding flash-attn against the correct PyTorch version (2.9.1) after vLLM downgraded it. The environment was stabilized with a compatible stack including PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1. The session then shifted to application deployment: the machine was upgraded to 8 GPUs, and the assistant was tasked with deploying the GLM-5-NVFP4 model using a nightly build of SGLang, followed by performance tuning and load testing.
Building an ML Server from Scratch: Drivers, CUDA Hell, and the Flash-Attention Gauntlet