From CUDA Toolkits to Model Serving: Building a Production ML Environment for GLM-5 Deployment

Introduction

The path from a bare Ubuntu 24.04 server to a production-ready multi-GPU model serving environment is rarely a straight line. It is a winding road through driver compatibility matrices, CUDA toolkit version conflicts, memory-exhausting compilation jobs, and the ever-present challenge of making diverse open-source libraries agree on a common dependency stack. This article examines the opening phase of an opencode coding session that tackled exactly this challenge: building a complete ML environment on a remote Ubuntu machine with multiple NVIDIA RTX PRO 6000 Blackwell GPUs, ultimately deploying the massive GLM-5-NVFP4 model using SGLang.

The work in this chunk spans two distinct but interconnected phases. The first phase is pure infrastructure: installing NVIDIA drivers, configuring CUDA toolkits, setting up a Python virtual environment, and wrestling with the notoriously finicky flash-attn library until it compiles correctly. The second phase is application deployment: upgrading the machine to 8 GPUs and beginning the process of deploying the GLM-5 model. This article synthesizes the key decisions, assumptions, failures, and recoveries that defined this effort.

Phase 1: Building the Foundation

Installing NVIDIA Drivers and CUDA

The session begins with the user requesting a full ML environment setup on a remote Ubuntu 24.04 machine. The assistant's first task is to install the latest NVIDIA drivers and CUDA Toolkit. This is a deceptively complex operation: the wrong driver version can render GPUs unusable, and CUDA toolkit versions must be compatible with both the drivers and the ML frameworks that will be installed later.

The assistant installs NVIDIA drivers version 590.48.01 — a bleeding-edge driver version for the RTX PRO 6000 Blackwell architecture. Alongside this, CUDA Toolkit 13.1 is installed and configured. The environment paths are set up so that nvcc and the CUDA runtime libraries are discoverable by build tools. Verification commands confirm that both GPUs are operational and recognized by the system.

This step establishes a critical assumption: that the latest drivers and CUDA toolkit will provide the best compatibility with the ML frameworks to be installed later. As we will see, this assumption is partially correct — the drivers work flawlessly — but the CUDA version introduces unexpected complications.

Creating the Python Environment with uv

With the GPU infrastructure in place, the assistant creates a Python virtual environment using uv, a fast Python package installer and resolver. The environment is created at /root/ml-env/ and populated with PyTorch and other core ML dependencies. The choice of uv over pip is notable: uv offers significantly faster dependency resolution and installation, which matters when dealing with large packages like PyTorch.

The environment is structured as a standard Python 3.12 virtual environment, with packages installed in /root/ml-env/lib/python3.12/site-packages/. This path becomes the reference point for all subsequent operations in the session.

The flash-attn Saga: A Case Study in Build Complexity

The installation of flash-attn — a library providing fast attention kernels for transformer models — becomes the single most time-consuming and error-prone step in the infrastructure setup. The challenges encountered here illustrate the fragility of building CUDA extensions from source.

The CUDA version mismatch. The system has CUDA Toolkit 13.1 installed, but PyTorch (installed via uv) was compiled against CUDA 12.8. When flash-attn attempts to build its CUDA kernels, it detects the system's CUDA 13.1 and tries to compile against it, but the PyTorch headers and libraries expect CUDA 12.8 APIs. This mismatch causes compilation errors.

The assistant's solution is to install a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation. This is a pragmatic workaround: multiple CUDA toolkits can coexist on the same system, and build tools can be directed to use a specific version via environment variables like CUDA_HOME and PATH. The assistant configures the build to use CUDA 12.8 for flash-attn while leaving CUDA 13.1 as the system default.

The memory exhaustion problem. Even with the correct CUDA version, the build fails repeatedly due to memory exhaustion. The root cause is the MAX_JOBS parameter, which controls how many compilation processes run in parallel. On a machine with many CPU cores, the default or auto-detected value can be extremely high — in this case, 128 parallel jobs. Each compilation process consumes significant memory, and 128 concurrent processes quickly exhaust the available RAM.

The assistant iteratively reduces MAX_JOBS: from 128 to 64, then to 32, then to 20. Each reduction increases build time but decreases memory pressure. The breakthrough comes only after the machine is rebooted with an expanded 432 GB of RAM — a hardware upgrade that makes the reduced parallel compilation feasible.

The vLLM downgrade breakage. After successfully building flash-attn against PyTorch, the assistant installs vLLM — only to discover that vLLM downgrades PyTorch to a different version as part of its own dependency resolution. This breaks the compiled flash-attn binary, which was built against the original PyTorch version. The assistant must rebuild flash-attn against the new PyTorch version (2.9.1) that vLLM requires.

This dependency cascade — where installing one package (vLLM) changes the version of another package (PyTorch), which breaks a third package (flash-attn) that was compiled against the original version — is a classic problem in ML environment management. The assistant's solution is methodical: identify the correct PyTorch version, rebuild flash-attn against it, and verify compatibility.

The Stabilized Stack

After extensive troubleshooting, the environment is stabilized with a fully compatible software stack:

Phase 2: Pivoting to Application Deployment

The Machine Upgrade to 8 GPUs

With the software environment stabilized, the focus shifts to hardware. The machine is upgraded from 2 to 8 RTX PRO 6000 Blackwell GPUs. This is a significant scaling step: each GPU has substantial memory and compute capacity, and 8 GPUs in parallel enable serving models that would be impossible on a single GPU.

The assistant verifies that all 8 GPUs are recognized by the system and accessible via CUDA. This involves checking nvidia-smi output, confirming that the GPUs are in a consistent compute mode, and ensuring that NVLink or other interconnects are properly configured for multi-GPU communication.

Deploying GLM-5-NVFP4 with SGLang

The deployment target is the GLM-5-NVFP4 model, a large language model from the GLM family that uses NVFP4 quantization — a 4-bit floating-point format that NVIDIA has optimized for its latest GPU architectures. The assistant installs a nightly build of SGLang, a high-performance inference engine designed for serving large language models on multiple GPUs.

The deployment process involves:

  1. Downloading the model weights — the GLM-5-NVFP4 model is large, requiring significant download time and disk space.
  2. Configuring SGLang — setting tensor parallelism across the 8 GPUs, configuring memory allocation, and tuning batch sizes.
  3. Loading the model — SGLang must parse the model architecture, allocate GPU memory, and load the quantized weights.
  4. Performance tuning — adjusting parameters like maximum sequence length, cache sizes, and scheduling policies to optimize throughput and latency.
  5. Load testing — sending sample requests to verify that the model produces correct outputs and measuring performance metrics. The assistant begins this process, positioning the environment for the next phase of work: actually running the model and validating its outputs.

Key Themes and Lessons

The Fragility of CUDA Builds

The flash-attn installation saga illustrates a fundamental challenge in the ML ecosystem: building CUDA extensions is fragile. The combination of CUDA toolkit version, PyTorch version, compiler flags, and available memory must all align perfectly. A single mismatch — whether it's the CUDA version, the number of parallel jobs, or a dependency version change — can cause the build to fail.

The assistant's approach to this fragility is instructive: it does not assume that the first attempt will succeed. Instead, it treats build failures as diagnostic signals, iteratively adjusting parameters until the build succeeds. This trial-and-error approach, while time-consuming, is often the only reliable way to resolve CUDA build issues.

The Dependency Cascade Problem

The vLLM-induced PyTorch downgrade that broke flash-attn is a textbook example of the dependency cascade problem. In a complex ML environment, packages have overlapping and sometimes conflicting dependency requirements. Installing one package can change the version of a shared dependency, breaking other packages that were compiled against the original version.

The solution — rebuilding flash-attn against the new PyTorch version — is conceptually simple but operationally expensive. Each rebuild takes significant time and computational resources. The lesson is that environment management tools (like uv, conda, or pip) can resolve dependency versions but cannot prevent the need for recompilation when shared libraries change.

The Value of Methodical Troubleshooting

Throughout this chunk, the assistant demonstrates a methodical approach to troubleshooting. When flash-attn fails to build, it does not try random fixes. Instead, it:

  1. Diagnoses the error — identifies whether the failure is due to CUDA version mismatch, memory exhaustion, or something else.
  2. Formulates a hypothesis — e.g., "reducing parallel compilation jobs will reduce memory pressure."
  3. Tests the hypothesis — changes MAX_JOBS and rebuilds.
  4. Iterates — if the fix doesn't work, adjusts the parameters and tries again. This systematic approach, combined with the ability to adapt when assumptions prove incorrect, is what ultimately enables the successful environment setup.

The Infrastructure-to-Deployment Transition

The chunk's arc — from bare-metal driver installation to model deployment — reflects the full lifecycle of ML infrastructure work. The early steps (drivers, CUDA, Python environment) are generic and reusable across many projects. The middle steps (flash-attn compilation, vLLM installation) are specific to the chosen model and inference engine. The final steps (model download, SGLang configuration) are deployment-specific.

Each layer builds on the previous one, and failures at any layer can block progress. The assistant's ability to work through all layers — from hardware configuration to application deployment — is essential to the session's success.

Conclusion

The work in this chunk represents the foundational phase of a complex ML deployment effort. From installing NVIDIA drivers and resolving CUDA toolkit conflicts, to wrestling with flash-attn's build system and ultimately deploying the GLM-5 model on 8 GPUs, the assistant navigates a landscape of technical challenges that are characteristic of modern ML infrastructure work.

The key lessons — the fragility of CUDA builds, the dependency cascade problem, the value of methodical troubleshooting, and the layered nature of infrastructure work — are applicable far beyond this specific session. They represent the practical knowledge that ML engineers develop through experience: that building a production environment is rarely a linear process, and that success depends on the ability to diagnose, adapt, and persist through multiple rounds of failure and recovery.

The stage is now set for the next phase: actually running the GLM-5 model, validating its outputs, and tuning its performance for production use. The infrastructure foundation, hard-won through the efforts described in this chunk, makes that next phase possible.