Building an ML Server from Scratch: Drivers, CUDA Hell, and the Flash-Attention Gauntlet

Introduction

The opencode session captured in Segment 0 is a remarkable case study in the full lifecycle of ML infrastructure work. It begins with a terse twelve-word user request — "Install latest nvidia drivers, cuda 13, python/uv/venv/etc ML tooling" on a remote Ubuntu 24.04 machine — and unfolds into a multi-hour odyssey touching nearly every layer of modern ML deployment: hardware driver installation, CUDA toolkit management, Python environment configuration, compiled CUDA extension debugging, dependency resolution warfare, and finally, model serving with performance optimization.

This article synthesizes the entire segment, tracing the narrative arc from bare metal through infrastructure stabilization to the pivot into production deployment. Along the way, it examines the key challenges — dual CUDA toolkit management, parallel compilation memory exhaustion, silent ABI-breaking dependency downgrades, and the transition from infrastructure to application — and distills the lessons learned for anyone building GPU-accelerated ML environments.

Phase 1: Laying the Foundation — Drivers, CUDA, and Python

Assessing the Machine

The session opens with the user providing an SSH target (10.1.230.175) and a high-level specification of what needs to be built ([msg 0]). The assistant's first move is to assess the machine — discovering it runs Ubuntu 24.04.4 LTS, has 128 CPU cores, and is named llm-one — a clear signal of its intended purpose as a large language model server. The machine initially has no NVIDIA drivers installed, and the assistant must build the entire stack from scratch.

The driver installation proceeds through several phases. The assistant adds the NVIDIA CUDA repository, installs the NVIDIA driver package (version 590.48.01), and then discovers something surprising: CUDA Toolkit 13.1 is actually available, contradicting its earlier assumption that "CUDA 13 doesn't exist yet." This moment of discovery — when the assistant's static knowledge is corrected by live system interrogation — is a recurring theme throughout the session. The assistant installs CUDA 13.1, configures environment paths, and verifies that two RTX PRO 6000 Blackwell GPUs are operational.

Python Environment with uv

The Python environment is set up using uv, a modern Rust-based Python package manager that the user specifically requested. The assistant creates a virtual environment at ~/ml-env and installs PyTorch. At this point, the foundation appears solid: NVIDIA drivers, CUDA 13.1, Python 3.12 with uv, and PyTorch with GPU support. The machine is ready for ML tooling.

But the foundation has a hidden crack. PyTorch's official wheels are compiled against CUDA 12.8, not the system's CUDA 13.1. This discrepancy will soon become the source of cascading failures.

Phase 2: The Flash-Attention Crucible

The installation of flash-attn — a CUDA-accelerated attention kernel library — transforms the session from a routine setup into a debugging epic. Flash-attn is a critical dependency for many modern LLM serving frameworks (including vLLM and SGLang), providing the fused attention kernels that make inference efficient on modern GPUs. Building it from source on a new system exposes the full complexity of the CUDA compilation toolchain.

The CUDA Version Mismatch

The first attempt at a bulk package install fails because flash-attn's build system, which uses PyTorch's cpp_extension module, performs a strict CUDA version check. The system's CUDA 13.1 does not match PyTorch's CUDA 12.8 base, and the build refuses to proceed.

The assistant installs a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation, setting CUDA_HOME to point at the older version during extension builds. This dual-CUDA workaround resolves the version mismatch, but it introduces a split personality in the environment — the system default remains CUDA 13.1, but extension builds use CUDA 12.8. This arrangement is fragile but functional, and it's a common pattern in ML environments where the system CUDA and PyTorch's expected CUDA diverge.

The Memory Wall

With the CUDA version issue resolved, the assistant launches the flash-attn build — and immediately hits a second wall: memory exhaustion. The machine has 128 CPU cores, and flash-attn's build system (using Ninja) spawns up to 128 parallel nvcc and ptxas processes. Each CUDA compiler process consumes gigabytes of RAM, and the system quickly runs out of memory, becoming unresponsive ([msg 35]).

What follows is a remarkable collaborative debugging session between the user and the assistant. The user provides terse, precise guidance — "oom, go 32" ([msg 40]), "oom, go 20" ([msg 56]) — while the assistant executes the builds, cleans up after failures, and reports results. The MAX_JOBS parameter is systematically reduced: 128 → 64 → 48 → 32 → 20. Each attempt requires killing lingering compiler processes, dropping kernel caches, and verifying that memory has been fully reclaimed before the next try.

The machine is rebooted with expanded RAM — from 288GB to 432GB — yet even then, builds at MAX_JOBS=32 consume 84GB before being killed. Only at MAX_JOBS=20 does the build finally succeed, taking nearly 10 minutes of compilation ([msg 60]). The assistant also optimizes the build by setting TORCH_CUDA_ARCH_LIST="10.0", targeting only the Blackwell GPU architecture (sm_100) rather than compiling for multiple architectures, which reduces both build time and memory pressure.

Why CUDA Compilation Is So Memory-Hungry

This phase is a masterclass in CUDA build memory management. The key insight is that CUDA kernel compilation is not like regular C++ compilation. Each ptxas invocation loads the entire PTX code for a kernel variant and performs optimization and register allocation, consuming gigabytes of memory per job. Flash-attn generates dozens of kernel variants (for different head dimensions, sequence lengths, data types, and attention mask configurations), and each variant goes through the full compilation pipeline.

The memory consumption is super-linear in the number of parallel jobs because each nvcc process spawns multiple sub-processes (device code compilation, host code compilation, linking), and the peak memory usage occurs when many of these sub-processes overlap. The only reliable way to determine the safe parallelism level is through empirical testing, which is exactly what the assistant conducted. The final value of MAX_JOBS=20 on a 128-core machine with 432GB of RAM is a striking data point: CUDA compilation requires roughly 21.6 GB of RAM per parallel job on this system, or about 6.5× the per-core memory ratio that typical CPU compilation would need.

Phase 3: Dependency Hell — When Package Managers Fight Back

The flash-attn build succeeds, and the assistant verifies that PyTorch 2.10.0, flash-attn 2.8.3, and two RTX PRO 6000 GPUs are all operational. The remaining ML packages — transformers, vLLM, bitsandbytes, and others — are installed in a separate command.

The Silent Catastrophe

And then everything breaks.

The verification script in message 64 reveals a silent catastrophe: flash-attn's CUDA extension fails to load with an ImportError on the compiled .so file ([msg 64]). The cause is insidious. vLLM 0.15.1 has a dependency constraint that requires PyTorch ≤ 2.9.x, and the package resolver silently downgraded PyTorch from 2.10.0 to 2.9.1. Flash-attn's compiled binary was built against PyTorch 2.10's C++ ABI, and the downgrade rendered it incompatible. The Python-level imports all succeed — PyTorch works, CUDA is detected — but the C++ extension cannot load because the symbols it expects from PyTorch's runtime are different ([msg 65]).

This is a particularly dangerous failure mode because it is silent. The package manager (uv/pip) treats packages as independent units and has no awareness that a compiled CUDA extension is ABI-locked to a specific PyTorch build. The downgrade happens without any warning or error. The verification script — a comprehensive import test of every package in the stack — is the only reason the problem was detected before runtime.

The Rebuild Loop

The assistant enters a new phase of debugging: rebuilding flash-attn against the correct PyTorch version. But the dependency resolver keeps fighting back. When the assistant pins PyTorch at 2.10.0 and rebuilds flash-attn, the torchvision package forces a downgrade back to 2.9.1 ([msg 66]). Each rebuild takes ~10 minutes. The assistant must uninstall flash-attn, clean cached build artifacts, pin PyTorch, install vLLM first (which sets the PyTorch version), and then rebuild flash-attn against that pinned version.

The final successful build in message 75 is almost anticlimactic — it completes in 4.39 seconds because uv reuses cached build artifacts from a previous compilation ([msg 75]). The assistant recognizes the anomaly, questions it, and verifies that the cached binary is compatible with the current PyTorch 2.9.1. The verification command python3 -c "import flash_attn; print(flash_attn.__version__)" returns 2.8.3 — a clean, unambiguous success.

The ABI Problem in Context

The PyTorch ABI (Application Binary Interface) problem is a fundamental challenge in the Python ML ecosystem. Python packages are typically distributed as pure Python (.py files) or as precompiled wheels (.so files). Pure Python packages are version-flexible — any compatible Python interpreter can load them. But compiled extensions are linked against specific C++ ABIs, and if the host package (like PyTorch) changes its ABI, the extension silently breaks.

PyTorch uses C++17 features and has a complex C++ API that changes between versions. The cpp_extension module compiles extensions against the installed PyTorch headers, embedding ABI-specific symbols. When PyTorch is downgraded, the runtime loader cannot find those symbols, producing the ImportError seen in message 64. The error message — a long path to a .so file followed by a platform-specific dynamic linker error — is the only signal that something went wrong.

The final verified environment includes PyTorch 2.9.1+cu128, flash-attn 2.8.3 (built for Blackwell sm_100), vLLM 0.15.1, transformers 4.57.6, and a dozen other packages — all working together ([msg 77]). The assistant marks all four initial high-priority todos as completed, signaling that the infrastructure phase is done.

Phase 4: The Pivot to Model Deployment

The user's next message pivots the session sharply: "Added 8 GPUs; Deploy glm-5 nvfp4 — probably requires main/nightly sglang; After running tune the params and allow for more parallel queries; ./ iirc has some load testing tool" ([msg 81]).

This message transforms the conversation. The machine has been upgraded from 2 to 8 GPUs — a fourfold increase in compute capacity. The focus shifts from building infrastructure to deploying a specific model: GLM-5-NVFP4, a large language model in NVIDIA's 4-bit floating-point quantization format, optimized for Blackwell architecture. The serving framework is SGLang, specifically a nightly build that supports the NVFP4 format.

What Is NVFP4?

NVFP4 (NVIDIA 4-bit Floating Point) is a quantization format introduced with the Blackwell architecture. It stores model weights in 4-bit floating-point values, achieving a 4× reduction in memory footprint compared to FP16, while maintaining better accuracy than traditional integer quantization. The GLM-5-NVFP4 model is a quantized version of the GLM-5 model, a large bilingual (Chinese/English) language model developed by Tsinghua University and Zhipu AI.

Deploying this model requires SGLang — a fast serving framework for LLMs that supports tensor parallelism across multiple GPUs, continuous batching, and various quantization formats. The nightly build is needed because NVFP4 support is a recent addition that hasn't yet made it into a stable release.

The Deployment Challenge

The user's message contains several uncertainties — "probably requires main/nightly sglang" and "iirc has some load testing tool" — that the assistant must resolve through exploration and experimentation. The performance requirements are explicit: tune parameters for parallel query throughput and validate with load testing. This is a production-grade deployment task, not a development experiment.

The deployment involves several steps:

Downloading the model from Hugging Face — the GLM-5-NVFP4 model is likely hundreds of gigabytes, requiring significant download time and disk space.
Installing the SGLang nightly build — this requires identifying the correct nightly wheel or building from source, and ensuring compatibility with the existing CUDA and PyTorch stack.
Configuring tensor parallelism across 8 GPUs — SGLang must be configured to shard the model across all available GPUs, with appropriate communication settings for NVLink or PCIe interconnects.
Tuning inference parameters — batch size, scheduling policy, max sequence length, and other parameters must be optimized for throughput and latency.
Load testing — the user mentions a load testing tool in the home directory, which the assistant must locate, configure, and run to validate performance.

The Infrastructure-to-Application Transition

The pivot from infrastructure to application is a critical moment in any ML project. Infrastructure work — driver installation, CUDA configuration, package management — is about building a stable foundation. Application work — model deployment, inference optimization, load testing — is about using that foundation to deliver value.

The transition is rarely clean. Infrastructure issues often bleed into application work (as seen with the flash-attn ABI problem), and application requirements often force infrastructure changes (as seen with the dual CUDA toolkit setup). The assistant's systematic approach to both phases — documenting each step, verifying at each milestone, and cleaning up after failures — provides a model for managing this transition effectively.

Themes and Lessons

The Fragility of CUDA Extension Builds

Building flash-attn from source exposed the full complexity of the CUDA compilation toolchain. Memory management during parallel compilation, ABI compatibility across PyTorch versions, and the interaction between system CUDA and PyTorch's expected CUDA version are all sources of failure that have no simple solution. The only reliable approach is systematic empirical testing — reducing MAX_JOBS, cleaning caches, and verifying at each step.

For teams building ML infrastructure, the lesson is clear: budget significant time for CUDA extension compilation, especially on new hardware architectures. The combination of a new GPU architecture (Blackwell), a new CUDA version (13.1), and a complex extension (flash-attn) creates a perfect storm of compatibility issues that cannot be resolved through documentation alone.

The Silent Dependency Problem

The most dangerous failures in ML environment management are the ones that don't produce errors during installation. The PyTorch downgrade caused by vLLM's dependency constraints was invisible until runtime. This highlights a fundamental limitation of current package managers: they track Python-level dependencies but have no concept of C++ ABI compatibility between compiled extensions and their host packages.

The solution is comprehensive verification. The assistant's verification script — importing every package in the stack and printing version numbers — was the tool that repeatedly revealed hidden problems. Without it, the silent PyTorch downgrade would have gone undetected until runtime, causing far more confusing failures.

The Collaborative Debugging Pattern

The user and assistant worked together in a tight feedback loop. The user provided terse, precise guidance ("oom, go 32," "oom, go 20") based on system-level observations, while the assistant executed the builds, cleaned up after failures, and reported results. This pattern — human strategic oversight combined with AI tactical execution — proved highly effective for navigating complex infrastructure problems.

The collaboration was asymmetric but complementary. The user had system-level awareness (knowing when the machine was OOMing, knowing to expand RAM) and domain knowledge (knowing that MAX_JOBS could be tuned). The assistant had execution capability (running commands, parsing output, cleaning up state) and persistence (repeating the build cycle a dozen times without fatigue). Together, they solved a problem that neither could have solved alone in the same timeframe.

The Importance of Cleanup

A recurring pattern in the session is the assistant's meticulous cleanup between build attempts. After each failed build, the assistant kills lingering processes, drops kernel caches, and verifies memory is fully reclaimed before proceeding. This discipline is essential because CUDA compilation failures often leave behind zombie processes that consume memory and file handles, making subsequent attempts more likely to fail.

The cleanup commands — pkill -9 nvcc, pkill -9 ptxas, sync && echo 3 > /proc/sys/vm/drop_caches — are not glamorous, but they are the difference between a system that recovers from failure and one that spirals into progressively worse states.

The Arc from Infrastructure to Application

The session's arc — from bare-metal driver installation through CUDA toolkit management, Python environment setup, compiled extension debugging, dependency resolution, and finally model deployment — mirrors the full lifecycle of ML infrastructure work. Each phase builds on the previous, and each phase has its own failure modes and debugging techniques.

The infrastructure phase (messages 0-80) consumed the bulk of the session's time and effort. This is typical: building a stable ML environment from scratch on a new system is a multi-hour endeavor, even for experienced practitioners. The application phase (messages 81+) is where the value is delivered, but it depends entirely on the quality of the infrastructure foundation.

Conclusion

The opencode session captured in Segment 0 is a microcosm of modern ML infrastructure work. It demonstrates that setting up an ML environment is never a straightforward installation process — it is an iterative, collaborative journey through multiple layers of software complexity.

The assistant's systematic approach to debugging — forming hypotheses, testing them, cleaning up after failures, and verifying results — combined with the user's strategic guidance, produced a working environment that could serve a production model across 8 GPUs. The session stands as a testament to the value of methodical debugging, the importance of comprehensive verification, and the power of human-AI collaboration in tackling complex systems problems.

For anyone building GPU-accelerated ML environments, the lessons from this session are clear: expect CUDA compilation to be memory-intensive, verify your environment comprehensively after every change, budget for dependency resolution surprises, and never underestimate the value of a good cleanup routine. The path from bare metal to model serving is never a straight line — but with systematic debugging and collaborative problem-solving, it is a navigable one.## References

[1] From Bare Metal to Model Serving: The Complete Arc of an ML Infrastructure Build — the companion article covering the broader chunk context, including the initial driver installation and the full message sequence from messages 0-83.