From Bare Metal to Model Serving: Building a Production ML Environment on Ubuntu 24.04

Introduction

Setting up a production-grade machine learning inference environment is rarely a straightforward affair. When the target hardware includes cutting-edge NVIDIA Blackwell GPUs, the software stack involves bleeding-edge CUDA toolkits, and the model requires specialized attention kernels like flash-attn, the process becomes a delicate orchestration of version matching, build configuration, and iterative troubleshooting. This article examines the first chunk of an opencode coding session (segment 72, chunk 0) that documents exactly such a journey: the complete setup of an ML environment on Ubuntu 24.04, from bare-metal driver installation to a working model deployment with SGLang.

The chunk captures the full arc of infrastructure work: installing NVIDIA drivers and CUDA Toolkit 13.1, creating a Python virtual environment with uv, resolving the notoriously difficult flash-attn build process across multiple rounds of debugging, stabilizing a compatible dependency stack, and ultimately pivoting to deploy the GLM-5-NVFP4 model on an 8-GPU machine using a nightly build of SGLang. The work is characterized by methodical troubleshooting, deep system-level knowledge, and a willingness to iterate through failed builds until a working configuration is found.

The Starting Point: A Fresh Ubuntu 24.04 Machine

The session opens with the user requesting a full ML environment setup on a remote Ubuntu 24.04 machine equipped with NVIDIA RTX PRO 6000 Blackwell GPUs. This is not a trivial request. Ubuntu 24.04 is a relatively new release, and the Blackwell architecture represents NVIDIA's latest GPU generation, meaning that driver and CUDA support may not be as mature as for previous generations. The user's requirements are specific: the latest NVIDIA drivers, CUDA Toolkit 13.1, a Python virtual environment managed by uv, and core packages including PyTorch, flash-attn, vLLM, and the SGLang inference framework.

The assistant begins by establishing connectivity and verifying the baseline system state. The machine is accessible via SSH, and the assistant immediately begins the process of installing the NVIDIA driver stack. This involves downloading the NVIDIA driver version 590.48.01, which is the latest driver compatible with the Blackwell architecture at the time of the session. The installation requires navigating the standard Linux driver installation流程: disabling the open-source nouveau driver, entering runlevel 3 to avoid X server conflicts, running the NVIDIA installer, and rebooting to activate the new driver.

Installing CUDA Toolkit 13.1 and Verifying the GPUs

With the drivers in place, the assistant proceeds to install CUDA Toolkit 13.1. This is a critical step because the CUDA toolkit provides the compiler (nvcc), runtime libraries, and development headers needed to build GPU-accelerated libraries like flash-attn and PyTorch. The installation is performed using NVIDIA's runfile installer, which provides maximum control over the installation path and avoids conflicts with system package managers.

After installation, the assistant configures the environment paths — adding CUDA to PATH, LD_LIBRARY_PATH, and CUDA_HOME — and verifies that the GPUs are operational. The verification confirms two RTX PRO 6000 Blackwell GPUs, each with substantial memory and compute capability. The nvidia-smi output shows the GPUs are recognized, the driver version is correct, and the CUDA toolkit version is properly linked.

This phase of the work is straightforward but essential. Without correct driver and CUDA installation, nothing else in the stack can function. The assistant's methodical approach — install, configure, verify — establishes a solid foundation for the more complex work ahead.

Creating the Python Environment with uv

With the GPU stack verified, the assistant creates a Python virtual environment using uv, a modern Python package manager that offers significantly faster dependency resolution than traditional pip. The use of uv is notable because it reflects a preference for modern tooling that can handle complex dependency graphs more efficiently — a practical concern when dealing with the intricate dependency chains of ML frameworks.

The assistant installs PyTorch into this environment, selecting a version compatible with CUDA 13.1. This is where the first signs of complexity emerge: PyTorch's pre-built binaries are compiled against specific CUDA versions, and CUDA 13.1 is new enough that compatibility must be verified. The assistant navigates this by selecting the appropriate PyTorch build and confirming that it can detect the GPUs and perform basic tensor operations.

The Flash-Attn Bottleneck: A Case Study in Build Engineering

The installation of flash-attn — a high-performance attention kernel that is critical for many modern LLM architectures — becomes the major bottleneck of the setup process. Flash-attn must be compiled from source against the specific CUDA toolkit and PyTorch versions installed on the system, and this compilation process reveals a series of interconnected issues that require multiple rounds of debugging to resolve.

The CUDA Version Conflict

The first issue is a CUDA version mismatch. PyTorch's pre-built binaries are compiled against CUDA 12.8, but the system has CUDA 13.1 installed. When flash-attn's build process detects CUDA 13.1, it attempts to use headers and libraries from that version, which may be incompatible with the PyTorch binaries that expect CUDA 12.8. This creates a conflict: flash-attn's compiled kernels need to be compatible with both the CUDA runtime (13.1) and the PyTorch binaries (built against 12.8).

The assistant's solution is pragmatic: install a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation. This allows flash-attn to be compiled against CUDA 12.8 (matching PyTorch's expectations) while the system retains CUDA 13.1 for other purposes. The assistant configures the build to point at the CUDA 12.8 installation, resolving the header and library conflicts.

The Memory Exhaustion Problem

The second issue is more fundamental: the flash-attn build process exhausts system memory. The machine has significant RAM, but flash-attn's compilation uses parallel jobs to speed up the build, and each compilation job consumes substantial memory. With the default setting of MAX_JOBS=128 (matching the number of CPU cores), the build process spawns 128 simultaneous compiler instances, each compiling a different kernel variant. The aggregate memory consumption exceeds available RAM, causing the build to fail with out-of-memory errors.

The assistant iteratively reduces MAX_JOBS through trial and error: first to 64, then 32, then 24, and finally to 20. Each reduction increases build time but decreases peak memory usage. The breakthrough comes when the machine is rebooted with an expanded 432GB of RAM, which provides enough headroom for the compilation to complete even with a moderate number of parallel jobs. This highlights a key insight about building large CUDA extension packages: the memory requirement scales with parallelism, and finding the right balance between build speed and memory pressure is essential.

The vLLM Dependency Downgrade

The third issue emerges after flash-attn is successfully compiled. When vLLM is installed into the environment, it downgrades PyTorch from the version that flash-attn was compiled against. This breaks the flash-attn binary, which was compiled against a specific PyTorch ABI (Application Binary Interface). The mismatch causes runtime errors when flash-attn attempts to interact with PyTorch tensor objects.

The assistant resolves this by rebuilding flash-attn against the correct PyTorch version (2.9.1) after vLLM has finished its dependency resolution. This ensures that flash-attn's compiled kernels match the PyTorch runtime that vLLM expects. The sequence matters: install PyTorch, install vLLM (which may adjust PyTorch), then rebuild flash-attn against the final PyTorch version.

The Stabilized Stack

After navigating these issues, the environment is stabilized with a fully compatible stack:

The Pivot to Model Deployment

With the environment stabilized, the conversation pivots sharply from infrastructure setup to application deployment. The machine is upgraded from 2 to 8 GPUs, dramatically increasing its inference capacity. The user tasks the assistant with deploying the GLM-5-NVFP4 model — a variant of the GLM architecture that uses NVFP4 (NVIDIA 4-bit floating point) quantization — using a nightly build of SGLang.

This pivot represents a shift in focus from "can the system run" to "can the system serve a model efficiently." The nightly build of SGLang is chosen because it may include optimizations or features not yet available in the stable release, particularly for the NVFP4 quantization format and the Blackwell GPU architecture.

The user's instructions for this phase include not just deployment but also performance tuning and load testing. This suggests that the environment setup is not an end in itself but a foundation for ongoing optimization work. The assistant is expected to ensure that the model serves requests correctly, achieves acceptable latency and throughput, and can handle production-scale load.

Themes and Patterns

Several themes emerge from this chunk of work:

Iterative Troubleshooting

The flash-attn build process exemplifies the iterative nature of infrastructure work. Each attempt reveals new information: the CUDA version conflict, the memory pressure issue, the PyTorch downgrade problem. The assistant responds to each failure by adjusting parameters, installing additional toolkits, or changing the build sequence. There is no single "correct" configuration that works on the first try — the working configuration is discovered through a process of elimination and refinement.

System-Level Knowledge

The assistant demonstrates deep knowledge of the Linux system administration and CUDA ecosystem. Understanding how to install NVIDIA drivers without breaking the display system, how to manage multiple CUDA toolkit installations, how to configure environment variables for build tools, and how to diagnose memory pressure during compilation — these are skills that come from experience with GPU-accelerated systems.

Dependency Chain Management

The version conflicts between PyTorch, flash-attn, and vLLM illustrate the fragility of ML software stacks. Each package has specific version requirements for its dependencies, and the transitive closure of these requirements can create conflicts that are difficult to resolve. The assistant's approach — install in a specific order, rebuild when dependencies change, and verify compatibility at each step — is a practical strategy for managing this complexity.

The Infrastructure-to-Application Pipeline

The chunk traces a complete arc from bare-metal setup to model deployment. This is a common pattern in ML engineering: the infrastructure work (drivers, CUDA, Python environment, compiled extensions) must be completed before any application work (model deployment, performance tuning) can begin. The chunk captures both phases, showing how they connect and how the quality of the infrastructure work determines the feasibility of the application work.

Conclusion

This chunk of the opencode session documents a complete ML environment setup journey on Ubuntu 24.04 with NVIDIA Blackwell GPUs. The work spans driver installation, CUDA toolkit configuration, Python environment creation, and the notoriously difficult flash-attn build process, culminating in a stabilized stack and the beginning of model deployment work. The process is characterized by methodical troubleshooting, deep system-level knowledge, and a willingness to iterate through multiple build attempts to find a working configuration.

The flash-attn build saga, in particular, serves as a case study in the challenges of building CUDA extensions in heterogeneous environments. The CUDA version conflict, memory pressure during compilation, and dependency downgrade issues are all common problems that ML engineers face when setting up production systems. The assistant's approach to resolving these issues — install secondary toolkits, reduce parallel compilation jobs, rebuild after dependency changes — provides a template for handling similar situations.

With the environment stabilized and the model deployed, the session is positioned for the next phase of work: performance tuning, load testing, and the deep debugging of inference issues that will occupy the remainder of segment 72. The infrastructure foundation laid in this chunk makes all subsequent work possible.## The Pivot to Debugging: CUDA Graph Capture Corruption

The infrastructure setup, while essential, was merely the prelude to the session's main work. Once the environment was stable and the GLM-5-NVFP4 model was deployed, the focus shifted dramatically to a deep, multi-week investigation of a persistent corruption bug in the inference pipeline. This bug — a 17% output corruption rate specifically when using bf16 (brain floating point 16-bit) index keys under CUDA graph capture during decode — became the central technical challenge of the session.

The bug exhibited a distinctive fingerprint that made it both maddening and illuminating. It appeared only under a specific combination of conditions: bf16 precision combined with CUDA graph capture. Under eager execution (no graph capture), bf16 produced 0% corruption. Under graph capture with fp8 precision, corruption was also 0%. Only the intersection of bf16 and captured graphs produced the 17% corruption rate. This pattern — clean in eager, clean in fp8/captured, broken in bf16/captured — provided a crucial diagnostic signature that the assistant would use throughout the investigation.

The prime hypothesis, articulated in extraordinary detail by the user in a subagent task specification (see [msg 0]), was that the corruption stemmed from non-persistent input temporaries created by the Triton kernel wrapper function. The function bf16_paged_mqa_logits_triton_sm120 performs several .to(), .contiguous(), and .reshape() calls on its inputs before launching the Triton kernel. The hypothesis was that these conversions create temporary tensors whose storage addresses are snapshotted during CUDA graph capture, and on replay, the kernel reads from those stale addresses rather than from fresh data.

The assistant was tasked with five specific sub-tasks: trace the exact dtype and contiguity of each input to determine which conversions actually allocate new memory; analyze the Triton kernel to understand how it consumes inputs; confirm the capture hazard reasoning is consistent with all empirical measurements; investigate whether the match_num_queries helper function also needs fixing; and propose a minimal, capture-safe patch.

The Investigation Unfolds

The assistant began by reading the local copy of the indexer file (dsv4_indexer.py) and establishing connectivity to the remote server to explore the full codebase structure ([msg 1], [msg 2]). This initial groundwork — understanding the module structure, identifying the relevant files, and planning the investigation strategy — was essential before any deep analysis could begin.

Over the course of dozens of messages, the assistant systematically traced every input tensor back to its source allocation, verified dtypes and contiguity at every intermediate step, and built a comprehensive picture of the data flow through the indexer. The investigation involved reading metadata files, examining the CUDA graph runner code, analyzing git history for relevant commits, and deploying canary instrumentation to detect unexpected memory writes during graph replay.

A pivotal moment came when the assistant confronted a fundamental contradiction between its analysis and the empirical data ([msg 43]). After tracing all five input conversions, the assistant concluded that every single one was a no-op in the live configuration — none allocated new memory. The .to(int32) calls on already-int32 tensors returned the same object. The .contiguous() calls on already-contiguous tensors returned the same object. The .reshape() calls on already-shaped tensors returned views sharing storage. The prime hypothesis — that stale temporaries caused the corruption — could not explain the bug because no temporaries were being created.

This contradiction forced a deeper investigation. The assistant considered and systematically eliminated alternative hypotheses: a sizing bug in the memory pool (already fixed in the current checkout), Triton kernel re-specialization during capture, breakable graph path metadata refresh issues, multi-stream execution hazards, and memory aliasing between layers. Each hypothesis was tested against the empirical data and either refined or discarded.

The Breakthrough: Multi-Stream Overlap Race

The decisive evidence came from a differential test: disabling SGLANG_OPT_USE_MULTI_STREAM_OVERLAP completely eliminated the corruption, achieving 0% across 80-session stress tests compared to a 15-18% baseline. The root cause was identified as a multi-stream-overlap race condition: the C4 sparse indexer runs on an alternate CUDA stream under capture, and its bf16 read-path transient intermediates alias or race with main-stream tensors in the shared captured-graph memory pool.

The fix was remarkably simple — a single environment variable, requiring zero code changes. The corruption was definitively resolved by disabling the multi-stream overlap feature, which prevented the race condition from occurring. This finding was confirmed on a clean git build and thoroughly documented in the project report.

From Debugging to Performance Optimization

With the corruption fixed, the session pivoted once again — this time from debugging to performance optimization. The user's next priority was improving decode throughput scaling from C60 to C90. The assistant created a comprehensive project plan based on live bottleneck analysis, which revealed that decode was latency- and occupancy-bound, with a step time formula of approximately 18ms + 1.05ms per request.

The assistant rigorously investigated several optimization strategies. Two-Batch Overlap (TBO) was evaluated through parallel subagent analysis and determined to be a definitive no-go due to hard architectural blockers: it requires Expert Parallelism, the DSV4 decoder layers are not wired for it, and it targets EP all-to-all communication rather than the TP all-reduce which accounts for only ~4% of step time. This analysis saved significant engineering effort that would have been wasted on an infeasible approach.

The assistant then A/B tested the top-ranked lever: re-enabling the overlap scheduler on the decode worker. This showed a modest but real throughput improvement of approximately 5-7% at high concurrency and passed basic serial wedge tests. However, an aggressive stress test involving abort cascades triggered a structural desync hazard, confirming that the underlying bug was still present. The assistant committed to implementing the correct fix — an unconditional "agree-or-defer" all_reduce barrier on the TP CPU group — to safely ship the throughput gain while prioritizing correctness above all else.## Production Incident Response

The session also included a real production incident that tested the assistant's diagnostic skills under pressure. After a restart, under real agentic load (~100s of requests with tool rounds), some requests became stuck while others flowed normally. The user suspected the overlap-schedule change that had been recently deployed, but the assistant systematically refuted that hypothesis through live evidence: --disable-overlap-schedule was confirmed active, decode GPUs were idle (0% utilization, 165W power) rather than spinning on a collective, and the NIXL abort-fix was present.

The decisive clue was that prefill had run continuously for 17 hours while decode had been restarted multiple times that day for A/B testing. This degraded the prefill-to-decode NIXL bootstrap state, causing silent transfer failures (21 on prefill vs 0 on decode). Requests were stuck in preflight, timing out, while decode sat idle with no error logs. The fix was a full clean co-restart of the PD pair (prefill → decode → router), which re-established the bootstrap from scratch.

Post-fix verification showed 8/8 sequential requests completing at ~0.59s, an agentic repro with 30/30 sessions clean (0% corruption, 0 errors), transfer_failed held at 0, and prefill inflight draining normally from 17 to 0. All deployed improvements were confirmed live and innocent. The incident was documented with operational guidance: never restart decode alone against a long-running prefill, monitor num_transfer_failed_reqs_total, and consider installing py-spy and a liveness watchdog for deeper diagnosis if it recurs.

This incident response demonstrated the assistant's ability to resist plausible but incorrect hypotheses, gather live process state and queue metrics, identify the actual mechanism (degraded bootstrap from decode-only restarts), apply a targeted fix that preserved all performance improvements, and produce actionable operational documentation.

Client-Side Deadlock Diagnosis

The final major work in this chunk was the diagnosis of a persistent hang in the user's session-bible tool, which orchestrates multiple parallel LLM agents. Despite previous fixes for server-side deadlocks, the tool was hanging again with agents failing to progress past write(). The user provided a SIGQUIT goroutine dump as definitive evidence.

The goroutine dump revealed a complete client-side deadlock: the main orchestrator was blocked waiting for parallel agents, all agent goroutines were stuck in net/http calls to the LLM API, and the rate limiter (Pacer) was blocked waiting for a refill. The underlying TCP connections were in IO wait, indicating the LLM API was not responding. This isolated the bottleneck entirely to the HTTP layer, distinguishing this hang from the previously fixed server-side PD deadlocks.

The task shifted from server-side debugging to hardening the client: adding timeouts, circuit breakers, and connection limits to the LLMClient to prevent a single upstream stall from freezing the entire process. This diagnosis highlighted the fragility of synchronous concurrent I/O in agentic systems when the upstream API becomes unresponsive.

Themes and Patterns Across the Chunk

Several overarching themes emerge from this chunk of work:

Evidence-driven debugging. Across all phases — infrastructure setup, corruption investigation, performance optimization, incident response, and client-side diagnosis — the assistant consistently prioritizes empirical evidence over plausible hypotheses. When the user suspects the overlap scheduler, the assistant checks the live configuration. When the prime hypothesis about input temporaries seems elegant, the assistant traces each tensor and proves the mechanism doesn't apply. This commitment to evidence over intuition is the hallmark of rigorous engineering.

Iterative refinement. The work proceeds through cycles of hypothesis formation, evidence gathering, conclusion drawing, and pivot. The flash-attn build goes through multiple iterations of MAX_JOBS adjustment. The corruption investigation cycles through half a dozen hypotheses before landing on the multi-stream overlap race. Each cycle narrows the search space and produces new knowledge that informs the next cycle.

Deep systems knowledge. The assistant demonstrates fluency across the entire stack: Linux system administration (driver installation, runlevel management), CUDA ecosystem (toolkit versioning, nvcc compilation), Python packaging (uv, dependency resolution), GPU kernel programming (Triton, CUDA graphs), distributed systems (NIXL bootstrap, TP collectives), and application architecture (agent orchestration, HTTP clients). This breadth of knowledge is essential for debugging problems that span multiple layers.

Intellectual honesty. Perhaps the most striking pattern is the assistant's willingness to deliver negative results. When the analysis shows that the requested patch won't fix the bug, the assistant says so explicitly. When the investigation reveals that the user's suspected cause is innocent, the assistant presents the evidence clearly. This honesty builds trust and prevents wasted effort on dead ends.

Conclusion

Segment 72, chunk 0 of this opencode session documents a remarkable breadth of work: from bare-metal infrastructure setup on Ubuntu 24.04 with Blackwell GPUs, through the resolution of complex build issues with flash-attn, to the deep debugging of a CUDA graph capture corruption bug, performance optimization for decode throughput, production incident response, and client-side deadlock diagnosis. The common thread running through all this work is a methodical, evidence-driven approach to problem-solving.

The flash-attn build saga demonstrates the challenges of compiling CUDA extensions in heterogeneous environments and the importance of managing version compatibility, memory pressure, and build ordering. The corruption investigation shows how systematic hypothesis elimination, combined with deep understanding of CUDA graph capture semantics, can isolate a bug to a specific interaction between bf16 precision and multi-stream execution. The incident response illustrates the value of live evidence gathering over speculation. And the client-side diagnosis highlights the need for robust error handling in agentic systems.

Together, these episodes form a case study in production ML engineering — a field where success depends not on any single skill but on the ability to navigate across layers of abstraction, from GPU kernel launches to HTTP connection management, while maintaining a rigorous commitment to empirical evidence and intellectual honesty.