From Bare Metal to Model Serving: Building an 8-GPU ML Environment and Deploying GLM-5-NVFP4 with SGLang

Introduction

Deploying large language models at scale is never a straightforward affair. Between the hardware setup, driver compatibility, kernel compilation, and framework configuration, the path from a fresh Ubuntu installation to a running model server is paved with challenges that test even experienced engineers. This article chronicles one such journey: a multi-session opencode conversation that began with a bare Ubuntu 24.04 machine and ended with a production-ready SGLang deployment of the GLM-5-NVFP4 model across 8 NVIDIA RTX PRO 6000 Blackwell GPUs. Along the way, the team navigated NVIDIA driver installation, CUDA toolkit conflicts, flash-attn compilation nightmares, memory exhaustion, PyTorch version mismatches, and a deep investigation into KV cache offloading strategies—all while building toward a high-performance model serving infrastructure.

This chunk of the conversation captures the critical transition from infrastructure setup to application deployment. It is a story of iterative problem-solving, where each layer of the stack revealed new challenges that had to be methodically addressed before the next layer could be built. The work demonstrates a pattern that will be familiar to anyone who has deployed ML models in production: the environment is never quite ready, the dependencies never quite aligned, and the path forward requires equal parts technical skill, patience, and systematic debugging.

The Infrastructure Foundation: NVIDIA Drivers and CUDA Toolkit

The session began with a straightforward but critical task: installing the latest NVIDIA drivers and CUDA toolkit on a remote Ubuntu 24.04 machine. The assistant connected to the server at 10.1.230.174 and immediately set to work installing the NVIDIA driver version 590.48.01, a cutting-edge driver build required for the RTX PRO 6000 Blackwell architecture. This was followed by the installation of CUDA Toolkit 13.1, the latest available version at the time.

The choice of CUDA 13.1 is notable. Most ML frameworks at the time were built against CUDA 12.x, and this version mismatch would become a recurring theme throughout the session. However, for Blackwell GPUs (SM120 architecture), the latest driver and CUDA support were essential—older toolkits simply did not support the new hardware. The assistant configured the environment paths (PATH, LD_LIBRARY_PATH, CUDA_HOME) and verified that two RTX PRO 6000 Blackwell GPUs were operational and recognized by the system.

With the GPU stack in place, the assistant turned to the Python environment. Rather than using the traditional pip or conda, the assistant chose uv—a modern, fast Python package manager written in Rust. This choice reflected a preference for tooling that could handle complex dependency resolution efficiently, a decision that would prove valuable as the dependency graph grew more tangled. A virtual environment was created, and core packages including PyTorch were installed. At this point, the foundation seemed solid: drivers working, GPUs recognized, Python environment ready.

The flash-attn Saga: A Masterclass in Build Debugging

The installation of flash-attn—a high-performance CUDA kernel for FlashAttention—became the session's first major bottleneck and the most instructive debugging sequence. Flash-attn is a critical dependency for many LLM serving frameworks, providing the fused attention kernels that enable efficient inference. But building it from source on a new CUDA toolkit with new GPU architecture proved anything but straightforward.

The CUDA Version Conflict

The first problem was a CUDA version mismatch. PyTorch, at the time, was built against CUDA 12.8. The system had CUDA 13.1 installed. When flash-attn attempted to compile its CUDA kernels, it found CUDA 13.1 but the PyTorch headers and runtime expected CUDA 12.8. This mismatch caused compilation errors that manifested as cryptic linker failures and missing symbol errors.

The solution was to install a secondary CUDA 12.8 toolkit alongside the primary 13.1 installation. This is a common pattern in ML environments where different components depend on different CUDA versions. The assistant installed CUDA 12.8 and configured the build to use it specifically for flash-attn compilation, while keeping CUDA 13.1 as the system default for driver-level operations.

Memory Exhaustion and the MAX_JOBS Tango

With the CUDA version resolved, the build process ran into a different wall: memory exhaustion. The server had a substantial amount of RAM, but flash-attn's build process, by default, spawned as many parallel compilation jobs as there were CPU cores. On a high-core-count server, this meant 128 simultaneous compiler processes, each consuming significant memory. The system quickly ran out of RAM, causing the build to fail with out-of-memory errors.

The assistant and user engaged in a collaborative trial-and-error process to find a sustainable MAX_JOBS setting. Starting from the default (128), they reduced to 64, then 32, then 24, and finally 20—each reduction requiring a full rebuild and waiting for the compilation to either succeed or fail. Even at 20 jobs, the build would occasionally fail, suggesting that memory pressure was not the only issue. The problem was only fully resolved after the machine was rebooted with an expanded 432 GB of RAM, providing enough headroom for the compilation to complete.

This episode highlights a subtle but important aspect of building CUDA extensions: the compilation process is itself a computationally intensive task that can stress system resources as much as the final kernels do. The MAX_JOBS environment variable, which controls parallel compilation, is often overlooked but can be the difference between a successful build and a cryptic crash.

The vLLM Downgrade and PyTorch Mismatch

Just when flash-attn was finally compiled and working, a new dependency conflict emerged. The installation of vLLM—another critical component of the serving stack—triggered a PyTorch downgrade. vLLM's dependency specification required a specific PyTorch version that was older than the one installed, and the package manager dutifully downgraded PyTorch to satisfy this constraint.

The problem was that flash-attn had been compiled against the original PyTorch version (2.9.1). After the downgrade, the compiled flash-attn binary was incompatible with the new PyTorch runtime, causing segmentation faults and cryptic import errors. The assistant had to rebuild flash-attn from scratch against the correct PyTorch version, this time ensuring that vLLM's version constraints were satisfied before the build.

This dependency chain—flash-attn compiled against PyTorch, PyTorch constrained by vLLM—is a classic example of the "dependency diamond" problem in ML infrastructure. Each component has its own version requirements, and satisfying all of them simultaneously can require careful orchestration. The assistant ultimately stabilized the environment with a compatible stack: PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1.

Environment Stabilization and the 8-GPU Upgrade

With the core dependencies resolved, the environment was finally stable. The assistant had navigated CUDA version conflicts, memory exhaustion during compilation, and dependency-induced version mismatches. The stack was now coherent: Ubuntu 24.04 with NVIDIA drivers 590.48.01, CUDA 13.1 (with CUDA 12.8 available for compilation), PyTorch 2.9.1, flash-attn 2.8.3, and vLLM 0.15.1.

At this point, the machine was upgraded from 2 to 8 GPUs. This hardware expansion dramatically increased the available compute and memory capacity, enabling larger model deployments and higher throughput. The assistant verified that all 8 RTX PRO 6000 Blackwell GPUs were recognized and operational.

Pivot to Model Deployment: GLM-5-NVFP4 with SGLang

With the environment stable and the hardware expanded, the conversation pivoted sharply from infrastructure setup to application deployment. The user tasked the assistant with deploying the GLM-5-NVFP4 model using a nightly build of SGLang, followed by performance tuning and load testing.

This shift is significant. The first half of the session was entirely about building the foundation—installing drivers, resolving compilation issues, and stabilizing the environment. The second half was about using that foundation to serve a real model. The GLM-5-NVFP4 model, with its NVFP4 (NVIDIA FP4) quantization, represented a cutting-edge deployment target that required the latest SGLang features and Blackwell-specific optimizations.

The deployment involved building SGLang from a nightly branch to ensure support for the latest model architectures and quantization formats. This meant yet another round of compilation, but this time the environment was prepared—CUDA toolkits were in place, PyTorch was stable, and the build infrastructure was tested.

The KV Cache Offloading Investigation: A Subagent Deep Dive

Embedded within this chunk is a significant subagent session dedicated to researching KV cache offloading options in SGLang. The user, anticipating that GPU memory would become a bottleneck for serving long-context requests, spawned a subagent with a focused research task: investigate every mechanism in SGLang v0.5.9 that could expand effective KV cache capacity by utilizing the server's abundant system RAM (~396 GB).

The subagent conducted a systematic codebase exploration, examining server_args.py, scheduler.py, memory_pool.py, memory_pool_host.py, hiradix_cache.py, cache_controller.py, and several other files. The investigation produced a comprehensive report covering five distinct mechanisms:

--enable-hierarchical-cache (HiCache): The primary recommendation—a two-tier caching system that uses CPU RAM as a secondary tier for evicted KV cache entries. The subagent traced its implementation through HiRadixCache, the host memory pool classes, and the HiCacheController that manages async GPU-to-CPU transfers using dedicated CUDA streams.
--cpu-offload-gb: Correctly identified as a model weight offloading mechanism, not a KV cache feature. Dismissed as irrelevant since the model weights already fit comfortably in GPU memory.
--enable-lmcache: An external third-party KV cache management library that integrates with SGLang. Found to be mutually exclusive with HiCache and more suited for distributed cache sharing across instances.
--disaggregation-decode-enable-offload-kvcache: A PD disaggregation feature explicitly incompatible with speculative decoding (spec_v2). Marked as not applicable to the user's setup.
Internal memory pool utilities: get_cpu_copy() and load_cpu_copy() methods in the memory pool, identified as internal utilities not exposed as user-facing features. The subagent's report was notable for its thoroughness and practical orientation. It provided quantitative projections (e.g., with --hicache-size 45, each GPU would gain approximately 688,000 additional token slots), flagged a soft concern about CUDA graph execution during HiCache loading, and offered a concrete recommended configuration. This research would prove invaluable as the deployment scaled to handle more concurrent requests. This subagent session produced the 14 articles referenced in this chunk ([1] through [14]), which together form a comprehensive analysis of SGLang's KV cache offloading landscape. The articles cover everything from the initial research prompt design to the final verification of hierarchical cache compatibility with speculative decoding.

Themes and Lessons

Several themes emerge from this chunk of work:

The fragility of ML infrastructure. Every layer of the stack introduced unexpected complications. The CUDA version mismatch between the system toolkit and PyTorch's expectations, the memory exhaustion during parallel compilation, the dependency-induced PyTorch downgrade—each of these could have been a showstopper without systematic debugging.

The value of systematic investigation. The flash-attn debugging process, with its iterative reduction of MAX_JOBS and careful tracking of which CUDA toolkit was active, exemplifies a methodical approach to build problems. Similarly, the KV cache offloading research followed a clear pattern: read the source code, trace the implementation, check for compatibility, and synthesize findings.

The importance of forward-looking research. The KV cache offloading investigation was conducted proactively, before memory became a bottleneck in production. This is a hallmark of mature engineering—anticipating constraints before they become crises.

The power of the subagent model. The ability to spawn a focused research subagent that independently explores a codebase and produces a structured report is a powerful pattern. It allows the main session to continue working on infrastructure while the subagent investigates architectural options in parallel.

Conclusion

This chunk captures a pivotal transition in a large-scale ML deployment effort. The team moved from a bare Ubuntu installation through driver setup, CUDA configuration, Python environment creation, and the arduous process of building flash-attn against a non-standard CUDA toolkit. They stabilized a complex dependency stack, upgraded from 2 to 8 GPUs, and began deploying the GLM-5-NVFP4 model using a nightly build of SGLang. Along the way, a subagent conducted a deep investigation of KV cache offloading strategies that would inform the next phase of performance optimization.

The work demonstrates that deploying LLMs at scale is not a single task but a layered process of building, debugging, and optimizing. Each layer—drivers, CUDA, Python packages, compiled kernels, model frameworks—has its own failure modes and requires its own debugging strategies. The successful deployment described here is a testament to systematic problem-solving and the power of AI-assisted engineering.## The Article Ecosystem: 14 Deep-Dive Analyses

The KV cache offloading investigation was not a single message but a rich ecosystem of 14 articles, each examining a different facet of the problem. These articles, referenced as [1] through [14] in this chunk, form a comprehensive body of analysis that any engineer deploying SGLang at scale would find valuable.

The first article, "The Art of the Research Prompt" [1], deconstructs the user's initial research request itself—a masterful example of how to delegate investigation to an AI agent. It examines the user's assumptions, the structure of their questions, and the thinking process visible in how they prioritized features and bounded the scope of research.

The middle articles trace the assistant's investigation step by step. "Opening the Black Box" [12] covers the initial exploration of server arguments and help text. "Digging Into the Depths" [13] follows the assistant as it reads the core implementation files—hiradix_cache.py, memory_pool_host.py, and cache_controller.py. "Tracing the Thread" [2] shows how a single code reference (cpu_offloading_chunk_size = 8192) led the assistant to discover the memory pool's internal CPU offloading mechanisms.

Several articles focus on the critical compatibility question: does hierarchical cache work with speculative decoding? "The Critical Compatibility Check" [7], "Tracing Compatibility Boundaries" [8], and "Verifying Compatibility" [5] all examine this question from different angles, ultimately confirming that no hard blocker exists while noting a developer TODO about CUDA graph execution.

The final article, "The Anatomy of a Deep-Dive Analysis" [14], is a meta-analysis of the entire investigation. It examines the assistant's methodology, the assumptions underlying its recommendations, and the structure of its final report. It serves as both a summary and a critique, identifying potential blind spots like NUMA effects and PCIe bandwidth contention that the assistant did not fully address.

Together, these 14 articles represent a remarkable depth of analysis—over 30,000 words of investigation into a single feature set of a single framework. They demonstrate what is possible when systematic codebase exploration is combined with clear technical communication.

The Transition to Production

The chunk ends with the environment ready for production deployment. The drivers are installed, the GPUs are operational, the Python environment is stable with a compatible dependency stack, and the KV cache offloading strategy has been thoroughly researched. The user is now positioned to make informed decisions about how to configure SGLang for maximum throughput and memory efficiency.

The next phase of work would involve actually deploying the model, tuning the hierarchical cache configuration, running load tests, and iterating on performance. But the foundation laid in this chunk—both the technical infrastructure and the knowledge base—provides a solid starting point for that work.

References

[1] "The Art of the Research Prompt: Deconstructing a KV Cache Offloading Investigation in SGLang" — Analysis of the user's initial research request and its underlying assumptions.

[2] "Tracing the Thread: How a Single Code Reference Revealed SGLang's Internal KV Cache Offloading Architecture" — How the assistant traced cpu_offloading_chunk_size through the memory pool code.

[3] "Tracing the Integration: How SGLang's KV Cache Offloading Meets Speculative Decoding" — Examination of how HiCache integrates with the speculative decoding pipeline.

[4] "The Final Verification: How One Bash Command Confirmed SGLang's Hierarchical Cache Compatibility with Speculative Decoding" — The final compatibility check.

[5] "Verifying Compatibility: The Critical Check Between Hierarchical Cache and Speculative Decoding" — Detailed verification of the HiCache + speculative decoding combination.

[6] "Pivoting Through Dead Ends: How a Research Agent Systematically Uncovered SGLang's KV Cache Offloading Architecture" — The assistant's method of pivoting when initial searches yielded no results.

[7] "The Critical Compatibility Check: Tracing SGLang's Hierarchical Cache and Speculative Decoding Interaction" — Analysis of the scheduler code for compatibility issues.

[8] "Tracing Compatibility Boundaries: How One Message Uncovered the Delicate Dance Between Hierarchical Cache and Speculative Decoding in SGLang" — Examination of edge cases in the compatibility boundary.

[9] "The Systematic Researcher: Tracing SGLang's KV Cache Offloading Options Through Code" — Overview of the assistant's systematic research methodology.

[10] "Tracing the Hierarchical Cache: A Deep Dive into SGLang's Scheduling Integration" — How HiCache integrates with the scheduler's batch formation logic.

[11] "The Discovery of a Hidden Offloading Path: Uncovering disaggregation_decode_enable_offload_kvcache in SGLang" — Discovery of the disaggregation-specific offloading flag.

[12] "Opening the Black Box: A Systematic Exploration of SGLang's KV Cache Offloading Architecture" — Initial exploration of server arguments and help text.

[13] "Digging Into the Depths: How an AI Assistant Researched SGLang's Hierarchical KV Cache Implementation" — Deep dive into the core implementation files.

[14] "The Anatomy of a Deep-Dive Analysis: Understanding SGLang's KV Cache Offloading Landscape" — Meta-analysis of the entire investigation, including methodology, assumptions, and blind spots.