The Moment Memory Was Reclaimed: A Deep Dive Into Linux Cache Management During ML Build Failures

In the midst of a grueling session to set up a machine learning environment on Ubuntu 24.04, a single assistant message at index 53 represents a critical turning point — a moment of diagnostic clarity that turned a stalled, memory-exhausted system back into a productive build environment. The message is brief, but it encapsulates a deep understanding of Linux memory management that many practitioners overlook. Here is the message in full:

Still 369GB used, something lingering. Let me clean up properly:

>

``bash ssh 10.1.230.175 'pkill -9 -f "uv pip" ; pkill -9 -f "flash" ; pkill -9 -f ptxas ; pkill -9 -f nvcc ; pkill -9 -f cicc ; pkill -9 -f cc1plus ; pkill -9 -f ninja ; sleep 3 ; sync ; echo 3 | sudo tee /proc/sys/vm/drop_caches ; sleep 2 ; free -h' ``

To understand why this message matters, we must trace the arduous path that led to it.

The Flash-Attention Build Saga

The conversation up to this point had been a masterclass in the fragility of CUDA-based ML toolchains. The assistant had successfully installed NVIDIA drivers (590.48.01) and CUDA Toolkit 13.1 on a remote Ubuntu 24.04 machine with two (later eight) NVIDIA RTX PRO 6000 Blackwell GPUs. A Python virtual environment was created using uv, and core packages like PyTorch were installed. Then came the bottleneck: flash-attn.

Flash-attn is a CUDA extension that must be compiled from source when no precompiled wheel matches the exact combination of CUDA version and PyTorch version. The system had CUDA 13.1, but PyTorch had been compiled against CUDA 12.8 — a mismatch that triggered the build path. The assistant installed a secondary CUDA 12.8 toolkit to satisfy the build dependency, and then the real trouble began.

The build process for flash-attn compiles dozens of CUDA kernels in parallel using ptxas (the NVIDIA PTX assembler) and nvcc (the CUDA compiler). Each compilation job consumes significant memory. On a 128-core machine, the default parallelism was aggressive. The assistant and user went through a painful trial-and-error cycle: MAX_JOBS=128 caused out-of-memory (OOM), then MAX_JOBS=64 also OOM'd, then MAX_JOBS=32 still OOM'd. The machine was rebooted and upgraded to 432GB of RAM, yet even MAX_JOBS=32 continued to fail.

The Lingering Memory Problem

After each failed build attempt, the assistant dutifully killed the build processes. Message 52 shows the result of one such cleanup:

Mem:           432Gi       369Gi        64Gi

Only 64GB free out of 432GB. The build processes had been killed, but 369GB of memory remained allocated. This is the puzzle that message 53 addresses.

The assistant's opening line — "Still 369GB used, something lingering" — is the key diagnostic insight. The word "lingering" is precise. It conveys that the memory isn't legitimately in use by active processes, but it also isn't free. Something is holding onto it, and normal process termination didn't release it.

The Solution: Understanding Linux Page Cache

The assistant correctly diagnosed that the culprit was the Linux kernel's page cache and related caches. When CUDA compilation runs, the compiler reads and writes many temporary files. The kernel caches these file contents in memory (the page cache) to speed up subsequent access. It also caches directory structures (dentries) and filesystem metadata (inodes). After a massive build with hundreds of compilation units, these caches can balloon to hundreds of gigabytes.

The kill commands only terminate processes — they don't touch kernel caches. The memory used by the page cache is not reclaimed until the kernel decides to evict it, or until it is explicitly dropped.

The assistant's cleanup command in message 53 is a carefully orchestrated sequence:

  1. pkill -9 -f "uv pip" — Kills the pip/uv process that was orchestrating the build.
  2. pkill -9 -f "flash" — Kills any remaining flash-attn related processes.
  3. pkill -9 -f ptxas — Kills the NVIDIA PTX assembler processes, which are the main memory consumers during compilation.
  4. pkill -9 -f nvcc — Kills the CUDA compiler driver.
  5. pkill -9 -f cicc — Kills the CUDA internal compiler compiler (the backend compiler).
  6. pkill -9 -f cc1plus — Kills the GCC C++ compiler frontend, which is used during the build.
  7. pkill -9 -f ninja — Kills the Ninja build system, which orchestrates parallel compilation.
  8. sleep 3 — Waits for processes to fully terminate and for the kernel to process the signals.
  9. sync — Flushes all filesystem buffers to disk, ensuring no dirty pages remain in the page cache.
  10. echo 3 | sudo tee /proc/sys/vm/drop_caches — This is the critical step. Writing 3 to /proc/sys/vm/drop_caches tells the kernel to free clean page cache entries (1), dentries (2), and inodes (2) — the 3 is a bitmask that combines both operations. This reclaims all the cached file data that the build left behind.
  11. sleep 2 — Waits for the cache drop to take effect.
  12. free -h — Checks the result. The result, visible in message 54, was dramatic:
Mem:           432Gi       7.0Gi       427Gi

From 369GB used to 7GB used — 362GB of memory reclaimed in a single command sequence. The system went from nearly full to almost empty, ready for a fresh build attempt.

Assumptions and Risks

The assistant made several assumptions in this message. First, it assumed that the lingering memory was indeed in kernel caches (page cache, dentries, inodes) rather than in memory leaks from zombie processes or kernel memory fragmentation. This was a reasonable assumption given that all build processes had been killed and only the page cache could account for such a large, persistent allocation.

Second, it assumed that dropping caches was safe. Writing to /proc/sys/vm/drop_caches is a non-destructive operation — it only frees clean caches (those already written to disk). The sync command before it ensures that any dirty pages are flushed first, making them safe to drop. However, this operation does cause a temporary performance penalty because the kernel must re-read files from disk when they are next accessed. In the context of a fresh build, this penalty was acceptable.

Third, the assistant assumed that the build processes had been fully killed by the previous commands. The pkill -9 sends SIGKILL, which cannot be caught or ignored by processes, so this was a safe assumption. However, the sleep 3 after the kills was a pragmatic acknowledgment that process termination and resource cleanup are not instantaneous.

What This Message Reveals About the Assistant's Thinking

This message is a window into the assistant's reasoning process. The explicit statement "Still 369GB used, something lingering" shows that the assistant is monitoring state, comparing expected outcomes (memory should be free after killing processes) with actual outcomes (369GB still used), and forming hypotheses about the discrepancy.

The phrase "Let me clean up properly" is telling. It implicitly acknowledges that previous cleanup attempts (messages 31-52) were incomplete. The assistant recognized that its earlier approach — simply killing processes — was insufficient for the problem at hand. This is a learning moment within the conversation itself.

The command construction also reveals systematic thinking. The assistant didn't just add drop_caches to the existing kill commands — it reordered and expanded the kill list to include cc1plus (the GCC compiler frontend) and ninja, which were missing from earlier kill commands. It added sync before the cache drop, which is essential for safety. It added sleep intervals to allow operations to complete. The entire sequence is a pipeline where each step prepares for the next.

The Broader Lesson

For anyone working with CUDA-based ML toolchains, this message encapsulates a crucial operational lesson. Building flash-attn, vLLM, or similar CUDA extensions from source on high-core-count machines is notoriously memory-intensive. Failed builds leave behind not just stray processes but gigabytes of kernel caches. A simple kill is not enough — you must also purge the page cache.

The /proc/sys/vm/drop_caches interface is a powerful tool, but it is often overlooked. Many practitioners, when faced with "out of memory" errors after a failed build, either reboot the machine (as the user did earlier in this conversation) or give up. The assistant's approach in message 53 offers a third path: targeted cache eviction that reclaims memory without a reboot.

This message also demonstrates the value of reading system state carefully. The assistant didn't just run free -h and move on — it interpreted the numbers, formed a hypothesis about "something lingering," and designed a targeted intervention. This kind of diagnostic reasoning is what separates effective system administration from rote command execution.

Conclusion

Message 53 is a small but pivotal moment in a larger conversation about ML environment setup. It represents the transition from a cycle of failed builds and frustrated cleanup to a properly functioning system. The assistant's recognition that kernel caches were the problem, and its knowledge of how to safely reclaim them, turned a stalled 432GB machine back into a productive build environment. The very next message (54) confirmed the cleanup worked, and message 55 launched a successful build with MAX_JOBS=32 that finally completed.

In the broader narrative of this coding session, message 53 is the turning point — the moment when deep system knowledge overcame a stubborn operational problem. It is a reminder that in complex ML infrastructure work, understanding the operating system is just as important as understanding the model architecture.