The Pinned Memory Experiment: Testing CUDA's Defiance of Linux ulimits

Introduction

In the middle of deploying a GPU-accelerated proof-of-spacetime proving system to memory-constrained cloud instances, the development team encountered a puzzling warning. A custom memory-check utility (memcheck.sh) had flagged a critical problem: the container's ulimit -l (maximum size for memory-lockable pages) was set to a mere 8192 kB—roughly 8 MB. For a system that planned to allocate gigabytes of pinned (page-locked) memory for CUDA operations, this seemed like a showstopper. The conventional wisdom in CUDA programming holds that pinned memory allocation requires sufficient memlock limits, and 8 MB is laughably insufficient for workloads that need to pin multiple gigabytes of host memory for high-speed GPU transfers.

But conventional wisdom is not always correct when kernel drivers are involved. Message [msg 3926] captures the moment when the team decided to stop guessing and start measuring—a single SSH command that would determine whether months of deployment work needed a fundamental redesign, or whether the NVIDIA driver's special relationship with Linux memory management would save the day.

The Context: Deploying to Hostile Territory

The broader project involved deploying cuzk, a CUDA-based zero-knowledge proving engine, to vast.ai cloud instances. These instances are Docker containers running on shared GPU hardware, often with strict cgroup memory limits that are far lower than the host machine's total RAM. The team had already invested significant effort in making the system cgroup-aware: the Rust detect_system_memory() function was rewritten to read cgroup v2 (memory.max) and v1 (memory.limit_in_bytes) limits rather than trusting /proc/meminfo, which reports the host's full RAM even inside a constrained container. This fix alone prevented catastrophic over-allocation that would have caused OOM kills.

The deployment pipeline included a memcheck.sh utility that inspected the container environment and produced a JSON report with budget recommendations. When the team deployed to their first test instance—a machine with 2003 GiB of host RAM but only 962 GiB available to the container via cgroup limits—memcheck.sh ran successfully but flagged two issues. The first was a JSON parsing bug where GPU names with spaces (e.g., "NVIDIA GeForce RTX 4090") were incorrectly split, producing invalid JSON. The second was the ulimit -l warning: the container's memlock limit was 8192 kB, far below the 50 GiB threshold that memcheck.sh considered safe for pinned memory operations.

The Assumption Under Test

The ulimit -l warning was based on a reasonable assumption: that CUDA's cudaHostAlloc function, which allocates page-locked (pinned) host memory, requires the process to have a sufficiently high RLIMIT_MEMLOCK value. This assumption is documented in NVIDIA's own literature and is a common source of configuration issues in Docker deployments. The standard fix is to pass --ulimit memlock=-1:-1 to docker run, which removes the limit entirely. However, vast.ai's instance creation API (via vastai create instance) did not support passing Docker runtime flags like --ulimit. The team was stuck—they couldn't raise the limit from inside the container (both soft and hard limits were 8192 kB), and they couldn't pass the flag at container creation time through the vast.ai API.

But there was a subtlety worth testing. The NVIDIA kernel driver (nvidia.ko and related modules) manages pinned memory through its own DMA mapping infrastructure, not through the standard Linux mlock() system call. When cudaHostAlloc is called, the driver interacts with the GPU's memory management unit and the PCIe bus controller to establish direct memory access (DMA) mappings. While the CUDA runtime does check RLIMIT_MEMLOCK in some configurations, the actual enforcement path is complex and depends on the driver version, the kernel version, and whether the nvidia-container-runtime is in use. On vast.ai instances, the container runtime might not be the full NVIDIA container toolkit—it could be a plain Docker container with GPU passthrough via --gpus all, which changes the memory pinning behavior.

The team had a hypothesis: on this particular vast.ai setup, cudaHostAlloc might work despite the low ulimit because the NVIDIA driver handles pinned memory allocation through its own kernel-level mechanisms, bypassing the RLIMIT_MEMLOCK check. This hypothesis was based on observations from other developers who had successfully used pinned memory in Docker containers without setting --ulimit memlock=-1:-1. But it was just a hypothesis—and the cost of being wrong was high. If pinned memory allocation failed at runtime, the SRS (Structured Reference String) loading phase would crash, the proving pipeline would stall, and the entire deployment would be unusable on vast.ai instances.

The Experiment: One SSH Command

Message [msg 3926] is the execution of a carefully designed empirical test. The assistant constructs a Python script that uses the ctypes foreign function library to call CUDA's C API directly, bypassing any higher-level abstractions that might mask the underlying behavior. The script does three things:

Initialize the CUDA driver via cuInit(0), which establishes communication with the NVIDIA driver and the GPU.
Allocate 1 GiB of pinned host memory via cudaHostAlloc, passing a pointer reference and the size in bytes. The third argument (flags) is 0, requesting default pinned memory.
Free the memory via cudaFreeHost if allocation succeeded, and print a conclusive message. The SSH command itself reveals several important details about the environment. The LD_LIBRARY_PATH is set explicitly to include /usr/lib/x86_64-linux-gnu (for libcuda.so.1) and /usr/local/cuda-13.0/targets/x86_64-linux/lib (for libcudart.so.13). This tells us that the container has CUDA 13.0 installed, but the library paths are not in the default search path—a common configuration on Docker images where multiple CUDA versions might be present. The assistant had to discover these paths through earlier exploration ([msg 3925]), where a find command located the CUDA libraries scattered across the filesystem. The Python script is wrapped in a single-quoted string to prevent shell expansion, and 2>&1 redirects stderr to stdout for complete capture. The output is clean and unambiguous:

cuInit: 0
cudaHostAlloc(1GiB): 0 (0=success)
Freed OK — pinned memory works despite ulimit!

All three operations succeeded. The return code 0 from cuInit confirms that the CUDA driver initialized successfully. The return code 0 from cudaHostAlloc confirms that 1 GiB of pinned memory was allocated without error. The final message confirms that the memory was freed cleanly.

What This Result Means

This single test result had profound implications for the deployment strategy. It meant that:

The ulimit -l warning in memcheck.sh was a false positive for this specific environment. The NVIDIA driver on this system (CUDA 13.0, driver 590.48.01, on a Linux kernel with cgroup v2) did not enforce RLIMIT_MEMLOCK for cudaHostAlloc allocations. The pinned memory pool that cuzk relied on for zero-copy GPU transfers would work without any Docker runtime configuration changes.
No changes to the vast.ai instance creation workflow were needed. The team had been considering workarounds like requesting custom Docker runtime flags from vast.ai support, modifying the instance template, or even switching to a different cloud provider. All of these options were now unnecessary.
The memcheck.sh utility needed a logic adjustment. The warning threshold for ulimit -l should be downgraded from a critical error to an informational note, since the actual CUDA behavior depends on the driver and runtime configuration. The team could add a runtime probe (like the one in this message) to memcheck.sh itself, testing pinned memory allocation directly rather than inferring capability from the ulimit value.
The deployment could proceed immediately. The test instance was ready for the full cuzk benchmark run without any ulimit-related modifications. The only remaining issues were the GPU JSON parsing fix (already applied in [msg 3917]) and the entrypoint resilience fix (applied in [msg 3918]).

The Thinking Process Visible in the Message

The message reveals a methodical, hypothesis-driven approach to debugging. The assistant did not simply accept the memcheck.sh warning as definitive. Instead, it recognized that the warning was based on an indirect heuristic (checking ulimit -l) rather than a direct test of the actual CUDA behavior. The chain of reasoning visible across the preceding messages shows:

Awareness of the abstraction gap: The assistant knew that cudaHostAlloc goes through the NVIDIA kernel driver, which may have its own memory management paths that bypass RLIMIT_MEMLOCK. This is not widely documented and requires deep knowledge of the CUDA driver architecture.
Progressive information gathering: The assistant first checked the ulimit values ([msg 3923]), finding both soft and hard limits at 8192 kB. Then it attempted a simple Python test but failed because libcuda.so wasn't in the default library path ([msg 3924]). It then located the actual library paths via find ([msg 3925]). Only after understanding the environment's library layout did it construct the final test command with the correct LD_LIBRARY_PATH.
Minimal, targeted experiment: The test allocates exactly 1 GiB—a meaningful size that exercises the pinned memory allocation path without being so large as to risk OOM or take excessive time. The script is self-contained and produces clear, machine-parseable output.
Risk awareness: The assistant could have simply assumed the ulimit warning was correct and spent days redesigning the deployment to avoid pinned memory or to work around the vast.ai API limitations. Instead, it invested a few minutes in an empirical test that either validated or invalidated the assumption. This is a classic engineering tradeoff: a small, targeted experiment can eliminate weeks of unnecessary work.

Broader Significance

This message is a case study in the importance of empirical validation in systems engineering. The assumption that RLIMIT_MEMLOCK controls CUDA pinned memory allocation is reasonable and widely believed, but it turns out to be environment-dependent. On this particular combination of CUDA version, NVIDIA driver, kernel version, and container runtime, the assumption was false. The team's willingness to test the assumption directly—rather than accepting the warning and embarking on a complex workaround—saved significant time and effort.

The result also highlights the evolving nature of GPU virtualization. Modern NVIDIA drivers and container runtimes are increasingly sophisticated about memory management. The nvidia-container-runtime and nvidia-container-toolkit handle many of the low-level configuration details automatically, including memory pinning capabilities. On vast.ai, which uses a custom container orchestration layer, the driver's behavior may differ from both bare-metal and full NVIDIA-container-stack deployments. The only way to know for certain is to test on the actual target environment.

For the broader project, this message represents a turning point. The cgroup-aware memory detection had already solved the OOM problem. The GPU JSON parsing fix had resolved the configuration pipeline issue. And now, the ulimit concern was empirically dismissed. The deployment could proceed to the next phase: running the full benchmark to measure proof throughput and validate the entire system under realistic loads. The pinned memory experiment was the last unknown in the deployment puzzle, and it yielded the answer the team needed.