The 296-Gigabyte Problem: Data Sharing Between Virtualization Boundaries

In the sprawling effort to deploy the GLM-5-NVFP4 model across eight NVIDIA RTX PRO 6000 Blackwell GPUs, the conversation captured in message 479 of this opencode session represents a quiet but pivotal moment. It is not a message about debugging kernel modules, wrestling with CUDA initialization errors, or tuning attention backends. It is a message about something far more mundane yet equally critical: how to move a 296 GB model from one storage location to another without re-downloading it over the network.

The message itself is deceptively brief. The assistant writes:

The VM's disk is a ZFS zvol at rpool/data/vm-128-disk-0. I can't easily mount that as a bind mount into the LXC. Better to download the model fresh or share via a different method. But since the model is ~296GB, let me try to mount the zvol and copy the cache, or better yet — create a shared dataset:

Then follows a bash command that creates a new ZFS dataset on the Proxmox host and mounts it at /shared. The output confirms success: a fresh, empty directory ready to receive data.

On its surface, this looks like a routine storage operation. But beneath the simplicity lies a cascade of architectural decisions, constraints, and trade-offs that reveal the true complexity of running large-scale ML inference across virtualized GPU infrastructure. This article unpacks that single message in detail, examining the reasoning that produced it, the assumptions it rests on, and the knowledge it both consumes and creates.## The Strategic Context: Why This Message Exists

To understand why the assistant is thinking about ZFS datasets at this moment, we must understand the broader strategic situation. The session had been pursuing an LXC container approach to bypass a fundamental limitation of the KVM-based virtual machine: the VFIO/IOMMU GPU passthrough mechanism was inserting a PHB (PCIe Host Bridge) topology between all GPUs, preventing direct peer-to-peer (P2P) DMA communication. For an ML inference workload spanning eight GPUs with tensor parallelism, this was a performance disaster — every cross-GPU communication had to traverse the host bridge rather than direct PCIe links.

The LXC container approach had just achieved a major victory. Inside the container, nvidia-smi topo -m showed the true bare-metal topology: NODE connectivity between GPUs within the same NUMA socket, and SYS connectivity across sockets. This was the topology needed for efficient P2P DMA. The NVIDIA driver (590.48.01) was installed on the Proxmox host, the container had been converted from unprivileged to privileged, and all eight GPU device nodes were bind-mounted and accessible.

But a new problem immediately emerged. The model — GLM-5-NVFP4, a massive 296 GB checkpoint — was sitting inside the KVM virtual machine (VM 128), stored on a ZFS zvol at rpool/data/vm-128-disk-0. The LXC container (129) had its own root filesystem on a separate ZFS subvol at rpool/data/subvol-129-disk-0. These are different ZFS volumes with different mount points. The model was trapped inside the VM's disk, inaccessible to the container that needed to run it.

This is the moment message 479 addresses. The assistant has just finished verifying that the container sees all eight GPUs with correct topology. Now it must solve the data problem before any ML work can begin.## The Reasoning Process: Weighing the Options

The assistant's reasoning, visible in the message text, reveals a clear decision-making process. Three options are explicitly or implicitly considered:

Option 1: Mount the ZFS zvol directly into the LXC container. The assistant immediately rejects this: "I can't easily mount that as a bind mount into the LXC." This is a technically accurate assessment. A ZFS zvol is a raw block device, not a filesystem. It appears as /dev/zvol/rpool/data/vm-128-disk-0 and requires a filesystem (like ext4 or xfs) to be created on it before it can be mounted. Even then, the VM's disk likely contains a partition table, boot loader, and the full Ubuntu filesystem — not just the HuggingFace cache. Bind-mounting a specific directory from within a zvol's filesystem into an LXC container is not straightforward without mounting the zvol on the host first, which would conflict with the running VM.

Option 2: Re-download the model. The assistant mentions this explicitly: "Better to download the model fresh or share via a different method." But then immediately notes the size: "~296GB." Downloading nearly 300 GB over the internet would take hours, consume bandwidth, and potentially incur costs. For a team iterating on deployment, this is a non-starter.

Option 3: Create a shared ZFS dataset and copy the model. This is the chosen approach. The assistant creates a new ZFS dataset (rpool/data/shared) with mount point /shared on the Proxmox host. This dataset can then be bind-mounted into the LXC container, just like the GPU device nodes were. The model files can be copied from the VM's disk to this shared location, either by mounting the VM's disk temporarily (after stopping the VM) or by copying over the network from within the VM.

The elegance of this approach is that it leverages ZFS's native capabilities — datasets are lightweight, can be created on the fly, and can be mounted anywhere. The shared dataset becomes a bridge between the VM's storage and the container's storage, avoiding both the complexity of zvol mounting and the cost of re-downloading.## Assumptions Embedded in the Decision

Every technical decision rests on assumptions, and this message is no exception. Several are worth surfacing:

Assumption 1: The VM can be stopped to access its disk. Copying a 296 GB model from a ZFS zvol while the VM is running is risky — the filesystem inside the VM may have cached or partially-written data that a host-level mount would not see consistently. The assistant implicitly assumes that the VM can be taken offline, or that the model can be copied from within the VM via SSH/rsync. Both approaches work, but each has trade-offs: stopping the VM disrupts any running workload, while network copy adds latency.

Assumption 2: ZFS datasets are the right abstraction. The assistant chooses to create a new ZFS dataset rather than, say, an NFS share, a CIFS mount, or a simple directory on the host's root filesystem. This assumes that the Proxmox host has ZFS storage available (it does — the root pool is ZFS) and that the LXC container can bind-mount ZFS paths. Both are true, but the assumption is worth noting because it ties the solution to ZFS-specific features.

Assumption 3: The model cache structure is portable. The HuggingFace cache (~/.cache/huggingface/hub/) stores models in a specific directory layout with symlinks and hash-based filenames. The assistant assumes that copying this cache to a new location and pointing the container's HuggingFace environment variables at it will work seamlessly. This is generally true, but version mismatches in the transformers library or tokenizer files can cause subtle breakage.

Assumption 4: Bind-mounting a directory from the host into the LXC container is straightforward. The assistant had already configured bind-mounts for GPU device nodes earlier in the session. Extending this pattern to a shared dataset is natural but not trivial — it requires adding an lxc.mount.entry line to the container config and ensuring the mount point exists inside the container. The assistant's confidence in this approach is justified by prior success with the GPU mounts.

These assumptions are reasonable, but they are not guaranteed. The session's subsequent messages will reveal whether they hold.## Input Knowledge: What You Need to Understand This Message

To fully grasp what is happening in message 479, a reader needs knowledge spanning several domains:

ZFS storage concepts. The message references ZFS zvols, subvols, datasets, and mount points. A zvol is a ZFS volume presented as a block device — it is not a filesystem and cannot be bind-mounted directly. A ZFS dataset (created with zfs create) is a filesystem that can be mounted at an arbitrary path. The distinction is critical to understanding why the assistant rejects the zvol approach and creates a new dataset instead.

Proxmox VE storage architecture. Proxmox uses ZFS pools for VM and container storage. VM disks are typically zvols (e.g., vm-128-disk-0), while container root filesystems are subvols (e.g., subvol-129-disk-0). Understanding this layout explains why the model cache is not directly accessible from the container.

LXC bind-mount mechanics. LXC containers can mount host directories into the container using lxc.mount.entry configuration lines. The assistant had already used this pattern extensively in prior messages to expose GPU device nodes. The shared dataset approach extends this same mechanism to data storage.

HuggingFace model cache structure. The assistant knows that HuggingFace stores downloaded models in ~/.cache/huggingface/hub/ and that this cache can be relocated by setting the HF_HOME or HF_HUB_CACHE environment variable. This knowledge informs the decision to copy the cache rather than re-download.

The scale of modern LLMs. The 296 GB figure is not arbitrary — it reflects the size of the GLM-5-NVFP4 model, a large language model using 4-bit NVFP quantization. Understanding that this is too large for a quick re-download over typical internet connections is essential to appreciating why the shared dataset approach is necessary.## Output Knowledge: What This Message Creates

Message 479 produces several concrete outputs that shape the subsequent session:

A new ZFS dataset. The command zfs create -o mountpoint=/shared rpool/data/shared creates a persistent, mountable filesystem on the Proxmox host. This dataset exists independently of both the VM and the container, making it a durable bridge for data sharing.

A mount point at /shared. The dataset is mounted at /shared on the host, ready to be bind-mounted into the LXC container. This path becomes the staging ground for the model cache transfer.

A documented decision. The message captures the reasoning behind choosing the shared dataset approach over alternatives. This documentation is valuable for future debugging — if something goes wrong with the model cache, the team can trace back to this decision point.

A template for future data sharing. The pattern established here — create a ZFS dataset, mount it on the host, bind-mount into the container — can be reused for any future data that needs to move between VMs and containers on the same Proxmox host.

A subtle shift in architecture. Before this message, the model cache was entirely within the VM's storage boundary. After this message, it moves to a shared location accessible to both the VM and the container. This is a small but meaningful step toward decoupling the model data from any single virtualization domain.

The Unseen Trap: What This Message Does Not Yet Know

There is a poignant irony in message 479 that only becomes visible with hindsight from later messages. The assistant is carefully solving the data sharing problem, creating a shared dataset, and preparing to copy the model cache — all under the assumption that the LXC container will be the final deployment environment. But a much deeper problem is about to surface.

In the messages immediately following this one, the assistant discovers that CUDA runtime initialization fails inside the LXC container. The cuInit call returns error code 3 (CUDA_ERROR_NOT_INITIALIZED). The NVIDIA driver sees the GPUs — nvidia-smi works perfectly — but the CUDA runtime cannot initialize. The root cause turns out to be a GSP (GPU System Processor) firmware incompatibility between the NVIDIA driver version 590.48.01 and the Blackwell architecture on the Proxmox VE kernel (6.8.12-9-pve).

The shared dataset, so carefully created in message 479, will sit empty for the remainder of the session as the team pivots to debugging the driver issue. The model cache is never copied. The elegant ZFS-based data bridge is never used.

This does not diminish the message's importance. On the contrary, it illustrates a fundamental truth about complex engineering work: you must solve the problems you can see before you can see the problems you cannot. The data sharing problem was visible, understood, and solvable — so the assistant solved it. The GSP firmware problem was invisible until the driver stack was fully exercised. Message 479 represents the kind of careful, forward-looking infrastructure work that is necessary but not sufficient for success. It clears one bottleneck only to reveal the next one deeper in the system.