The Missing Python Files: Diagnosing a Model Loading Failure in sglang

In the middle of a long and arduous deployment session, a single discovery derailed the launch of the GLM-5-NVFP4 model on eight Blackwell GPUs. The assistant had spent hours battling CUDA initialization failures, kernel module incompatibilities, and dependency version mismatches. The LXC container was finally working — P2P bandwidth between GPUs reached 53 GB/s, persistence mode was enabled, and all the key packages (flashinfer, sgl_kernel, torch) reported compatible versions. Yet when the sglang server tried to load the model, it crashed with a cryptic error: KeyError: 'glm_moe_dsa'. The model type was not recognized.

Message [msg 592] captures the moment the assistant identified the root cause. It is a short but pivotal message — a diagnostic turn that reoriented the entire deployment effort. In it, the assistant runs a single ls command on the model snapshot directory and immediately interprets the result: "No Python files! The model snapshot doesn't have the custom modeling code. This was likely downloaded as LFS blobs only." The discovery was both simple and devastating: the Hugging Face model cache contained weights, configuration JSON files, and tokenizer data — but none of the Python source files that trust_remote_code depends on to load the custom glm_moe_dsa architecture.

The Road to This Moment

To understand why this message matters, one must appreciate the context that led to it. The session had been a marathon of infrastructure troubleshooting. The user was deploying GLM-5-NVFP4 — a quantized version of the GLM-5 large language model using NVIDIA's NVFP4 precision — across eight RTX PRO 6000 Blackwell GPUs. The environment was an LXC container running on a Proxmox VE host, a setup that had already consumed hours of debugging.

The journey began in the KVM virtual machine, where VFIO passthrough had created a "PHB" (PCIe Host Bridge) topology that prevented direct GPU-to-GPU P2P access. This caused severe latency penalties during tensor-parallel inference. The team migrated to an LXC container to bypass VFIO entirely, but that introduced a new blocker: CUDA initialization failed with error code 3 because the NVIDIA open kernel module's Heterogeneous Memory Management (HMM) feature was incompatible with the Proxmox kernel. The fix — setting uvm_disable_hmm=1 — was discovered only after extensive investigation.

With CUDA working, the assistant verified the GPU topology and confirmed that all eight GPUs had direct P2P access at 53 GB/s within the same NUMA node. This was the breakthrough they had been chasing. The next step was obvious: launch the sglang inference server and start benchmarking.

But the server wouldn't start.

The Error That Stopped Everything

When the assistant launched sglang with the same configuration that had worked in the KVM VM, the server immediately crashed. The traceback pointed to transformers.models.auto.configuration_auto raising a KeyError for the model type 'glm_moe_dsa'. This was not a standard Hugging Face model type — it was a custom architecture specific to the GLM-5 family, and loading it required the --trust-remote-code flag, which tells Hugging Face Transformers to execute the custom Python modeling files bundled with the model repository.

The assistant's debugging process was methodical. First, they verified that sglang was indeed passing trust_remote_code=True to the underlying Hugging Face calls. They inspected the sglang source code at multiple points — server_args.py, model_config.py, and hf_transformers_utils.py — confirming that the flag was being forwarded correctly. The code path was clean; the bug was not in sglang's argument handling.

Next, they checked whether the model's custom Python files existed in the Hugging Face cache. The model snapshot directory was at /root/.cache/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/. A quick ls *.py returned nothing — no Python files at all. This was the clue.

The Discovery in Message 592

Message [msg 592] is the moment this clue becomes a confirmed diagnosis. The assistant writes:

No Python files! The model snapshot doesn't have the custom modeling code. This was likely downloaded as LFS blobs only. Let me check what files are in the snapshot.

Then they run a detailed ls -la on the snapshot directory. The output reveals a directory full of symbolic links pointing to the Hugging Face Hub's LFS blob store:

lrwxrwxrwx ... chat_template.jinja -> ../../blobs/2ab98ef068d62829d17c5ade1827b9f013fa2bbf
lrwxrwxrwx ... config.json -> ../../blobs/5228d5805664fd337a31fbba683dc963049149ee
lrwxrwxrwx ... generation_config.json -> ../../blobs/ea7c83eafc45ce62fe0c934f5ca913022b6c9a8d
lrwxrwxrwx ... input_scales.safetensors -> ....

Every file in the snapshot was a symlink to a blob. The model weights (model-00001-of-00008.safetensors, etc.), the configuration files, the tokenizer data — all present. But the .py files that contain the custom model class definition — the code that trust_remote_code needs to execute — were absent.

The assistant's inference was sharp: "This was likely downloaded as LFS blobs only." They correctly identified that the model repository on Hugging Face Hub stores its Python modeling files as Git LFS objects, and the download process had only fetched the blobs for the weight files and configuration, not for the Python source. Without those source files, trust_remote_code has nothing to trust — it cannot load the model architecture because the architecture definition simply does not exist on disk.

Why This Happened

The Hugging Face Hub uses Git LFS (Large File Storage) to manage large binary files like model weights and safetensors. When a user runs huggingface_hub.snapshot_download() or uses transformers to download a model, the library fetches the Git repository and resolves LFS pointers to actual blobs. However, the behavior depends on how the model repository is structured.

In the GLM-5-NVFP4 repository, the custom Python modeling files (typically modeling_glm.py or similar) are stored as regular Git files or as LFS objects. If they are LFS objects and the download tool did not explicitly pull them — perhaps because it only recognized safetensors and JSON files as "needed" — they would be missing from the snapshot. The assistant's earlier check (in message [msg 591]) using ls *.py returned nothing, confirming this hypothesis.

This is a subtle but critical failure mode. The --trust-remote-code flag in Hugging Face Transformers works by looking for Python files in the model snapshot directory, importing them dynamically, and registering the custom model architecture. If those files are absent, the flag is useless — the model type remains unrecognized, and the KeyError is inevitable.

The Assumptions That Led Here

Several assumptions collided to create this situation. First, the assistant assumed that because the model had worked in the KVM VM, the cache was complete. In the KVM VM, the model was stored at /shared/huggingface/hub/ and had been downloaded earlier, possibly with a different tool or with additional steps like git lfs pull. When the model cache was transferred or re-downloaded in the LXC container, the Python files were silently omitted.

Second, the assistant assumed that --trust-remote-code was sufficient to handle any custom model type. This is generally true — but only if the custom code is actually present on disk. The flag is a permission mechanism, not a download mechanism. It tells Transformers "it is safe to execute the code in this directory," but it cannot conjure files that were never fetched.

Third, the Hugging Face Hub download logic itself made an assumption about which files are essential. The safetensors and JSON files were fetched because they are recognized as model data. The Python files, being smaller and potentially stored as regular Git objects, may have been skipped during the LFS blob resolution phase.

Knowledge Required to Understand This Message

To fully grasp what happened in message [msg 592], one needs several pieces of background knowledge:

Hugging Face Hub caching internals: The Hub stores model repositories as Git repositories with LFS-managed blobs. The cache directory structure uses symlinks from snapshot directories to a shared blob store. Understanding this layout is essential to interpret the ls -la output.

Git LFS mechanics: Large files in Hugging Face repositories are stored as pointers in Git, with the actual content in a separate LFS store. A download that does not explicitly pull LFS objects for all file types will leave some files as unresolved pointers or simply absent.

The trust_remote_code mechanism: In Hugging Face Transformers, custom model architectures require Python source files that define the model class. The trust_remote_code flag permits the dynamic import of these files from the model directory. Without the files, the flag is inert.

sglang server initialization flow: The server loads the model configuration early in its startup, during __post_init__ of the server arguments class. This happens before any user-facing error handling, so a missing model type crashes the entire server process.

Output Knowledge Created by This Message

Message [msg 592] transforms an opaque KeyError into a concrete, actionable diagnosis. Before this message, the error could have been caused by any number of issues: an incompatible Transformers version, a bug in sglang's argument forwarding, a corrupted model configuration, or a missing model type registration. After this message, the cause is clear: the model snapshot is incomplete.

This knowledge immediately suggests the solution path. The Python files must be obtained — either by re-downloading the model with explicit LFS pull for all file types, by copying the files from the KVM VM where they exist, or by manually fetching them from the Hugging Face repository. The assistant's next steps (in the following messages) explore these options, eventually discovering that the KVM VM's cache also lacks Python files, leading to a deeper investigation of how the model was originally loaded.

The Thinking Process on Display

The assistant's reasoning in this message is a textbook example of systematic debugging. The chain of inference proceeds as follows:

  1. Observe the symptom: Server crashes with KeyError: 'glm_moe_dsa'.
  2. Form a hypothesis: Perhaps trust_remote_code is not being passed correctly.
  3. Test the hypothesis: Inspect sglang source code at multiple call sites. Confirm the flag is forwarded.
  4. Refine the hypothesis: If the flag is correct, perhaps the model files themselves are missing.
  5. Test the refined hypothesis: Check for Python files in the snapshot directory. Find none.
  6. Interpret the result: The model snapshot lacks custom modeling code, likely due to incomplete LFS download. Each step eliminates a possible cause and narrows the search space. The assistant does not jump to conclusions — they verify the code path before blaming the file system. And when the ls *.py command returns nothing, they do not stop there; they run a full ls -la to see exactly what is present, producing the detailed symlink listing that confirms the diagnosis. The message also shows the assistant working within the constraints of the environment. They cannot browse the Hugging Face repository directly from the LXC container — they must infer the repository structure from the cache contents. The symlink pattern tells a story: the blobs directory contains the actual data, and the snapshot is just a collection of pointers. The missing Python files are not simply absent from the snapshot; they are absent from the blob store entirely, meaning they were never downloaded.

Broader Significance

This message is a reminder that even in highly automated environments, the assumptions baked into our tools can create invisible failure modes. The Hugging Face Hub's LFS-based caching is designed for efficiency — it deduplicates blobs and minimizes disk usage. But that efficiency comes at a cost: the caching logic must decide which blobs to fetch, and that decision is not always correct for every use case. Custom model code, which is small in size but critical in function, can fall through the cracks.

For the broader deployment effort, this message marks the transition from infrastructure debugging to model loading debugging. The CUDA issues, the P2P topology, the dependency versions — all of those were resolved. Now the problem is purely about data: the model's source code is not where it needs to be. It is a simpler problem than kernel module configuration, but it is no less blocking.

The assistant's response to this discovery — methodical, calm, and precise — is what makes the difference between a stalled deployment and a successful one. They do not panic, do not restart from scratch, and do not blame the tools. They identify the missing piece and prepare to acquire it. That is the mark of an effective debugger.