Reading the Blueprint: How One Bash Command Unlocked the Architecture of GLM-5-NVFP4

The Message

ssh 10.1.230.175 'source ~/ml-env/bin/activate && python3 -c "
import json
with open(\"/home/theuser/.cache/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/config.json\") as f:
    cfg = json.load(f)
for k in sorted(cfg.keys()):
    v = cfg[k]
    if isinstance(v, (int, float, str, bool)) or (isinstance(v, list) and len(v) < 5):
        print(f\"{k}: {v}\")
" 2>&1'

The output reveals a cascade of architectural parameters: _name_or_path: zai-org/GLM-5, architectures: [&#39;GlmMoeDsaForCausalLM&#39;], hidden_size: 6144, intermediate_size: 12288, first_k_dense_replace: 3, head_dim: 64, kv_lora_rank: 512, and many more.

This message, at first glance, appears to be a mundane debugging step — a Python one-liner to dump a JSON config file. But in the arc of this conversation, it represents a critical pivot point. The assistant, having been asked whether all experts could be replicated across all GPUs to avoid PCIe communication bottlenecks, hit a wall: the previous attempt to compute memory requirements failed with a KeyError: &#39;num_local_experts&#39;. The config file's structure was not what the assistant expected. This message is the act of stepping back, reading the raw source of truth, and rebuilding the mental model from first principles.

The Context That Made This Message Necessary

To understand why this message exists, we must trace the conversation that led to it.

The deployment of GLM-5-NVFP4 on eight RTX PRO 6000 Blackwell GPUs had been a saga of iterative debugging. The team had resolved NaN crashes during decode by selecting the correct NSA backends (trtllm), established baseline throughput of approximately 225 output tokens per second, and identified the fundamental performance bottleneck: the GPUs were connected only via PCIe, with no NVLink, running inside a Proxmox/KVM virtual machine that lacked direct GPU peer-to-peer support. Cross-GPU communication, required for every layer's all-reduce in tensor parallelism, was forced through host memory, creating a severe latency bottleneck.

In message 243, the user posed a pivotal question: "Would expert-parallel be faster here given it's 8x massive gpu but on pcie / no nvlink?" This question reframed the entire optimization problem. Instead of trying to tune around the PCIe bottleneck within the existing tensor-parallel configuration, perhaps a fundamentally different parallelism strategy could bypass it.

The assistant analyzed expert parallelism versus tensor parallelism in message 244, concluding that EP8 could potentially be significantly better on PCIe because MoE layers would use all-to-all (token routing) instead of all-reduce of full hidden states. However, this analysis depended on assumptions about the model's architecture — specifically, how many experts existed and how much memory they consumed.

Then came the user's truly creative proposal in message 247: "Can we have all experts on all gpus and dispatch each expert activation to one of the 8 gpus?" This was a hybrid strategy — replicate all experts on every GPU (avoiding any expert sharding), then load-balance individual expert activations across GPUs. It was essentially MoE data parallelism rather than expert parallelism.

The assistant's response in message 248 began analyzing this proposal but immediately hit a problem. The Python script attempting to compute memory requirements crashed with a KeyError: &#39;num_local_experts&#39;. The config key the assistant expected did not exist. This failure revealed an incorrect assumption: the assistant had assumed the config file would use a specific key name for the number of experts, but the actual GLM-5 configuration used a different schema.

What This Message Actually Does

Message 249 is the direct response to that failure. Rather than guessing at key names or continuing to work from assumptions, the assistant takes a radically simple approach: dump the entire config file, sorted by key, filtering only simple values. This is a reconnaissance mission.

The Python script is carefully constructed:

  1. It opens the raw config.json file directly from the Hugging Face cache directory, bypassing any model loading code that might transform or validate the config.
  2. It iterates over keys in sorted order, ensuring no parameter is missed.
  3. It filters to show only primitive types (int, float, str, bool) or short lists, excluding large nested structures like weight tensors or attention matrices that would clutter the output.
  4. It redirects stderr to stdout (2&gt;&amp;1) to capture any errors. The output reveals the true architecture of GLM-5. The model uses GlmMoeDsaForCausalLM as its architecture class. The hidden size is 6144, the intermediate size is 12288, and critically, there is no num_local_experts key — instead, the config reveals ep_size: 1, first_k_dense_replace: 3, and parameters like kv_lora_rank: 512 that hint at the MLA (Multi-head Latent Attention) architecture borrowed from DeepSeek.

Input Knowledge Required

To fully understand this message, a reader needs several layers of context:

Technical knowledge of MoE architectures: One must understand that Mixture-of-Experts models have multiple "expert" sub-networks, with each token being routed to a subset of experts. The number of experts, their size, and how they're distributed across GPUs are fundamental to performance analysis.

Understanding of parallelism strategies: Tensor parallelism (splitting each layer's computation across GPUs), expert parallelism (distributing experts across GPUs), and data parallelism (replicating the model and splitting the batch) are distinct strategies with different communication patterns. The conversation assumes familiarity with these concepts.

Knowledge of the GLM-5 model family: The model is a 744B-parameter MoE architecture with approximately 40B active parameters per token. It uses NVFP4 quantization (4-bit floating point) for the expert weights and BF16 for attention layers. Understanding that it has 256 experts, 61 layers, and uses MLA (from DeepSeek's architecture) is crucial.

The deployment environment: Eight RTX PRO 6000 Blackwell GPUs (96GB each), connected via PCIe inside a Proxmox VM, with no NVLink and no direct GPU peer-to-peer support. This topology makes cross-GPU communication the primary bottleneck.

The conversation history: The assistant had just failed to compute memory requirements because it assumed a config key (num_local_experts) that didn't exist. This message is the recovery from that failure.

Output Knowledge Created

This message produces concrete, actionable knowledge:

  1. The exact config schema of GLM-5-NVFP4: The output reveals that the model uses ep_size: 1 (not an explicit expert count), first_k_dense_replace: 3 (3 dense layers at the start, the rest are MoE), kv_lora_rank: 512 (MLA compression dimension), head_dim: 64, and hidden_size: 6144.
  2. The correct key names for future queries: Now the assistant knows to look for ep_size rather than num_local_experts, and understands the model uses a GlmMoeDsaForCausalLM architecture.
  3. The foundation for memory calculations: With hidden_size: 6144 and intermediate_size: 12288, the assistant can now compute expert MLP sizes. Each expert has three matrices (gate, up, down) of size hidden × intermediate = 6144 × 12288 ≈ 75.5M parameters each, totaling ~226.5M parameters per expert. In NVFP4 (4-bit), that's approximately 113MB per expert. With 256 experts across 58 MoE layers (61 total minus 3 dense), that's roughly 256 × 58 × 113MB ≈ 1.68TB of expert weights — far exceeding the 96GB per GPU.
  4. Confirmation that the user's proposal is infeasible: The output implicitly answers the user's question from message 247. Replicating all experts on every GPU would require each GPU to hold the full expert weight set, which is clearly impossible given the ~1.68TB total versus 96GB available per GPU. This is a negative result, but a crucial one — it rules out an entire class of optimization strategies.

Assumptions and Potential Mistakes

The message itself is clean — it's a straightforward data dump. But the assumptions that led to it are worth examining:

The assistant assumed the config schema was standard. The KeyError in message 248 occurred because the assistant expected a key like num_local_experts (common in DeepSeek/Mixtral-style configs), but GLM-5 uses ep_size and doesn't explicitly list the expert count. This is a reasonable assumption that happened to be wrong.

The assistant assumed it could compute memory from config alone. While the config provides dimensions, the actual memory footprint depends on quantization format (NVFP4 has variable compression ratios), weight sharing patterns, and implementation-specific overheads. The config dump gives dimensions, but precise memory calculations would require additional information about the NVFP4 format's scale factor storage.

The assistant assumed the Hugging Face cache path was correct. The hardcoded path includes a specific snapshot hash (6944a23f9ffb9668ac970901526c6cc72c34f4e2). If the model had been updated or the cache structure changed, this path would fail. It didn't, but it's a fragile assumption.

The output filtering may hide important information. The script filters out lists with 5+ elements and complex nested structures. This means it might miss critical parameters stored as arrays (e.g., eos_token_id: [154820, 154827, 154829] is shown because it has 3 elements, but a longer list of layer-specific parameters would be hidden).

The Thinking Process Visible in the Message

This message reveals a specific mode of reasoning: when analysis fails, go back to the raw data.

The assistant's thought process, reconstructed, goes something like this:

  1. "My previous attempt to compute expert memory failed because I assumed a key name that doesn't exist."
  2. "I don't know what keys this config actually has. Guessing is wasting time."
  3. "Let me read the entire config file, dump every key, and see what's actually there."
  4. "I'll sort the keys and filter for simple values so the output is readable."
  5. "From this raw dump, I can identify the correct parameter names and rebuild my memory model." This is a classic debugging pattern: when a mental model fails, discard assumptions and re-examine the ground truth. The assistant doesn't try to fix the key name by trial and error — it reads the entire schema at once, ensuring no other assumptions go uncorrected. The choice to use a Python one-liner over SSH rather than, say, reading the file with cat and parsing it manually, shows a preference for structured data extraction. The Python script handles JSON parsing, sorting, and filtering in one step, producing clean output that can be immediately analyzed.

Why This Message Matters in the Larger Arc

This message is a turning point in the optimization effort. Before it, the assistant and user were exploring creative parallelism strategies (expert parallelism, expert replication, data parallelism) based on incomplete architectural knowledge. After it, they have the ground truth: the model's exact dimensions, the correct config schema, and the data needed to definitively rule out certain approaches.

The message also demonstrates a methodological principle: when the map doesn't match the territory, redraw the map from survey data. The config.json is the canonical source of truth for the model's architecture. By reading it directly, the assistant bypasses any intermediate abstractions (model loading code, documentation, assumptions) that might introduce errors.

In the messages that follow this one, the assistant will use the config data to compute precise memory requirements, determine that expert replication is infeasible, and ultimately shift the optimization strategy toward different approaches. This single bash command, for all its apparent simplicity, provides the factual foundation for all subsequent architectural decisions.

The output also reveals something unexpected: ep_size: 1 in the config suggests the model was designed with expert parallelism in mind (the parameter exists), but defaults to no EP. The kv_lora_rank: 512 confirms MLA (Multi-head Latent Attention), which has implications for KV cache memory and attention computation patterns. The first_k_dense_replace: 3 indicates the first 3 layers are dense (not MoE), which affects the communication pattern — those layers use standard all-reduce even in an EP configuration.

Conclusion

Message 249 is a deceptively simple data-gathering step that embodies a crucial debugging methodology: when analysis fails, return to primary sources. The assistant, confronted with a KeyError from incorrect assumptions about the model config, doesn't patch the specific key — it dumps the entire schema and rebuilds its understanding from scratch. The output provides the architectural foundation for all subsequent optimization decisions, definitively answering the question of whether expert replication is feasible (it is not, given the memory constraints) and revealing the true structure of the GLM-5-NVFP4 model. In the larger narrative of deploying a 744B-parameter model on eight PCIe-bound GPUs, this message is the moment where speculation gives way to measurement, and assumptions yield to data.