The Anatomy of a Model: Reading GLM-5-NVFP4's Configuration for Theoretical Performance Analysis

In the middle of an intensive optimization campaign for the GLM-5-NVFP4 model running on eight RTX PRO 6000 Blackwell GPUs, the assistant issued a seemingly simple command: read the model's config.json and print a dozen key parameters. This message, <msg id=1094>, is a quintessential example of a data-gathering step that appears mundane on the surface but carries immense strategic weight. It represents the pivot point between empirical optimization (trying things and measuring results) and theoretical analysis (computing fundamental performance ceilings). The command and its output would directly inform the assistant's ability to compute whether the model was HBM-bandwidth-bound, PCIe-allreduce-bound, or compute-bound — a question that had been the central mystery of the entire optimization campaign.

The Strategic Context

To understand why this message was written, one must appreciate the arc of the optimization work that preceded it. The assistant had spent dozens of rounds deploying the GLM-5-NVFP4 model, resolving CUDA initialization issues in an LXC container, enabling FlashInfer CUTLASS MoE autotune for SM120 (the Blackwell GPU architecture), and systematically testing optimization strategies including piecewise CUDA graphs, MSCCLPP allreduce, single-batch overlap, and expert parallelism. Each idea was implemented, benchmarked, and either adopted or discarded based on real measurements.

Most recently, the assistant had updated sglang to the latest commit (yielding a dramatic 2x throughput improvement at 256 concurrency), implemented Opportunistic Expert Activation (OEA) as a decode-time routing optimization, and retried Expert Parallelism (EP8) with a memory-safe configuration. The OEA implementation showed near-zero average throughput improvement on random data, confirming that its benefit depends on non-uniform expert routing patterns. EP8 loaded successfully but crashed during warmup due to CUTLASS tile failures — the 128×256×128 tile configuration exceeded SM120's 100KB shared memory limit.

Now the assistant was pivoting to a different kind of analysis: computing the theoretical maximum single-stream performance for this exact model and hardware combination. This required precise knowledge of the model's architectural parameters — not approximations or educated guesses, but the exact values straight from the source.

The Message Itself

The message executes a Python one-liner over SSH on the remote machine at root@10.1.230.174:

import json
with open("/shared/huggingface/hub/models--lukealonso--GLM-5-NVFP4/snapshots/6944a23f9ffb9668ac970901526c6cc72c34f4e2/config.json") as f:
    c = json.load(f)
    for k in ["n_group", "topk_group", "num_experts_per_tok", "n_routed_experts", "num_experts", "moe_intermediate_size", "norm_topk_prob", "n_shared_experts", "model_type", "scoring_func", "num_fused_shared_experts", "hidden_size"]:
        print(f"{k}: {c.get(k)}")

The output reveals the model's Mixture-of-Experts architecture in precise detail:

Why These Parameters Matter

Each of these values feeds directly into the theoretical maximum throughput calculation. The combination of hidden_size=6144, moe_intermediate_size=2048, n_routed_experts=256, and num_experts_per_tok=8 determines the compute and memory requirements for each MoE layer forward pass. The fact that n_group=1 and topk_group=1 means the router performs a simple top-8 selection across all 256 experts — no grouped masking or correction bias is needed, which simplifies the routing path but also means the model cannot exploit expert locality optimizations that grouped routing enables.

The small per-expert intermediate size (2048) relative to the hidden size (6144) means each expert's GEMM operations are relatively small — a 6144×2048 matrix multiply followed by a 2048×6144 matrix multiply per expert. With 8 experts per token and 256 experts total, the compute-to-communication ratio is low, making this architecture particularly sensitive to memory bandwidth and allreduce overhead. This is exactly the bottleneck the assistant had been struggling with throughout the optimization campaign: small per-expert GEMMs that cannot fully utilize the GPU's compute units.

The scoring_func: sigmoid is also significant. Many MoE models use softmax for routing, but sigmoid routing produces scores in [0,1] independently per expert rather than a probability distribution over experts. This has implications for how the OEA optimization would need to work — the sigmoid scores are already well-behaved for thresholding, which is exactly what OEA does.

The Path That Led Here

This message did not emerge from nowhere. It was the culmination of a specific sequence of reasoning visible in the preceding messages. The assistant had just finished implementing OEA and benchmarking EP8, and the chunk summary tells us it was "beginning to compute the theoretical maximum single-stream performance." To compute throughput ceilings, one needs the model's exact parameter counts, which derive from these architectural constants.

Earlier attempts to read the config had failed. In <msg id=1090>, the assistant tried using a wildcard path (snapshots/*/config.json) which failed because the shell glob wasn't expanding correctly in the Python string. In <msg id=1091>, it used find to locate the config file, and in <msg id=1093>, it discovered the actual snapshot hash: 6944a23f9ffb9668ac970901526c6cc72c34f4e2. Message <msg id=1094> uses this correct hash to finally read the config successfully.

Assumptions and Potential Pitfalls

The assistant made a reasonable assumption: that the model's config.json accurately reflects the architecture used at runtime. This is generally true for HuggingFace models, but there are edge cases. For instance, the num_fused_shared_experts field being None could mean the model doesn't use fused shared experts, or it could mean the field simply wasn't set in this config. The assistant would need to verify this against the actual model code in glm4_moe.py (which it had examined in <msg id=1089>).

Another assumption is that these parameters fully determine the model's compute and memory requirements. In reality, the attention mechanism, vocabulary size, number of layers, and other architectural details also matter. The assistant was gathering a specific subset of parameters relevant to the MoE layer analysis — it would need additional data for a complete theoretical model.

Input and Output Knowledge

Input knowledge required to understand this message includes: familiarity with Mixture-of-Experts architectures (routed experts, shared experts, top-k routing, grouped top-k), understanding of the GLM model family, knowledge of how hidden_size, moe_intermediate_size, and expert counts determine GEMM operation sizes, and awareness of the optimization context (the assistant was computing theoretical throughput ceilings).

Output knowledge created by this message is the precise set of architectural parameters that drive all subsequent analysis. These numbers would be used to compute:

The Thinking Process

The reasoning visible in this message and its surrounding context reveals a methodical, evidence-driven approach. The assistant did not guess the model parameters or rely on documentation. It went directly to the source — the actual config.json file on disk — and extracted exactly the values needed. When the first attempt failed due to a path error, it debugged by finding the correct snapshot directory. When it had the correct path, it printed only the relevant subset of parameters (12 out of potentially dozens in the config), showing clear intent and focus.

The choice of parameters to query also reveals sophisticated understanding. The assistant asked for n_group and topk_group to understand the routing topology, num_fused_shared_experts to check for fused expert optimization opportunities, scoring_func to understand the routing score semantics (relevant for OEA implementation), and norm_topk_prob to understand how weights are normalized. These are not random parameters — they are the specific architectural details that constrain optimization possibilities.

Conclusion

Message <msg id=1094> is a small but pivotal step in a larger intellectual journey. It represents the transition from empirical optimization ("try things and measure") to theoretical analysis ("compute the fundamental limits"). By extracting the model's architectural DNA, the assistant armed itself with the data needed to answer the question that had been haunting the entire campaign: is this model compute-bound, memory-bandwidth-bound, or communication-bound? The answer to that question would determine which optimization strategies were worth pursuing and which were fundamentally limited by hardware constraints. In the end, the assistant would use these parameters to compute that the theoretical maximum single-stream throughput was indeed within reach, confirming that the optimization campaign had been pushing against real hardware limits rather than software inefficiencies.