The Parallelism Hierarchy Revelation: How One Debugging Session Uncovered SGLang's Expert Parallelism Architecture

Introduction

In the high-stakes world of large language model (LLM) deployment, few decisions have as dramatic an impact on throughput as the choice of parallelism strategy. For Mixture-of-Experts (MoE) models like Kimi K2.6—a 384-expert behemoth running across 8× NVIDIA RTX PRO 6000 Blackwell GPUs—the difference between a well-chosen configuration and a poorly-chosen one can be a factor of 2.5× or more in tokens-per-second. This article examines a single message (index 11521) from an opencode coding session in which the assistant, after two failed attempts to deploy a hybrid tensor-parallelism/expert-parallelism configuration, reverse-engineered SGLang's parallelism hierarchy from source code, diagnosed the root cause of a ZeroDivisionError, and produced a corrected mental model that fundamentally reshaped the deployment strategy.

The message is remarkable not for its length—it is a concise assistant response—but for the depth of the reasoning process it reveals. In the span of a few paragraphs of agent reasoning, the assistant moves from confusion about why --tp-size 2 --ep-size 4 failed, through a careful reading of SGLang's engine source code, to a precise understanding of how expert parallelism nests inside tensor parallelism in SGLang's architecture. This moment of insight, captured in the message's "Agent Reasoning" section, represents a genuine debugging breakthrough: the realization that tp_size in SGLang is not the tensor-parallelism degree but the world size, and that ep_size subdivides that world size for MoE layers, creating a strict divisibility constraint that the user's attempted configurations had violated.

The Sequence of Events

To understand message 11521, one must understand the chain of events that led to it. The session had been benchmarking Kimi K2.6 across multiple parallelism strategies on an 8-GPU PCIe-connected machine. Earlier messages had established the following baseline performance:

• TP8 (pure tensor parallelism across all 8 GPUs): 26.3 tok/s at C=1, peaking at ~808 tok/s aggregate
• PP8 (pipeline parallelism across 8 stages): 36.9 tok/s at C=1, but only ~396 tok/s peak due to pipeline bubbles
• EP8 (expert parallelism, each GPU holds 48 experts): 65.3 tok/s at C=1, peaking at ~961 tok/s EP8 had emerged as the clear winner, especially for single-request latency. The user, however, was not satisfied with a single data point. In message 11514, the user issued a simple command: "try tp4 ep2." This was a natural experiment: perhaps a hybrid approach—tensor parallelism across groups of 4 GPUs combined with expert parallelism across 2 groups—could combine the benefits of both strategies. The assistant dutifully deployed TP4 EP2 (message 11515), creating a systemd service with --tp-size 4 --ep-size 2. The service failed immediately with a CUDA out-of-memory error (message 11516): GPU 2 had 94.83 GiB allocated out of 94.97 GiB total, leaving only 136.94 MiB free. The assistant's initial analysis (message 11517) attributed this to TP4 requiring each GPU to hold 1/4 of the attention weights instead of 1/8, doubling the memory footprint. The assistant then pivoted to TP2 EP4 (--tp-size 2 --ep-size 4), hoping that smaller TP groups would reduce the per-GPU memory pressure. The TP2 EP4 deployment (message 11518) failed even faster—15 seconds—but with a different error: ZeroDivisionError: integer division or modulo by zero. This was not a memory problem. This was a logic error in SGLang's parallelism initialization code. The assistant, now alerted that something deeper was wrong, inspected the source code (message 11520), reading lines 1300–1315 of SGLang's engine.py. This revealed the parallelism hierarchy comment and the critical computation:

# Parallelism hierarchy (outermost to innermost):
# - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
# - MoE: Global(TP) -> MOE_DP -> EP -> MOE_TP (innermost)

This source code inspection set the stage for message 11521—the subject of this article—in which the assistant synthesized the findings into a corrected understanding.

The Message: A Debugging Epiphany

Message 11521 is the assistant's response after reading the source code. It contains both the agent's reasoning (the internal thought process) and the final response to the user. The reasoning section is where the real action happens:

I see the issue. The parallelism hierarchy is: - Global TP size = tp_size (this is the total world size for the model) - EP is nested within TP: tp_size // moe_dp_size // ep_size = MoE TP size per EP group

>

So with tp_size=2, ep_size=4: tp_size // moe_dp_size // ep_size = 2 // 1 // 4 = 0 → division by zero.

This is the key insight. The assistant realizes that tp_size in SGLang is not the tensor-parallelism degree in the conventional sense—it is the total world size (number of GPUs). Expert parallelism is not a separate dimension of parallelism but a subdivision within the TP world. The computation tp_size // moe_dp_size // ep_size computes the MoE tensor-parallelism size per EP group, and when ep_size exceeds tp_size // moe_dp_size, integer division yields zero, triggering the ZeroDivisionError.

The reasoning continues, enumerating the valid configurations:

For 8 GPUs, the valid configurations are TP8 EP8 (each GPU its own EP group), TP8 EP4 (4 groups of 2 GPUs with TP2 per group), TP8 EP2 (2 groups of 4 GPUs), or TP8 EP1 (pure TP without EP).

The assistant then refines this understanding, noting that all valid configurations have tp_size=8:

Actually, I'm realizing the global tp_size must be 8 for 8 GPUs since --tp-size in SGLang represents the world size, and EP subdivides within it. So the valid configurations all have tp_size=8: TP8 EP1 uses AllReduce for everything, TP8 EP2 splits experts into 2 groups of TP4 each, TP8 EP4 creates 4 groups of TP2 each, and TP8 EP8 fully distributes experts so each GPU handles 48 experts.

The final response to the user is concise and actionable:

SGLang's EP is nested inside TP — tp_size is the world size, EP subdivides it. So valid combos for 8 GPUs are all --tp-size 8 --ep-size {1,2,4,8}. EP8 was the sweet spot (no TP AllReduce for experts). There's no TP4 EP2 — you'd need --tp-size 8 --ep-size 4 to get 4 EP groups with TP2 for experts within each group.

The Reasoning Process: A Window into Debugging

The "Agent Reasoning" section of message 11521 is particularly valuable because it shows the assistant's thought process evolving in real time. The reasoning begins with a clear statement of the discovered parallelism hierarchy, then works through the arithmetic that caused the ZeroDivisionError. But then something interesting happens: the assistant catches itself mid-thought.

The first paragraph states the constraint correctly: ep_size must divide evenly into tp_size // moe_dp_size. But the second paragraph begins with "Actually, I'm realizing..."—a self-correction. The initial framing ("TP8 EP4 (4 groups of 2 GPUs with TP2 per group)") was slightly imprecise. The assistant reframes: all valid configurations have tp_size=8, and EP subdivides within that. The naming convention shifts from TP{N} EP{M} to TP8 EP{M}, reflecting the correct understanding that TP size is always the world size.

This self-correction is a hallmark of good debugging. The assistant did not simply parrot the source code comment; it worked through the implications, found an inconsistency in its own framing, and corrected it before presenting the final answer. The reasoning also shows the assistant considering the user's perspective: "The user should try TP8 EP4 for attention tensor parallelism across 8 GPUs with expert dispatch split into 4 groups of 2 GPUs each, or TP8 EP2..."

The final response distills this complex reasoning into a single, clear statement: "There's no TP4 EP2 — you'd need --tp-size 8 --ep-size 4 to get 4 EP groups with TP2 for experts within each group." This translation from the user's intuitive mental model (TP4 EP2 = "tensor parallelism of degree 4, expert parallelism of degree 2") to SGLang's actual semantics (TP8 EP4 = "world size 8, 4 EP groups, each with TP2 for experts") is the core contribution of the message.

Assumptions and Mistakes

Several assumptions were made—and corrected—in the journey to message 11521.

The user's assumption was that --tp-size and --ep-size are independent dimensions of parallelism, like specifying two orthogonal axes. This is a natural assumption: in many distributed training frameworks (e.g., DeepSpeed, Megatron-LM), tensor parallelism and expert parallelism are independent dimensions that multiply to determine the world size. The user's command "try tp4 ep2" implicitly assumed that tp_size * ep_size = 8 (4 × 2 = 8), which would be the correct formula if the two were independent.

The assistant's initial assumption (in message 11517) was that the TP4 EP2 failure was purely a memory issue—that TP4 required each GPU to hold 1/4 of attention weights instead of 1/8, causing the OOM. This was a reasonable hypothesis given the error message, but it was incomplete. The assistant did not yet understand the parallelism hierarchy and assumed the configuration was valid but memory-intensive.

The corrected understanding revealed that both assumptions were wrong. SGLang's tp_size is the world size, not the TP degree. The formula is not tp_size * ep_size = world_size but rather tp_size = world_size, with EP subdividing within. The user's TP4 EP2 was actually interpreted as world_size=4, ep_size=2, which only uses 4 GPUs (not 8) and would leave 4 GPUs idle. The assistant's TP2 EP4 attempt was interpreted as world_size=2, ep_size=4, which causes the division-by-zero because ep_size exceeds world_size.

A subtle mistake in the assistant's reasoning is the initial statement that "ep_size must divide evenly into tp_size // moe_dp_size." This is technically correct but potentially misleading. The real constraint is that tp_size // moe_dp_size // ep_size must be at least 1 (since it represents the MoE TP size per EP group). If ep_size > tp_size // moe_dp_size, the result is 0. The assistant correctly identifies the division-by-zero but initially frames it as a divisibility constraint rather than a size constraint.

Input Knowledge Required

To fully understand message 11521, one needs knowledge in several areas:

Distributed parallelism concepts: Understanding the difference between tensor parallelism (splitting individual tensor operations across GPUs), pipeline parallelism (splitting model layers across GPUs), and expert parallelism (distributing MoE experts across GPUs) is essential. The message assumes familiarity with AllReduce (the communication pattern used in TP) and All-to-All (used in EP), and how these communication patterns interact with hardware topology (PCIe vs NVLink).

SGLang's architecture: The message references specific SGLang source code (engine.py lines 1300–1315) and command-line flags (--tp-size, --ep-size, --moe-dp-size). Understanding that SGLang uses a nested parallelism hierarchy (where EP subdivides the TP world) rather than an orthogonal one is critical—and this is precisely what the message teaches.

Mixture-of-Experts models: The Kimi K2.6 model has 384 experts, and the message references "48 experts per GPU" (384 ÷ 8). Understanding that MoE layers route each token to a subset of experts (e.g., top-8) and that expert parallelism distributes the expert weights across GPUs is necessary to appreciate why EP8 was the "sweet spot."

Integer arithmetic edge cases: The ZeroDivisionError arises from 2 // 1 // 4 = 0 in Python integer division. Understanding that integer division truncates toward negative infinity (floor division) and that 2 // 4 = 0 is the root cause helps connect the source code to the runtime error.

Output Knowledge Created

Message 11521 creates several pieces of valuable knowledge:

A corrected mental model of SGLang's parallelism: The most important output is the understanding that --tp-size in SGLang is the world size, and EP subdivides within it. This is counterintuitive for anyone coming from frameworks where TP and EP are independent. The message explicitly translates between the user's intuitive notation (TP4 EP2) and SGLang's actual semantics (TP8 EP4).

A complete enumeration of valid configurations: For 8 GPUs, the message lists all valid configurations: TP8 EP1, TP8 EP2, TP8 EP4, TP8 EP8. It explains what each means in terms of communication patterns (AllReduce vs All-to-All) and per-GPU expert counts.

A debugging methodology: The message demonstrates how to diagnose a ZeroDivisionError in a distributed serving framework: inspect the source code that computes parallelism dimensions, trace the arithmetic, and understand the hierarchy. This methodology is transferable to other frameworks and other errors.

A performance recommendation: The message reaffirms that EP8 was the best configuration (65 tok/s at C=1, 961 tok/s peak) and explains why: "no TP AllReduce for experts." This connects the architectural understanding (EP eliminates AllReduce on MoE layers) to the empirical results.

A naming convention clarification: The message establishes that the correct naming for configurations is always TP{world_size} EP{ep_size}, not TP{tp_degree} EP{ep_degree}. This prevents future confusion when discussing parallelism strategies.

Broader Significance

Message 11521 is more than a bug report or a configuration correction. It represents a genuine moment of knowledge creation in a human-AI collaborative debugging session. The assistant did not have this knowledge before reading the source code—it discovered it through the same process a human engineer would: form a hypothesis, test it, observe the failure, inspect the code, and synthesize the findings.

The message also illustrates an important principle in distributed systems: the mapping between intuitive names and actual semantics is not always what it seems. The flag --tp-size sounds like it specifies the tensor-parallelism degree, but in SGLang's architecture it specifies the world size. The flag --ep-size sounds like it specifies the expert-parallelism degree, but it actually specifies the number of EP groups that subdivide the TP world. These semantic mismatches are common in complex systems and are a frequent source of configuration errors.

Finally, the message demonstrates the value of reading source code. The assistant could have continued guessing configurations, trying random combinations until something worked. Instead, it invested a few seconds in reading SGLang's engine.py and obtained a definitive answer. This is a lesson for both human engineers and AI assistants: when configuration errors produce cryptic errors like ZeroDivisionError, the fastest path to resolution is often to trace the computation in the source code rather than to guess new parameter values.

The message also sets the stage for the next phase of the session. With the parallelism hierarchy understood, the user asks in the following message (11522): "Theorise what would be the best setup to get max throughput out of this model on those cards." The assistant can now answer this question with a precise understanding of the available configurations, their communication patterns, and their memory footprints—knowledge that was created in message 11521.