Reading the Source: How a Single `sed` Command Unraveled SGLang's Parallelism Hierarchy

In the middle of an intensive benchmarking session for the Kimi K2.6 model on 8× RTX PRO 6000 Blackwell GPUs, the assistant encountered a puzzling failure. The user had asked to try a tp4 ep2 configuration — tensor parallelism of 4 combined with expert parallelism of 2. It crashed with an out-of-memory error. The assistant then tried tp2 ep4, which crashed with a ZeroDivisionError. Two failed attempts, two different error modes, and no clear explanation for either. Then came a message that changed everything — not a complex algorithm or a lengthy analysis, but a single sed command that read 16 lines from SGLang's source code.

The Message

The subject message ([msg 11520]) is deceptively simple:

[assistant] [bash] ssh -o ConnectTimeout=5 root@10.1.2.200 "sed -n '1300,1315p' /root/venv_sglang211/lib/python3.12/site-packages/sglang/srt/entrypoints/engine.py" 2>&1
    # Parallelism hierarchy (outermost to innermost):
    # - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
    # - MoE: Global(TP) -> MOE_DP -> EP -> MOE_TP (innermost)
    attn_tp_size = server_args.tp_size // attn_dp_size // server_args.attn_cp_size
    attn_cp_rank = (tp_rank // attn_tp_size) % server_args.attn_cp_size
    moe_dp_rank = tp_rank // (server_args.tp_size // server_args.moe_dp_size)
    moe_ep_rank = (
        tp_rank
        % (server_args.tp_size // server_args...

This is an SSH command into the remote inference server (CT200, IP 10.1.2.200), using sed to extract lines 1300 through 1315 from SGLang's engine entrypoint file. The output reveals the parallelism hierarchy comment and the beginning of the rank calculation logic. The output is truncated mid-expression — the sed command only grabbed 16 lines — but it contains the critical insight.

The Debugging Journey That Led Here

To understand why this message matters, we must trace the path that led to it. The session had been systematically benchmarking parallelism strategies for Kimi K2.6, a Mixture-of-Experts (MoE) model with 384 experts running on 8 NVIDIA RTX PRO 6000 Blackwell GPUs connected via PCIe (no NVLink). The assistant had already tested three configurations:

TP8 (pure tensor parallelism across all 8 GPUs): 26 tok/s at C=1, peaking at ~808 tok/s. The AllReduce communication on every MoE layer was the bottleneck on PCIe.
PP8 (pipeline parallelism across 8 stages): 37 tok/s at C=1, but only 396 tok/s peak. Pipeline bubbles limited throughput.
EP8 (expert parallelism across 8 GPUs, with TP1 for attention): 65 tok/s at C=1, peaking at ~961 tok/s. This was the clear winner — by distributing the 384 experts across 8 GPUs (48 experts each), EP8 eliminated the AllReduce bottleneck on MoE layers, replacing it with a cheaper All-to-All dispatch pattern. The user then asked to try tp4 ep2 ([msg 11514]), a hybrid configuration that would use tensor parallelism within expert-parallel groups. The assistant launched this configuration ([msg 11515]) but it crashed with an OOM error ([msg 11516]): GPU 2 had 94.83 GiB allocated by PyTorch out of 94.97 GiB total, with only 136.94 MiB free. The assistant's reasoning in [msg 11517] shows confusion about the memory distribution — trying to calculate how many experts each GPU would hold under TP4 EP2, getting tangled in the arithmetic, and ultimately guessing that the OOM was caused by attention weights being replicated rather than sharded. The assistant then tried TP2 EP4 ([msg 11517]), which failed with a different error: ZeroDivisionError: integer division or modulo by zero ([msg 11518]). This was the more revealing failure — it wasn't a resource exhaustion issue but a logic error in the parallelism configuration itself. The assistant checked the journal logs ([msg 11519]) and saw the traceback originated from sglang/launch_server.py, line 69, but the full stack trace was truncated. At this point, the assistant had two data points and no working theory. The OOM suggested a memory layout problem; the ZeroDivisionError suggested a fundamental incompatibility between the requested parallelism dimensions. Rather than continue guessing, the assistant went to the source.

What the Source Code Revealed

The lines extracted by the sed command show the parallelism hierarchy as implemented in SGLang:

# Parallelism hierarchy (outermost to innermost):
# - Attention: Global(TP) -> DP -> ATTN_CP -> ATTN_TP (innermost)
# - MoE: Global(TP) -> MOE_DP -> EP -> MOE_TP (innermost)

This comment is the key. It reveals that SGLang does not treat tensor parallelism and expert parallelism as independent, orthogonal axes that can be combined arbitrarily. Instead, expert parallelism is nested inside tensor parallelism. The hierarchy for MoE layers is: Global TP → MoE Data Parallel → Expert Parallel → MoE Tensor Parallel. The tp_size parameter represents the total world size — the number of GPUs in the distributed group — and ep_size subdivides that group for the MoE layers.

The rank calculation code confirms this nesting:

moe_dp_rank = tp_rank // (server_args.tp_size // server_args.moe_dp_size)
moe_ep_rank = (
    tp_rank
    % (server_args.tp_size // server_args...

The moe_ep_rank is derived from tp_rank modulo the EP group size. This means ep_size must divide evenly into tp_size // moe_dp_size. With the default moe_dp_size=1, the constraint simplifies to: ep_size must be a divisor of tp_size.

The Incorrect Assumption

The assistant's fundamental mistake was treating tp_size and ep_size as independent parallelism dimensions. The user's request for tp4 ep2 was interpreted as "use 4-way tensor parallelism and 2-way expert parallelism," implying a total of 8 GPUs (4 × 2). But SGLang's API doesn't work that way. In SGLang, --tp-size is the total world size, and --ep-size subdivides that world. So --tp-size 4 --ep-size 2 means "use 4 GPUs total, with 2 expert-parallel groups of 2 GPUs each" — which would only use 4 GPUs, not 8. The OOM occurred because the weights intended for 8 GPUs were being loaded onto 4 GPUs.

The subsequent attempt, --tp-size 2 --ep-size 4, was even more wrong: it asked for 2 GPUs total with 4 expert-parallel groups, which is impossible because you can't have 4 groups across only 2 GPUs. This caused the ZeroDivisionError when the code tried to compute tp_size // moe_dp_size // ep_size = 2 // 1 // 4 = 0.

The Correct Mental Model

The source code reveals that for 8 GPUs, the valid configurations all use --tp-size 8 as the world size, with --ep-size being a divisor of 8:

TP8 EP1 (pure TP): moe_tp = 8, heavy AllReduce on every MoE layer. This was the 26 tok/s baseline.
TP8 EP2: 2 EP groups of 4 GPUs each. MoE layers use TP4 within each group. Reduces AllReduce cost by half.
TP8 EP4: 4 EP groups of 2 GPUs each. MoE layers use TP2 within each group.
TP8 EP8: 8 EP groups of 1 GPU each. No AllReduce on MoE layers — pure All-to-All dispatch. This was the 65 tok/s winner. The assistant's insight in the following message ([msg 11521]) synthesizes this correctly: "SGLang's EP is nested inside TP — tp_size is the world size, EP subdivides it. So valid combos for 8 GPUs are all --tp-size 8 --ep-size {1,2,4,8}."

Why This Message Matters

This message is a turning point in the debugging process. It represents the shift from guessing to understanding — from treating the system as a black box to reading its internals. The sed command itself is unremarkable (a simple line extraction), but its purpose is profound: it's the moment the assistant realizes that their mental model of SGLang's parallelism is wrong and goes to the source code to correct it.

The message also demonstrates an important debugging principle: when a system fails in ways you don't understand, stop guessing and read the code. The OOM error could have been interpreted as a memory capacity issue, leading to attempts to reduce memory usage. The ZeroDivisionError could have been seen as a bug. But together, and with the source code, they revealed a fundamental misunderstanding of the API.

The Broader Context

This debugging episode occurred within a larger effort to maximize inference throughput for Kimi K2.6 on PCIe-connected Blackwell GPUs. The assistant had already established that EP8 was the best configuration (65 tok/s at C=1, ~961 tok/s peak), and the user's request for TP4 EP2 was an attempt to find an even better hybrid. The source code reading confirmed that EP8 was indeed the optimal configuration for this hardware — the one that minimizes AllReduce communication on the PCIe bottleneck.

The lesson extends beyond this specific session: when deploying MoE models on multi-GPU systems, understanding the parallelism hierarchy of your inference engine is critical. SGLang's nesting of EP within TP means that the total world size must be set to the number of GPUs, and EP subdivides that world. This is different from frameworks where TP and EP are independent dimensions, and it has direct implications for memory layout, communication patterns, and ultimately throughput.

Conclusion

A single sed command reading 16 lines from a Python file may seem like a minor moment in a long coding session. But it was the diagnostic that resolved two failures, corrected a fundamental misunderstanding, and confirmed the optimal configuration for the deployment. It's a reminder that in complex systems, the fastest path to understanding is often to read the source code — and that the most important debugging tool is knowing where to look.

Reading the Source: How a Single sed Command Unraveled SGLang's Parallelism Hierarchy