The ZeroDivisionError That Rewrote a Parallelism Strategy

When an AI assistant deploys a large language model across eight GPUs, every configuration choice carries implications for memory, communication, and throughput. The search for the optimal parallelism strategy is a delicate dance of tradeoffs—and sometimes, the most informative error is not an out-of-memory crash but a humble ZeroDivisionError. This is the story of one such message: a simple bash script that checked whether a service had started, and instead returned a cryptic integer division error that forced the assistant to fundamentally revise its understanding of how SGLang combines tensor parallelism and expert parallelism.

The Message in Full

The subject message ([msg 11518]) is deceptively straightforward. After the assistant had attempted to launch a TP2 EP4 (tensor parallelism size 2, expert parallelism size 4) configuration for the Kimi K2.6 model, it ran a polling loop to check whether the service had started successfully:

for i in $(seq 1 90); do
    sleep 15
    st=$(ssh -o ConnectTimeout=5 root@10.1.2.200 "systemctl is-active sglang-k26-tp2ep4.service" 2>&1)
    if [ "$st" = "failed" ]; then
        echo "[$((i*15))s] FAILED"
        ssh -o ConnectTimeout=5 root@10.1.2.200 "journalctl -u sglang-k26-tp2ep4.service --no-pager -n 15 | grep -E 'Error|error|FAILED|OOM|assert' | tail -5" 2>&1
        break
    fi
    health=$(curl -s --max-time 5 "http://10.1.2.200:30001/v1/models" 2>/dev/null)
    if echo "$health" | grep -q '"id"' 2>/dev/null; then
        echo "[$((i*15))s] K2.6 TP2 EP4 READY!"
        break
    fi
    if [ $((i % 6)) -eq 0 ]; then echo "[$((i*15))s] loading..."; fi
done

The output was terse and devastating:

[15s] FAILED
May 26 00:07:18 dflash-train python[92616]: ZeroDivisionError: integer division or modulo by zero

A polling script that had worked flawlessly for EP8 and TP4 EP2 now returned an error that made no immediate sense. The service hadn't OOM'd—it had crashed during initialization with a mathematical impossibility.

The Context: A Search for the Perfect Parallelism Strategy

To understand why this message matters, we need to trace the conversation that led to it. The assistant and user had been systematically benchmarking the Kimi K2.6 model—a Mixture-of-Experts (MoE) architecture with 384 experts—across different parallelism strategies on an 8× RTX PRO 6000 Blackwell machine connected via PCIe.

The journey had already produced remarkable results:

TP8 (pure tensor parallelism): 26 tok/s at C=1, peaking at ~808 tok/s. Every token required AllReduce communication across all 8 GPUs for every layer, including the MoE layers. On PCIe, this communication overhead was crippling for single-request latency.
PP8 (pipeline parallelism): 37 tok/s at C=1, but only ~396 tok/s peak. The pipeline bubble—where GPUs sit idle waiting for the previous stage to finish—limited aggregate throughput.
EP8 (expert parallelism): A stunning 65 tok/s at C=1 and ~961 tok/s peak. By distributing the 384 experts across 8 GPUs (48 experts each), the assistant eliminated the AllReduce bottleneck on MoE layers. Each GPU processed its local experts independently, only communicating tokens via All-to-All dispatch rather than broadcasting weight gradients. The user then asked the assistant to try TP4 EP2 ([msg 11514]). The assistant interpreted this as a request for a hybrid strategy: 8 GPUs divided into 2 expert-parallel groups of 4 GPUs each, with tensor parallelism within each group. This would mean each GPU held 1/4 of the attention weights (via TP4) and 48 experts (192 experts per EP group, split across 4 GPUs via TP4). The result was an OOM error ([msg 11516])—GPU 2 had 93.6 GiB allocated out of 94.97 GiB capacity. The assistant's reasoning in [msg 11517] shows it trying to diagnose why: "TP4 means each GPU holds 1/4 of attention instead of 1/8, doubled memory for that." It concluded that the attention weights were the problem and pivoted to TP2 EP4, which would split attention across more GPUs (TP2 means each GPU holds 1/2 of attention weights, but with EP4 creating 4 groups of 2 GPUs each... or so the assistant thought).

The Critical Mistake: A Flawed Mental Model

The assistant's reasoning in [msg 11517] reveals a fundamental misunderstanding of how SGLang implements parallelism. The assistant believed that --tp-size and --ep-size were independent dimensions that multiplied to determine the total world size. Under this model, TP2 EP4 would use 2 × 4 = 8 GPUs, with 2 GPUs per tensor-parallel group and 4 expert-parallel groups across the cluster.

This assumption is natural and intuitive. Many distributed computing frameworks allow independent specification of parallelism dimensions. But SGLang's architecture works differently. In SGLang, --tp-size is the total world size—the number of GPUs participating in the model. Expert parallelism is nested within tensor parallelism: the EP size subdivides the TP group for MoE layers. The parallelism hierarchy, as revealed later in the source code ([msg 11520]), is:

Attention: Global(TP) → DP → ATTN_CP → ATTN_TP (innermost)
MoE: Global(TP) → MOE_DP → EP → MOE_TP (innermost)

The MoE tensor parallelism size is computed as tp_size // moe_dp_size // ep_size. With tp_size=2 and ep_size=4, this becomes 2 // 1 // 4 = 0—hence the ZeroDivisionError.

The assistant had been thinking about parallelism as a product (TP × EP = total GPUs), but SGLang treats it as a quotient (TP ÷ EP = MoE TP per group). This distinction is not merely semantic—it determines which configurations are valid. For 8 GPUs, the only valid configurations all use --tp-size 8, with --ep-size taking values that divide evenly into 8: 1, 2, 4, or 8.

Why the Error Was Inevitable—and Valuable

The ZeroDivisionError was not a random crash. It was a direct consequence of the assistant's incorrect mental model colliding with SGLang's actual implementation. The error message itself—"integer division or modulo by zero"—is unusually informative for a production system. It points directly to the line of code where the parallelism hierarchy is computed, and it reveals that the system attempted to compute moe_tp_size = tp_size // ep_size with ep_size > tp_size.

This error is valuable for several reasons. First, it's immediate—the service fails during initialization, not during inference after wasting time loading weights. Second, it's unambiguous—there's no ambiguity about whether the configuration is valid. Third, it forces the developer (or in this case, the AI assistant) to examine the actual parallelism model rather than relying on intuition.

The assistant's investigation in the subsequent messages ([msg 11519] through [msg 11521]) shows a productive debugging process. It first retrieved the full traceback, then examined the relevant source code in sglang/srt/entrypoints/engine.py, and finally reconstructed the correct parallelism hierarchy. The key insight—"ep_size must divide evenly into tp_size"—was extracted from the code itself.

Assumptions Made and Broken

This message reveals several assumptions the assistant was operating under:

Independence of parallelism dimensions: The assistant assumed TP and EP sizes were independent knobs that could be set arbitrarily. This was wrong—EP is nested within TP in SGLang's architecture.
Memory allocation intuition: The assistant assumed that TP4 EP2 OOM'd because attention weights were duplicated across fewer GPUs. While this was partially correct, the deeper issue was that the assistant didn't understand how SGLang partitions the model across the parallelism hierarchy.
Configuration validity: The assistant assumed that any combination of TP and EP sizes that summed to 8 GPUs was valid. The ZeroDivisionError proved otherwise.
The meaning of --tp-size: The assistant interpreted --tp-size as the tensor-parallel group size, when in SGLang it represents the total world size. These assumptions were not unreasonable—they reflect a common mental model of distributed inference where parallelism dimensions are orthogonal. The error served as a reality check, forcing the assistant to consult the actual source code rather than relying on analogy with other frameworks.

Input and Output Knowledge

To understand this message, a reader needs:

Knowledge of tensor parallelism (TP): Splitting individual layers across GPUs, requiring AllReduce communication after every layer.
Knowledge of expert parallelism (EP): Distributing MoE experts across GPUs, using All-to-All communication to route tokens to the GPU holding the required expert.
Knowledge of SGLang's architecture: Understanding that SGLang uses --tp-size as the world size and nests EP within TP.
Knowledge of the Kimi K2.6 model: A Mixture-of-Experts architecture with 384 experts, making EP particularly attractive.
Knowledge of systemd and service management: The polling script checks systemctl is-active and retrieves journal logs. The message creates new knowledge:
SGLang's parallelism hierarchy is not multiplicative but divisive: ep_size must divide evenly into tp_size.
Valid 8-GPU configurations are TP8 with EP ∈ {1, 2, 4, 8}: There is no TP4 EP2 or TP2 EP4.
EP8 was already optimal: The assistant's earlier EP8 benchmark (65 tok/s at C=1, ~961 tok/s peak) was using the best possible configuration for this hardware.
The ZeroDivisionError is a diagnostic signal: It points directly to the parallelism computation code, making debugging straightforward.

The Thinking Process in the Reasoning

The assistant's reasoning in [msg 11517] (the message immediately preceding the subject) shows a valiant but flawed attempt to diagnose the TP4 EP2 OOM. The assistant walks through the memory allocation:

"With TP4 EP2, each GPU needs to hold: 1/4 of attention weights (TP4), 192 experts (half of 384) split across 4 GPUs = 48 experts per GPU?"

It then catches itself: "But I need to figure out whether attention weights are replicated across the 2 EP groups or if they're also split by EP." This uncertainty is the crux of the problem. The assistant doesn't know whether attention is replicated or split across EP groups, and it doesn't have a clear mental model of the parallelism hierarchy.

The reasoning then pivots to a practical fix: "Let me try TP2 EP4 which keeps attention smaller." This is a reasonable heuristic—if TP4 caused OOM, TP2 should halve the per-GPU attention memory. But it's based on the same flawed assumption about how TP and EP combine.

The ZeroDivisionError in the subject message is the moment this assumption breaks. The assistant's reasoning after the error ([msg 11521]) shows a much more careful analysis, including reading the actual source code and reconstructing the correct hierarchy. The final conclusion—"EP8 was the sweet spot"—is not just a return to a previous configuration but a validated choice backed by correct understanding.

Broader Implications

This message is a microcosm of a larger pattern in AI-assisted development: the assistant makes reasonable but incorrect assumptions, encounters an error that seems unrelated to those assumptions, and must backtrack to discover the root cause. The ZeroDivisionError is a particularly elegant failure mode because it is:

Immediate: No wasted time loading weights or running inference.
Specific: Points directly to the parallelism computation.
Unambiguous: A division by zero cannot be misinterpreted. The message also highlights the importance of reading source code. The assistant's debugging process—from error message to traceback to source code inspection to corrected understanding—is a model of effective troubleshooting. The final answer in [msg 11521] shows a fundamentally revised understanding: "SGLang's EP is nested inside TP—tp_size is the world size, EP subdivides it."

Conclusion

The ZeroDivisionError in [msg 11518] is far more than a failed service startup. It is a diagnostic signal that exposed a flawed mental model, forced a deeper investigation, and ultimately led to a correct understanding of SGLang's parallelism architecture. The assistant's journey from confident but incorrect assumptions to validated, source-code-backed knowledge is a compelling example of how errors—even seemingly trivial arithmetic ones—can drive genuine learning.

In the end, the assistant returned to EP8, which had already proven to be the optimal configuration. But it returned with something more valuable than a working service: a correct model of how the system actually works. And that understanding would prove essential for the next phase of the project—deploying DFlash speculative decoding and pushing throughput even higher.