The Quiet Pivot: Launching EP8 with a Memory-Safe Config

Message 1176 — a single bash command, barely a line long, that represents one of the most consequential decisions in a months-long optimization campaign. On its surface, it is mundane: an SSH invocation to start a server process on a remote machine. But context transforms it into a hinge point — the moment a methodical engineer, having just closed the book on one optimization path with clean data, pivots to re-engage a previously failed approach armed with new understanding.

The Message

[assistant] [bash] ssh root@10.1.230.174 'nohup bash /run_tp8_ep8_memsafe.sh > /root/sglang-server-ep8-memsafe.log 2>&1 &' && echo "EP8 memsafe server starting"
EP8 memsafe server starting

That is the entirety of the message. A single tool call. A single line of shell. Yet it arrives at a carefully prepared moment, after a cascade of prior work that made this invocation meaningful.

The Road to This Moment

To understand why this message was written, one must trace the thread back through the preceding hour of work. The assistant had been systematically testing optimization strategies for deploying the GLM-5-NVFP4 model — a massive Mixture-of-Experts (MoE) language model — across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The campaign was methodical: each idea was implemented, benchmarked against a clean baseline, and either adopted or discarded based on real measurements rather than speculation.

The immediate predecessor to this message was a clean A/B comparison of Opportunistic Expert Activation (OEA), a decode-time routing optimization inspired by a recent paper from Tri Dao et al. (arXiv:2511.02237). The assistant had implemented OEA in sglang's topk.py, carefully handling unsorted top-k output and proper sigmoid score weighting. The benchmarks were definitive: OEA showed near-zero average throughput improvement on random data (within ±0.3% of baseline), though it did boost peak throughput by ~5% at high concurrency. The conclusion was honest and evidence-based: OEA's benefit depends on non-uniform expert routing patterns, which random benchmark data does not exhibit. The technique was not wrong — it simply needed the right workload to shine.

With OEA documented and filed, the assistant turned to the next priority on the todo list: Expert Parallelism (EP8). This was not a fresh attempt. EP8 had been tried earlier in the campaign and had crashed — a failure that left the server unresponsive under moderate load. The root cause was never fully isolated, but the working hypothesis was memory pressure: with eight GPUs splitting the model's expert layers, the memory footprint per GPU was higher than expected, and the default memory fraction (--mem-fraction-static) left insufficient headroom for runtime allocations.

The Decision Framework

The message encodes a decision that was made implicitly across several preceding rounds. The assistant had to answer three questions:

First, should EP8 be retried at all? The previous crash was a significant setback. EP8 promised better scaling for MoE models by distributing experts across GPUs rather than replicating them, but the implementation had proven brittle. The assistant chose to retry because the theoretical ceiling was high — expert parallelism could potentially double throughput by reducing the per-GPU expert computation. The risk was acceptable because the memory-safe config could be tested without disrupting the baseline server.

Second, what should the new configuration be? The launch script (run_tp8_ep8_memsafe.sh) reveals the answer. Two parameters were changed from the previous attempt: --mem-fraction-static was reduced from its default (typically 0.85–0.90) to 0.75, and --max-running-requests was cut from 1024 to 512. These were conservative choices, deliberately leaving headroom. The script also preserved NCCL environment variables tuned for the Blackwell GPU topology: NCCL_IB_DISABLE=1 (no InfiniBand), NCCL_P2P_LEVEL=5 (NVLink peer-to-peer), and NCCL_MIN_NCHANNELS=8 (aggressive channel count for allreduce).

Third, when should the switch happen? The assistant had just completed single-stream and dual-stream benchmarks on the baseline TP8 (Tensor Parallelism) server, recording 10.36 tok/s at concurrency 1 and 19.29 tok/s at concurrency 2 — excellent linear scaling. Those benchmarks were the last data point needed from the baseline configuration. With that data collected, the baseline server was killed (message 1173), GPU memory was verified to be fully freed (message 1174, showing "0 MiB" on all GPUs), and the launch script was inspected (message 1175). Only then, with all preconditions satisfied, did the assistant issue the launch command in message 1176.

Assumptions Embedded in the Launch

Every engineering decision rests on assumptions, and this message is no exception. The assistant assumed that the previous EP8 crash was primarily a memory problem — that reducing the memory fraction from ~0.85 to 0.75 would provide enough headroom for the server to initialize and run stably. It assumed that the NCCL environment variables, which had worked well for TP8, would also be appropriate for EP8's different communication pattern. It assumed that the server would start within a reasonable time (the log file would be checked in subsequent messages), and that the reduced max-running-requests of 512 would prevent the request queue from overwhelming GPU memory during peak load.

These assumptions were reasonable but not all correct. As the chunk summary reveals, the EP8 server did load successfully — consuming 59.5 GB per GPU with approximately 34 GB free — but then crashed during warmup due to a different problem entirely: CUTLASS tile failures. The specific tile configuration (128×256×128) exceeded the SM120 architecture's 100 KB shared memory limit, a constraint that had nothing to do with memory pressure. The memory-safe config solved the OOM issue but revealed a deeper compatibility problem between the CUTLASS kernel templates and Blackwell's shared memory budget.

Input Knowledge Required

A reader cannot fully appreciate this message without understanding several layers of context. One must know what Expert Parallelism is — that EP8 splits the model's expert modules across eight GPUs so that each GPU computes only a subset of experts, reducing computation per GPU at the cost of increased communication. One must understand that the GLM-5-NVFP4 model uses 4-bit NVFP4 quantization for its weights, which dramatically reduces memory but also constrains the available kernel implementations. One must know that SM120 is the compute architecture of the Blackwell GPU, and that it has specific hardware limits (100 KB shared memory per SM, specific tensor core capabilities) that differ from the previous Hopper (SM90) architecture for which most existing kernels were written. And one must understand the sglang server architecture — the distinction between TP and EP, the role of mem-fraction-static in memory management, and the NCCL environment variables that control inter-GPU communication.

Output Knowledge Created

This message produced, first and foremost, a running server process. But more importantly, it produced a new data point in the optimization campaign. The subsequent messages would reveal whether the memory-safe config resolved the EP8 crash, and if not, what new failure mode emerged. The log file (sglang-server-ep8-memsafe.log) would contain the server's initialization trace, including model loading progress, memory allocation, kernel compilation, and any error messages. This output would inform the next round of decisions — whether to pursue EP8 further, try a different MoE backend (such as flashinfer_cutedsl, which the assistant began investigating in the very next message), or abandon expert parallelism entirely.

The Thinking Process

The reasoning visible in the surrounding messages reveals a disciplined experimental methodology. The assistant does not jump between optimizations randomly. Each idea is pursued to completion: implemented, benchmarked against a clean baseline, documented in the findings file, and then either promoted or discarded. The todo list is updated after each step, with statuses changed from "pending" to "in_progress" to "completed." The assistant verifies preconditions before acting — checking that the baseline server is still running before benchmarking, checking that GPU memory is freed before launching EP8, reading the launch script to confirm its contents before executing it.

This message, for all its brevity, is the product of that discipline. It is not a random command typed in haste. It is the next logical step in a carefully sequenced plan, executed only after all prerequisites were satisfied and all prior data was collected. The quiet confidence of "EP8 memsafe server starting" is earned — it comes from having done the work to know that this attempt, even if it fails, will produce useful information.

The Broader Narrative

In the arc of the optimization campaign, message 1176 represents a transition. The assistant had exhausted the low-hanging fruit: sglang updates had yielded a 2x throughput improvement, OEA had been tested and filed as "marginal for random data," and the baseline TP8 configuration was thoroughly characterized. The remaining optimization paths — EP8, flashinfer_cutedsl, L2 cache pinning, CUTLASS 4.4.0 integration — were all higher-risk, higher-effort interventions. Message 1176 is the first step down that harder road, a deliberate re-engagement with a previously failed approach, guided by the lessons learned from that failure.

It is, in essence, the moment the optimization campaign got serious.