The Graceful Shutdown That Wasn't: A Study in Process Management During ML Infrastructure Tuning

Introduction

In the course of deploying and tuning a 744-billion-parameter Mixture-of-Experts model (GLM-5-NVFP4) across 8 NVIDIA RTX PRO 6000 Blackwell GPUs, a single message stands out as a quiet but revealing moment of infrastructure work. Message [msg 232] contains nothing more than a bash command:

[bash] ssh 10.1.230.175 'pkill -9 -f sglang; sleep 3; pgrep -af sglang || echo "All sglang processes stopped"'

On its surface, this is a simple process-kill command. But to understand why this message was written — and why it took the form it did — requires tracing the arc of the preceding minutes, where a carefully orchestrated benchmarking campaign had just concluded, and the assistant was preparing to restart the server with new tuning parameters. This message represents the critical transition point between measurement and optimization, and the specific command chosen reveals important lessons about process management in distributed ML systems.

The Context: A Hard-Won Victory and the Drive for Optimization

The story leading up to [msg 232] is one of persistent debugging triumph. For many rounds prior, the assistant had been wrestling with a catastrophic NaN crash during the decode phase of inference on the SM120 (Blackwell) GPUs. The model would load, the prefill would succeed, but the moment decoding began, the output would degenerate into garbage or crash entirely. After iterating through multiple attention backends — flashmla_kv, flashmla_sparse, and others — the breakthrough came in [msg 214]–[msg 218] with the discovery that the trtllm NSA (Native Sparse Attention) backends for both prefill and decode produced coherent, correct output on SM120 hardware.

With the model finally working, the assistant turned to performance characterization. Through a series of benchmarks using SGLang's bench_serving tool ([msg 219]–[msg 228]), a baseline was established: approximately 225 output tokens per second and 516 total tokens per second with 64 concurrent requests. The server log showed decode batch throughput peaking at around 208 tok/s. These numbers were respectable for a 744B MoE model running on PCIe-connected GPUs without CUDA graph optimization, but the user and assistant both recognized there was room for improvement.

The natural next step was tuning. The assistant identified two primary levers: increasing the memory fraction (--mem-fraction-static) from the current 0.85 to 0.92 to allocate more KV cache, and enabling CUDA graphs — a feature that had previously crashed but might now work with the newly discovered trtllm NSA backend. In [msg 229], the assistant presented these options to the user, and in [msg 230], the user approved proceeding with both tunings simultaneously.

The First Attempt: A Partial Shutdown

This brings us to [msg 231], the immediate predecessor of our subject message. The assistant issued:

ssh 10.1.230.175 'kill $(pgrep -f "sglang.launch_server" | head -1) 2>/dev/null; sleep 2; pgrep -af sglang || echo "Server stopped"'

This command attempted to kill only the sglang.launch_server process — a targeted, surgical approach. The output revealed the problem:

22760 [sglang::schedul] <defunct>
22761 [sglang::schedul] <defunct>
22762 [sglang::schedul] <defunct>
28585 bash -c kill ...

The launch_server process was gone, but three scheduler processes remained in a <defunct> (zombie) state, and the pgrep -af sglang check at the end actually matched its own bash process running the command, not any remaining server processes. The server was not fully shut down — or at least, not cleanly enough to be confident that a restart would work.

The Subject Message: A Forceful Correction

[msg 232] is the assistant's response to this partial failure. The command is deliberately more aggressive:

pkill -9 -f sglang

Every detail of this command reflects a lesson learned from the previous attempt:

pkill instead of kill: The previous command used pgrep to find the process ID, then kill to send SIGTERM (the default signal). pkill combines these steps, matching processes by name pattern and sending the signal directly. This is more robust because it doesn't depend on correctly parsing pgrep output or handling edge cases like multiple PIDs.

-9 (SIGKILL) instead of default SIGTERM: The previous attempt sent SIGTERM (signal 15), which asks processes to terminate gracefully. The defunct scheduler processes suggest that SIGTERM wasn't sufficient — the processes may have been stuck in an unkillable state or required SIGKILL to be fully reaped. SIGKILL cannot be caught or ignored by the process, ensuring termination.

-f flag: This matches against the full command line, not just the process name. This is critical because SGLang's Python processes may appear as python3 or python in the process name field, with the actual script name (sglang.launch_server, sglang::scheduler, etc.) only visible in the full command line. Without -f, pkill sglang would match nothing.

sleep 3: A three-second pause after the kill gives the kernel time to reap the processes and release resources (GPU memory, network ports, file handles) before the verification step.

Verification with pgrep -af sglang || echo "All sglang processes stopped": This provides clear confirmation. If any sglang processes somehow survived, pgrep would print their details. If none survived, the || fallback prints the success message.

The Result: Clean Slate

The follow-up in [msg 233] confirms the kill was successful:

28593 bash -c pgrep -af sglang || echo "Clean"; nvidia-smi --query-gpu=index,memory.used --format=csv,noheader
0, 0 MiB
1, 0 MiB
2, 0 MiB
3, 0 MiB
4, 0 MiB
5, 0 MiB
6, 0 MiB
7, 0 MiB

All 8 GPUs show 0 MiB memory used — the model weights, KV cache, and all other GPU allocations have been fully released. The system is ready for a clean restart with the new tuning parameters.

Deeper Analysis: Assumptions, Knowledge, and Decisions

This message reveals several important aspects of the assistant's reasoning:

Assumption about process naming: The assistant assumed that sglang appears in the command line of all SGLang-related processes. This turned out to be correct — the scheduler processes showed [sglang::schedul] in their command lines, making them matchable by pkill -f sglang. However, this is a fragile assumption; if SGLang had used different naming conventions (e.g., just python3 with no sglang identifier), the kill would have failed silently.

Assumption about SIGKILL safety: Using -9 is a nuclear option. It doesn't allow processes to clean up — close file descriptors, flush logs, release shared memory, or notify dependent services. In a production deployment, this could cause data loss or corruption. However, in this context — a development/tuning session where the server is about to be immediately restarted with new parameters — the risk is acceptable. The assistant implicitly judged that a clean shutdown wasn't necessary because no persistent state needed preservation.

Mistake in the first attempt: The earlier command in [msg 231] had a subtle bug. The pgrep -af sglang || echo "Server stopped" at the end would always match its own parent shell process (the bash -c ... invocation), because that shell's command line contained "sglang" (from the pgrep -f sglang pattern). This gave a false negative — the check appeared to show processes still running even when the server was actually gone. The assistant didn't catch this false positive, which may have contributed to the decision to use a more aggressive approach in [msg 232].

Input knowledge required: To understand this message, one needs to know: (a) how SGLang's server architecture works (launch_server process spawns scheduler processes), (b) the difference between SIGTERM and SIGKILL, (c) how pkill -f matches against full command lines, (d) that GPU memory is tied to process lifetime and won't be freed until all owning processes are killed, and (e) the context of the tuning workflow — that the server needs to be cleanly stopped before restarting with new flags.

Output knowledge created: This message produces a known-clean state: no sglang processes running, no GPU memory allocated. This is the precondition for the next step — restarting the server with --mem-fraction-static 0.92 and --enable-cuda-graph. The verification output in [msg 233] confirms this state, giving the assistant confidence to proceed.

The Thinking Process

The assistant's reasoning, visible across [msg 231]–[msg 233], follows a clear pattern:

1. Attempt gentle shutdown: Use kill with SIGTERM on just the launch process. This is the minimal-impact approach.
2. Observe incomplete result: The scheduler processes are defunct, memory is still allocated. The gentle approach didn't fully work.
3. Escalate: Switch to pkill -9 -f to force-kill everything matching "sglang". This is the maximal-impact approach.
4. Verify thoroughly: Check both process list and GPU memory to confirm complete cleanup. This escalation pattern — from targeted to broad, from gentle to forceful, from partial verification to comprehensive verification — is characteristic of robust infrastructure automation. The assistant didn't just repeat the same command; it recognized that the approach needed to change fundamentally.

Conclusion

Message [msg 232] is, on its surface, a trivial bash command. But in the context of the broader session, it represents a critical inflection point: the transition from measurement to optimization, enabled by the hard-won stabilization of the model on novel hardware. The command itself embodies lessons learned just seconds earlier, and its careful construction — the choice of pkill over kill, -9 over default signals, -f for full command matching, and the verification step — reflects a systematic approach to infrastructure management. For anyone working with large model serving infrastructure, this message serves as a compact case study in why process management matters and how small details in signal delivery can determine whether a tuning session proceeds smoothly or stalls on zombie processes and stuck GPU memory.