The Ghost Workers: A Diagnostic Pivot in vLLM Performance Tuning

The Message

ssh -o StrictHostKeyChecking=no root@10.1.230.174 'nvidia-smi --query-compute-apps=pid,name --format=csv,noheader'
186520, VLLM::Worker_TP0
186521, VLLM::Worker_TP1
186522, VLLM::Worker_TP2
186523, VLLM::Worker_TP3
186524, VLLM::Worker_TP4
186525, VLLM::Worker_TP5
186526, VLLM::Worker_TP6
186527, VLLM::Worker_TP7

At first glance, this is a trivial command: query NVIDIA's System Management Interface for running compute processes and get back a list of process IDs and names. The output shows eight VLLM worker processes, one per GPU, each bearing the standard VLLM::Worker_TP<N> naming convention for tensor-parallel distributed inference. But in the arc of the conversation, this message represents a critical inflection point — a moment when the assistant's assumptions about system state collided with reality, forcing a diagnostic recalibration.

Context: The Performance Tuning Campaign

To understand why this message was written, we must trace the preceding sequence. The assistant had been engaged in an intensive performance optimization campaign for the GLM-5 model running on vLLM across eight NVIDIA RTX PRO 6000 Blackwell GPUs. The model, a massive 402GB GGUF-quantized checkpoint, required tensor parallelism across all eight GPUs simply to fit in memory. Over the course of multiple rounds, the assistant had systematically tested NCCL environment variable configurations — NCCL_NTHREADS=64, NCCL_BUFFSIZE=1MB, NCCL_ALGO=Ring, NCCL_MIN_NCHANNELS=1 — only to find that single-request decode throughput stubbornly plateaued at approximately 57.6 tokens per second regardless of tuning ([msg 85], [msg 86]).

Concurrent request benchmarking had revealed something interesting: aggregate throughput scaled well with concurrency (97.4 tok/s for 2 requests, 144.4 tok/s for 4 requests), suggesting the model had idle capacity during single-request decode. But the assistant's primary goal remained improving single-request latency, and all NCCL-based approaches had failed to budge the needle.

In [msg 93], the assistant made a decisive move: it attempted to kill the running vLLM server entirely, intending to restart with a different configuration — specifically, testing --optimization-level 1 to see if simpler compilation could reduce overhead. The kill command was aggressive: pkill -9 -f "vllm.entrypoints" followed by kill -9 $(nvidia-smi --query-compute-apps=pid --format=csv,noheader 2>/dev/null). The intent was clear — terminate everything, leave no survivors.

The Discrepancy: Memory That Wouldn't Let Go

In [msg 94], the assistant checked GPU memory usage after the kill attempt. The result was puzzling:

0, 92967 MiB
1, 92967 MiB
2, 92967 MiB
3, 92967 MiB
4, 92967 MiB
5, 92967 MiB
6, 92967 MiB
7, 92967 MiB

All eight GPUs still showed approximately 93 GB of allocated memory — essentially the same as when the server was running. This was deeply suspicious. If the kill had succeeded, memory should have been freed. The CUDA driver does not hold GPU memory allocations orphaned after a process exits; when a process terminates, its GPU memory mappings are cleaned up by the driver. Persistent memory allocation strongly suggested that processes were still alive.

This is the precise moment captured in the subject message. The assistant, confronted with evidence that contradicted its expectation (memory should be zero but wasn't), pivoted from nvidia-smi --query-gpu=memory.used (which only shows how much memory is used) to nvidia-smi --query-compute-apps=pid,name (which shows which processes are using it). This is a classic diagnostic escalation: when a summary statistic doesn't match expectations, drill down to the causal data.

What the Message Revealed

The output confirmed the worst-case scenario for the assistant's kill attempt: all eight VLLM worker processes were still running, with PIDs 186520 through 186527. The pkill -9 -f "vllm.entrypoints" command had targeted the API server process (which matches "vllm.entrypoints" in its command line), but the worker processes — spawned by the engine core and named VLLM::Worker_TP<N> — did not contain "vllm.entrypoints" in their process name. They were child processes of the engine core, and the pkill pattern simply didn't match them.

The second kill command — kill -9 $(nvidia-smi --query-compute-apps=pid --format=csv,noheader 2>/dev/null) — should have caught them. But the command had a subtle flaw: the 2>/dev/null suppressed error messages, and the nvidia-smi --query-compute-apps=pid output format may have included a header line or been empty at the moment of execution if the processes were between states. More likely, the command was issued too quickly after the pkill, and the workers hadn't yet been registered as compute apps at that instant, or the kill -9 was issued to a different set of PIDs than intended.

Assumptions and Their Consequences

This message exposes several assumptions the assistant was operating under:

Assumption 1: pkill -9 -f "vllm.entrypoints" would terminate the entire vLLM process tree. This assumption conflated the API server process with the worker processes. In vLLM's architecture, the API server (the process matching "vllm.entrypoints.openai.api_server") is the orchestrator, but the actual GPU workers are separate processes that may not share the same command-line pattern. The -f flag matches against the full process command line, and while the API server's command line contains "vllm.entrypoints", the workers' command lines typically contain "VLLM::Worker_TP" instead. The pkill only killed the parent, leaving orphaned workers alive.

Assumption 2: The second kill -9 command would catch any survivors. The assistant assumed that nvidia-smi --query-compute-apps=pid would return the PIDs of all GPU-using processes at that exact moment. But there's a timing window: between the pkill and the nvidia-smi query, the API server process dies, its children may be briefly in a zombie or transitioning state, and the compute apps list could be momentarily incomplete. The sleep 2 between commands may not have been sufficient for the system to stabilize.

Assumption 3: GPU memory would be freed if processes were killed. This is generally correct, but it depends on the processes actually being killed. The persistent memory allocation was the canary in the coal mine — it signaled that the kill hadn't fully taken effect.

Input Knowledge Required

To interpret this message, a reader needs several pieces of domain knowledge:

  1. vLLM's process architecture: vLLM uses a distributed architecture where the API server process spawns separate worker processes for each GPU in the tensor-parallel group. These workers are named VLLM::Worker_TP<N> and are visible to nvidia-smi as compute applications because they hold CUDA contexts and GPU memory allocations.
  2. nvidia-smi query capabilities: The --query-compute-apps=pid,name flag queries the list of compute applications currently running on the GPUs, returning their process IDs and names. This is distinct from --query-gpu=memory.used which only shows aggregate memory consumption.
  3. Process management on Linux: The pkill -9 -f <pattern> command sends SIGKILL to all processes whose full command line matches the pattern. The -f flag matches against the full process command line (from /proc/PID/cmdline), not just the process name.
  4. GPU memory lifecycle: When a CUDA process exits normally or is killed, the NVIDIA driver cleans up its GPU memory mappings. Persistent GPU memory allocation after a kill attempt strongly implies surviving processes.

Output Knowledge Created

This message produced concrete, actionable knowledge:

  1. The kill attempt failed: All eight workers were still alive. This explained the persistent GPU memory allocation.
  2. The exact PIDs of the survivors: 186520 through 186527. These could now be targeted directly with kill -9 <PID> for each.
  3. The pattern mismatch: The workers' process names (VLLM::Worker_TP<N>) did not match the pkill pattern (vllm.entrypoints), revealing a gap in the assistant's kill strategy.
  4. A need for a more robust cleanup approach: Future kill attempts would need to either target the workers by their specific name pattern or use the PIDs from nvidia-smi more carefully.

The Thinking Process

The reasoning visible in this message is a textbook example of diagnostic backtracking. The assistant had formed a hypothesis: "I killed the server, so memory should be freed." When evidence contradicted this hypothesis (memory still allocated), it didn't double down on the original assumption. Instead, it asked a more fundamental question: "Is my assumption about the server being dead actually correct?" This led to choosing a different diagnostic tool — one that directly answers the question "what processes are using the GPUs?" rather than "how much memory is being used?"

The choice of --query-compute-apps=pid,name over other available queries (like ps aux | grep vllm or checking process existence via /proc) is also revealing. The assistant could have checked process state via SSH's ps command, but chose nvidia-smi instead. This is because nvidia-smi provides a GPU-centric view: it only shows processes that actually hold GPU resources. A process might exist in the system's process table but no longer hold GPU memory (e.g., if it's in a zombie state or in the process of cleaning up). By querying compute apps specifically, the assistant got a direct answer to the operational question: "Are any processes still holding GPU memory?"

Broader Significance

While this message is brief — a single command and its output — it represents a critical juncture in the optimization workflow. The assistant had been pursuing a strategy of iterative NCCL tuning, benchmarking each configuration against the same 57 tok/s baseline. The decision to kill and restart with --optimization-level 1 was itself a strategic pivot: moving from "tune the communication layer" to "tune the compilation layer." But that pivot was blocked by the failed kill. This diagnostic message cleared the blockage by revealing the true state of the system, enabling the assistant to proceed with a more targeted cleanup.

In software engineering terms, this is the difference between a "lucky" debugging session and a "rigorous" one. A less careful approach might have accepted the memory allocation as a side effect of some CUDA caching mechanism or driver quirk, and proceeded to launch a new server anyway — potentially causing conflicts, port binding failures, or GPU out-of-memory errors. Instead, the assistant verified its foundational assumption before building on it.

The message also illustrates a broader principle of distributed systems debugging: when a system's reported state (memory used) doesn't match your model of what should be happening (processes killed), you must reconcile the discrepancy before proceeding. The cheapest debugging step is always to check your assumptions.