A Pivotal Profiling Launch: Restarting vLLM with torch.profiler on 8x Blackwell GPUs

Introduction

In the course of a deep-dive performance analysis of the Kimi-K2.5 INT4 model running on 8x NVIDIA RTX PRO 6000 Blackwell GPUs, a single message marks the critical transition from indirect measurement to direct instrumentation. Message <msg id=2428> contains a single bash command — a carefully constructed invocation that restarts the vLLM inference server with the torch.profiler enabled. This is not merely a routine service restart; it is the culmination of a multi-phase benchmarking campaign, the moment where all prior measurements converge into a single decisive experiment designed to answer the fundamental question: what is actually consuming the GPU time during decode?

To understand why this message matters, one must appreciate the investigative path that led here. The assistant had already executed macro-level throughput benchmarks showing the system plateauing at approximately 1,536 tokens per second at high concurrency. Micro-benchmarks of individual Marlin W4A16 GEMM operations at exact Kimi-K2.5 dimensions had been collected. NCCL AllReduce burst measurements had been gathered, showing 6.81 milliseconds for the full set of 122 allreduce operations per decode step. But all of these were indirect measurements — component-level estimates that required assembly into a coherent picture. What was missing was a direct, kernel-by-kernel timeline of an actual decode step, captured by the CUDA profiler while the full model was loaded and running.

The Context of Discovery

The profiling campaign began with a clear plan outlined in <msg id=2407>: a three-phase approach moving from macro-level throughput tests to micro-benchmarks of individual operations, and finally to a deep torch.profiler capture. The assistant had successfully executed Phases 1 and 2. The macro benchmarks (Phase 1) revealed the system's throughput characteristics at various concurrency levels. The micro benchmarks (Phase 2) measured individual GEMM latencies and NCCL allreduce burst times. But Phase 3 — the torch.profiler deep dive — was blocked by a simple obstacle: the running vLLM instance had been started without the --profiler-config flag.

The assistant had attempted the HTTP profiler API in <msg id=2412>, receiving a {"detail":"Not Found"} response. This confirmed that the profiler endpoint is only registered when vLLM is launched with the appropriate configuration. There was no way to enable profiling on the running server without a restart. Given that model loading takes approximately 30 minutes for a 540GB parameter set spread across 8 GPUs, this was a significant operational decision: stop the production service, restart with profiling enabled, and wait for the model to load before capturing the critical data.

Anatomy of the Launch Command

The command in <msg id=2428> is dense with meaning. Let us examine its components in detail:

Shared Memory Cleanup

rm -f /dev/shm/psm_* /dev/shm/sem.mp-* /dev/shm/*vllm* /dev/shm/*nccl* 2>/dev/null

This cleanup targets several categories of shared memory artifacts. The psm_* files are remnants of NVIDIA's PSM (Performance Shared Memory) library, used for inter-process communication. The sem.mp-* files are POSIX semaphores created by multiprocessing operations. The vllm* and nccl* patterns catch any shared memory segments left behind by previous vLLM or NCCL instances. This cleanup reflects hard-won operational knowledge: shared memory segments persist across process lifetimes and can cause conflicts, port conflicts, or silent failures when a new instance attempts to initialize. The 2>/dev/null suppression of errors indicates this is a best-effort cleanup — the files may not exist, and that is fine.

NCCL Environment Variables

The NCCL configuration is the most technically significant part of the command:

NCCL_PROTO=LL: Selects the Low-Latency protocol for NCCL communications. This protocol is optimized for small message sizes — exactly the regime of Kimi-K2.5's allreduce operations, where each of the 122 allreduces per step carries only 14,336 bytes (hidden_size=7168 in BF16). The LL protocol minimizes per-message overhead at the cost of slightly higher per-byte bandwidth, which is the correct trade-off for this workload.
NCCL_ALGO=Ring: Selects the Ring allreduce algorithm. Ring allreduce is the most bandwidth-efficient algorithm for large clusters and is well-suited to the homogeneous PCIe topology of this 8-GPU system. Earlier NCCL tuning experiments in Segment 18 had explored alternatives like LL (Low Latency) protocol variants, and Ring emerged as the preferred choice.
NCCL_P2P_LEVEL=SYS: Forces NCCL to use system-level peer-to-peer communication rather than NVLink or NVSwitch paths. This is a critical setting for this hardware configuration: the RTX PRO 6000 Blackwell GPUs are connected via PCIe without NVLink bridges. Setting this flag prevents NCCL from attempting NVLink-based P2P that would fail or fall back silently.
NCCL_MAX_NCHANNELS=16 and NCCL_BUFFSIZE=16777216: These are tuned parameters from earlier NCCL benchmarking. The 16-channel configuration balances parallelism against PCIe bandwidth saturation. The 16MB buffer size (16777216 bytes) is a pragmatic choice that avoids excessive memory allocation while providing sufficient buffer space for the allreduce operations.
NCCL_NTHREADS=512: Allocates 512 CPU threads for NCCL communication operations. This is a high thread count that reflects the need to keep all 8 GPUs' communication pipelines fully occupied, especially given that CPU-side thread scheduling can become a bottleneck under heavy NCCL load.

CUDA and Environment Configuration

CUDA_DEVICE_MAX_CONNECTIONS=1

This setting limits the number of CUDA device connections, which can affect how CUDA streams and kernels are scheduled across multiple GPUs. The value of 1 is conservative and may reflect earlier debugging where higher values caused instability or performance regressions.

The Profiler Configuration

--profiler-config "{\"profiler\": \"torch\", \"torch_profiler_dir\": \"/tmp/vllm_profile\"}"

This JSON configuration enables vLLM's built-in profiler integration. The choice of "profiler": "torch" selects PyTorch's native profiler rather than the CUDA profiler (nsys). The torch profiler captures kernel-level timing data including CUDA kernel names, durations, launch counts, and memory operations. The output directory /tmp/vllm_profile will contain the trace files that can be analyzed with TensorBoard or the Chrome trace viewer.

The decision to use torch.profiler over nsys is pragmatic: torch.profiler can be enabled without restarting the container or modifying the CUDA environment, and it integrates naturally with vLLM's existing profiling hooks. A full nsys capture would require additional setup and might introduce more measurement overhead.

Background Execution Strategy

nohup ... > /tmp/vllm_profiler_launch.log 2>&1 &

The use of nohup and background execution (&) is essential because model loading takes approximately 30 minutes. The assistant cannot block on this command — it needs to proceed with other tasks, monitor the launch progress, and eventually run the profiling workloads once the server is ready. The log file captures both stdout and stderr for later inspection.

Decisions and Reasoning

Several implicit decisions are embedded in this message:

Restart the service rather than work without profiling data. The assistant could have attempted to infer the kernel breakdown from the micro-benchmark data alone, but chose instead to invest 30+ minutes of model loading time to get direct measurements. This reflects a commitment to evidence-based optimization over speculation.
Use torch.profiler rather than nsys. The torch profiler is lighter-weight and requires no special privileges or environment modifications. It integrates with vLLM's existing profiling infrastructure and produces trace files that are immediately analyzable.
Apply the tuned NCCL parameters from earlier experiments. Rather than using default NCCL settings, the assistant explicitly configures NCCL based on the learnings from Segment 18's NCCL tuning campaign. This demonstrates systematic knowledge accumulation across the session.
Clean shared memory proactively. This defensive measure prevents subtle failures that could waste the 30-minute model loading time.

Assumptions and Risks

The approach in this message rests on several assumptions:

The torch profiler will not significantly distort timing. Profiling instrumentation always introduces some overhead. The assumption is that this overhead is small relative to the kernel execution times and will not change the relative ranking of bottlenecks.
The NCCL environment variables are optimal. The chosen values are based on earlier tuning experiments, but those experiments were conducted on a different model (MiniMax-M2.5 FP8) and may not be perfectly transferable to Kimi-K2.5 INT4.
The model will load successfully with profiling enabled. There is always a risk that enabling the profiler triggers a different code path, exposes a latent bug, or causes an out-of-memory error due to profiler memory overhead.
The profiler output directory exists and is writable. The mkdir -p was done in the same command, so this is safe, but the directory path must be accessible from the vLLM process.

Knowledge Flow

Input knowledge required to understand this message includes: familiarity with vLLM's command-line interface and its --profiler-config option; understanding of NCCL environment variables and their effects on multi-GPU communication; knowledge of the hardware topology (8x PCIe-connected Blackwell GPUs without NVLink); awareness of the model's architecture (Kimi-K2.5 INT4 with hidden_size=7168, 61 layers, MoE); and the operational context of the profiling campaign.

Output knowledge created by this message includes: a running vLLM instance configured for profiling; a log file capturing the launch sequence; and ultimately, the profiler trace files that will reveal the exact kernel-level breakdown of decode time. These traces are the foundation for the bottleneck analysis that follows in subsequent messages.

The Broader Significance

This message represents the hinge point of the profiling campaign. Everything before it was preparation — macro benchmarks that showed what the system could do, micro benchmarks that measured how fast individual components could run. Everything after it will be analysis — examining the profiler traces to determine why the system performs the way it does.

The profiler data that this launch enables will reveal the shocking result: AllReduce consuming 51.5% of decode time (11.17ms per step), not the GEMMs that the micro-benchmarks had suggested would be the dominant cost. This discovery fundamentally reshapes the optimization strategy, leading away from kernel-level GEMM optimization and toward communication-level approaches like speculative decoding.

In the broader narrative of the session, this message embodies the tension between operational pragmatism and analytical rigor. The assistant could have settled for the indirect measurements and drawn plausible conclusions. Instead, it chose to invest significant time — 30 minutes of model loading plus the profiling runs — to obtain direct evidence. That investment paid off in the form of a clear, data-driven understanding of the true bottleneck, enabling the team to focus their optimization efforts where they would have the greatest impact.