From Environment Setup to Speculative Decoding: A Full-Stack ML Engineering Journey on 8-GPU Blackwell

Introduction

The gap between a model architecture on paper and a production inference deployment at scale is measured not in lines of code but in the sheer volume of unforeseen obstacles that emerge along the way. This article traces a single, sprawling opencode coding session that exemplifies this journey in its entirety: from installing NVIDIA drivers on a fresh Ubuntu 24.04 system, through resolving complex CUDA build issues for flash-attention, to deploying the GLM-5 large language model across eight RTX PRO 6000 Blackwell GPUs, and ultimately attempting to break through a communication-bound throughput ceiling using speculative decoding. The session is a masterclass in systematic debugging, dependency management, and the art of knowing when to pivot.

Phase 1: Building the Foundation

The session begins with a user task that is deceptively simple: set up a full ML environment on a remote Ubuntu 24.04 machine. But as anyone who has provisioned a multi-GPU Linux system knows, the devil is in the details. The assistant starts by installing the latest NVIDIA drivers (version 590.48.01) and CUDA Toolkit 13.1, configuring environment paths, and verifying that the two RTX PRO 6000 Blackwell GPUs are operational. This initial phase establishes connectivity and confirms the hardware baseline.

A Python virtual environment is created using uv, a modern package manager, and core packages like PyTorch are installed. This seems straightforward — until the assistant attempts to build flash-attn from source, which becomes a major bottleneck. The system's CUDA 13.1 conflicts with PyTorch's CUDA 12.8 base, requiring the installation of a secondary CUDA 12.8 toolkit. The build process repeatedly exhausts system memory, forcing a collaborative trial-and-error reduction of parallel compilation jobs (MAX_JOBS) from 128 down to 20. This is only resolved after the machine is rebooted with an expanded 432GB of RAM. Further dependency conflicts arise when vLLM downgrades PyTorch, breaking the compiled flash-attn binary, necessitating a targeted rebuild against the correct PyTorch version (2.9.1).

This phase alone — spanning driver installation, CUDA toolkit management, virtual environment creation, and flash-attn compilation — is a microcosm of the challenges facing anyone deploying modern ML infrastructure. Each step reveals a new assumption that must be corrected: that the latest CUDA toolkit is compatible with PyTorch, that default compilation parallelism is safe, that dependency resolution across packages is consistent. The assistant's methodical approach — observe failure, diagnose root cause, apply targeted fix, verify — establishes a pattern that will persist throughout the entire session.

Phase 2: The Performance Ceiling

With the environment stabilized — PyTorch 2.9.1, flash-attn 2.8.3, vLLM 0.15.1 — the machine is upgraded from 2 to 8 GPUs, and the assistant deploys the GLM-5-NVFP4 model using a nightly build of SGLang. The baseline performance is measured: approximately 43 tokens per second for single-request decoding with tensor parallelism across all 8 GPUs and CUDAGraph acceleration.

But profiling reveals a brutal reality: 87% of decode time is consumed by NCCL allreduce operations traversing PCIe interconnects. The eight RTX PRO 6000 Blackwell GPUs, despite being professional-grade hardware, are connected via PCIe rather than NVLink. Every forward pass requires expensive cross-GPU synchronization, and the communication overhead dominates the computation. This is a classic scaling pathology — adding more GPUs increases communication cost, and at some point the marginal throughput gain from additional parallelism is eaten by synchronization overhead.

The user's strategic response is captured in a single, remarkably dense message ([msg 0]) that serves as the blueprint for the entire investigation [1]. The message proposes speculative decoding as the primary approach to amortize the allreduce cost: by verifying multiple candidate tokens per forward pass, each expensive synchronization step produces more output tokens. Three approaches are outlined: ngram-based speculation (which requires no separate draft model), Eagle speculation or prompt lookup as fallbacks, and native Multi-Token Prediction (MTP) using GLM-5's built-in nextn layer. The user also suggests investigating whether vLLM's custom allreduce implementation can be force-enabled as a parallel investigation thread.

Phase 3: Codebase Reconnaissance

Before touching any configuration parameters or restarting the server, the assistant invests significant effort in understanding the vLLM codebase's capabilities. This reconnaissance phase, spanning messages 1 through 12, is a hallmark of mature engineering practice: understand the system before modifying it.

The assistant first confirms SSH connectivity and identifies the running vLLM server ([msg 1]) [10]. It then searches for speculative decoding configuration options across the vLLM v1 codebase, discovering references to speculative configuration in the CUDAGraph dispatcher and structured output modules ([msg 2]) [6]. This initial broad search is followed by more targeted investigations into GLM-specific MTP support.

The discovery is significant: vLLM's speculative configuration file contains explicit entries for glm4_moe_mtp, glm4_moe_lite_mtp, and glm_ocr_mtp model types ([msg 4]) [35]. The assistant drills deeper, reading the exact code that maps HuggingFace architecture names to vLLM's internal speculative decoding pipeline ([msg 5]) [15]. The critical finding is that when vLLM encounters a model whose HuggingFace architecture is Glm4MoeForCausalLM, it automatically rewrites the model type to glm4_moe_mtp and reads the num_nextn_predict_layers attribute to configure the MTP head. This is first-class, native support — not a hack or an afterthought.

The user had noted that the GGUF file contained 27 unmapped tensors from blk.78, which were the nextn/MTP layer weights that had been intentionally skipped during loading. The discovery that vLLM explicitly supports this architecture means those weights could potentially be activated, providing a draft model already baked into the original model weights.

Phase 4: Tracing the Allreduce Gate

Parallel to the speculative decoding investigation, the assistant pursues the user's secondary question: can vLLM's custom allreduce be force-enabled? This thread becomes a deep dive into vLLM's distributed communication architecture.

The assistant begins by searching for all references to custom allreduce across the codebase ([msg 6]) [41], finding the configuration flag disable_custom_all_reduce and the set_custom_all_reduce function. It then examines the platform-level is_fully_connected method in cuda.py ([msg 8]) [4], which queries NVML to check whether a set of GPUs are connected by NVLink within one hop. The platform layer unconditionally returns True for use_custom_allreduce() — it asserts capability. But the actual runtime decision happens deeper.

The critical discovery comes in messages 11 and 12. The assistant reads the custom allreduce initialization code in custom_all_reduce.py ([msg 11]) [8], finding the exact conditional: the code gathers physical device IDs across all ranks using dist.all_gather, then tests NVLink connectivity. The comment is explicit: "test nvlink first, this will filter out most of the cases where custom allreduce is not supported." The gate condition is world_size > 2 and not fully_connected ([msg 12]) [2]. Since the eight GPUs are PCIe-only with no NVLink, this check blocks both the standard custom allreduce and the quick allreduce paths.

This finding is a turning point. The assistant recognizes that the gate is in application code, not hardware-enforced, and could theoretically be monkey-patched. But the risk is significant — the custom allreduce implementation was explicitly designed for fully-connected GPU topologies, and using it across PCIe could cause silent data corruption or crashes. The assistant wisely concludes that force-enabling custom allreduce is not a viable path and pivots fully to the speculative decoding strategy ([msg 13]) [7].

Phase 5: The Kill Sequence and First Attempt

With the investigation phase complete, the assistant commits to action. The running vLLM server must be killed and restarted with speculative decoding configuration. What follows is a surprisingly difficult process termination saga spanning messages 13 through 18.

The first attempt uses pkill -f 'vllm.entrypoints' which sends SIGTERM — a polite termination request. The process ignores it. The second attempt escalates to kill -9 (SIGKILL), the nuclear option that cannot be caught or ignored. Yet the verification step still shows a process running ([msg 16]) [11]. The assistant switches tactics, using ps aux | grep vllm | grep -v grep to get a full process listing ([msg 17]) [9], finding nothing. The apparent survivor was a pgrep self-match — a classic pitfall where pgrep -f 'vllm' matches its own command line. The assistant resolves the ambiguity by directly querying the suspicious PID ([msg 18]) [12], confirming the process is truly gone.

This sequence, while seemingly mundane, reveals several important lessons about ML infrastructure management. vLLM's multi-process architecture (one entrypoint plus worker processes per GPU) means that killing the parent may leave children alive. CUDA contexts and NCCL communicators can delay process cleanup. And verification tools like pgrep have edge cases that can produce false positives. The assistant's methodical approach — try, verify, escalate, verify differently — is the correct response to these realities.

With the server killed, the assistant launches a new instance with ngram speculative decoding enabled ([msg 19]) [24]. The speculative configuration includes method: "ngram", num_speculative_tokens: 5, and prompt_lookup_num_tokens: 5. A health-check loop waits up to 5 minutes for the server to become ready.

Phase 6: Debugging the Configuration

The server fails to start. The health check times out after 300 seconds, and the log shows only the vLLM ASCII banner without completing initialization ([msg 20]) [18]. Something is wrong.

The assistant's response is instructive: rather than retrying with different parameters or searching for error messages, it goes directly to the source code. In message 21, the assistant states "prompt_lookup_num_tokens is not a valid parameter" and begins reading the speculative configuration file ([msg 21]) [13]. This is a critical insight — the parameter name provided by the user (and adopted by the assistant) simply does not exist in this version of vLLM.

The assistant reads the speculative config file in chunks, first lines 1-80 ([msg 21]), then lines 80-140 ([msg 22]) [21]. The correct parameter names are revealed in message 23: prompt_lookup_max and prompt_lookup_min, not prompt_lookup_num_tokens ([msg 23]) [5]. This is a textbook example of why source code is the authoritative documentation for rapidly evolving open-source projects. The vLLM nightly build (0.16.0rc2.dev313) has a different API surface than what the user expected, and only by reading the code can the discrepancy be resolved.

The assistant kills the failed server and restarts with the corrected configuration ([msg 24]) [27]. But the server crashes again — this time with a more fundamental error.

Phase 7: The Numba/NumPy Wall

The server log reveals the root cause: Numba needs NumPy 2.2 or less. Got NumPy 2.4 ([msg 26]) [48]. The ngram proposer in vLLM depends on Numba for JIT-compiled batch proposal logic, and Numba has not yet been updated to support NumPy 2.3+. The environment has NumPy 2.4.2, likely pulled in by a recent package upgrade.

This is a classic dependency conflict in the ML ecosystem. NumPy is a foundational library; virtually every package in the environment depends on it. Downgrading could break other components. The assistant faces a genuine engineering trade-off: fix the known issue (downgrade NumPy) or find an alternative speculative decoding method that doesn't require Numba.

The assistant's response demonstrates systematic decision-making ([msg 29]) [19]. It first checks whether suffix-based speculation might avoid the Numba dependency, reading the suffix_decoding.py header ([msg 31]) [17] and discovering it requires the arctic_inference package. A quick import test confirms the package is not installed ([msg 32]) [16]. The suffix decoding path is blocked.

The assistant then attempts to downgrade NumPy using pip, only to discover that pip is not installed in the virtual environment ([msg 33]) [14]. The environment was created with uv, which does not necessarily install pip. This is another environmental assumption corrected.

After installing pip or finding an alternative path, the assistant successfully downgrades NumPy to 2.2.6 ([msg 35]) [33], verifying that Numba now imports correctly ([msg 37]) [22]. The pin that held up speculative decoding is finally removed ([msg 36]) [39].

Phase 8: Launch and Evaluation

With the NumPy issue resolved, the assistant restarts the vLLM server with ngram speculative decoding ([msg 39]) [32]. The server starts successfully, and the assistant begins benchmarking.

The initial results are mixed. The server achieves approximately 44 tok/s ([msg 44]) [47], barely above the 43 tok/s baseline. The ngram proposer's effectiveness depends heavily on the predictability of the input text — short prompts with limited repetitive patterns yield few ngram matches, reducing the acceptance rate. The assistant digs into the speculative decoding metrics ([msg 48]) [76], analyzing the number of speculated tokens, acceptance rates, and the overhead introduced by the ngram proposer itself.

A deeper investigation reveals a more nuanced picture. The assistant runs multiple benchmarks with different speculative token counts (k=3, k=5) and prompt lengths ([msg 50]) [49], finding that the throughput variance is high and the average improvement over baseline is marginal. The allreduce bottleneck, which motivated the entire speculative decoding effort, is not being effectively amortized because the ngram proposer's acceptance rate is too low on typical prompts.

The assistant explores whether custom allreduce could be force-enabled as a complementary optimization ([msg 53]) [45], revisiting the NVLink gate discovered earlier. But the hardware topology remains the fundamental constraint — without NVLink, the custom allreduce implementation cannot be safely used. The assistant also investigates whether NCCL protocol tuning (e.g., NCCL_PROTO=LL) could improve the baseline allreduce performance ([msg 57]) [61], finding modest gains.

Phase 9: The Pivot to MTP

As the ngram speculation results plateau, the assistant revisits the most architecturally elegant option: GLM-5's native Multi-Token Prediction (MTP) support. The earlier discovery that vLLM has explicit support for glm4_moe_mtp model types ([msg 4]) [35] suggests that the model's built-in nextn prediction layer could be used as a draft mechanism without any external dependencies.

The assistant investigates whether the GGUF loader can handle the additional MTP weight tensors ([msg 81]) [74], reading the GGUF loader source code to understand how tensor names are mapped to model layers ([msg 80]) [80]. The 27 unmapped tensors from blk.78 correspond to the MTP head weights, which were intentionally skipped during loading because vLLM's GGUF support predated the MTP feature.

The assistant also reads the DeepSeek MTP implementation in vLLM ([msg 82]) [84] to understand how other models with similar architectures handle multi-token prediction. This comparative analysis reveals that MTP support is model-specific — each architecture requires explicit wiring between the speculative decoding infrastructure and the model's forward pass. For GLM-5, the glm4_moe_mtp model type exists in the configuration system, but the actual forward pass integration may not be complete for the GGUF format.

This investigation leads to a pragmatic conclusion ([msg 83]) [70]: while MTP support exists in vLLM's configuration layer, making it work end-to-end with a GGUF-quantized model would require significant engineering effort — potentially modifying the GGUF loader, the model forward pass, and the speculative decoding integration. The assistant judges this effort to be disproportionate to the expected gains, especially given that the ngram approach is already working (albeit with modest improvements).

Phase 10: Productionalization and Final Benchmark

The session concludes with a comprehensive benchmarking effort to definitively evaluate whether speculative decoding provides meaningful throughput improvements for this specific deployment. The assistant runs controlled experiments with and without speculative decoding, using consistent prompts and measurement methodology ([msg 74]) [64], establishing a clean baseline for comparison ([msg 75]) [72].

The final verdict, captured in message 90, is a nuanced assessment [86]: ngram speculative decoding provides marginal throughput improvements (approximately 5-10% in favorable cases) but introduces significant variance and can degrade performance on short or unpredictable prompts. The fundamental bottleneck remains the NCCL allreduce communication over PCIe, which speculative decoding can only partially amortize. The assistant recommends against deploying speculative decoding in production for this specific configuration, instead suggesting that efforts focus on reducing the allreduce overhead through NCCL tuning and CUDAGraph optimization — approaches that had already yielded improvements from ~20 tok/s to ~57 tok/s earlier in the session.

The assistant also productionalizes the optimized non-speculative configuration as a systemd service (vllm-glm5.service), ensuring that the tuned performance is maintained across reboots and can be managed through standard Linux service management tools.

Themes and Lessons

Several overarching themes emerge from this session:

Systematic debugging as a methodology. The assistant's approach — observe, hypothesize, investigate, fix, verify — is applied consistently across every obstacle. When the server fails to start, the assistant reads source code rather than guessing. When a dependency conflicts, the assistant explores alternatives before committing to a risky fix. When results are ambiguous, the assistant runs controlled experiments to isolate variables.

The source code as documentation. In a rapidly evolving open-source ecosystem where nightly builds are the norm, documentation lags behind code. The assistant's instinct to read source files directly — from speculative configuration to custom allreduce implementation to GGUF loader — is the correct response to this reality. Every significant discovery in this session came from reading code, not from consulting external resources.

The hardware topology as an immutable constraint. The PCIe-only GPU configuration is the single most important factor shaping the entire investigation. It blocks custom allreduce, limits the effectiveness of speculative decoding, and constrains the maximum achievable throughput. The assistant repeatedly returns to this constraint, accepting it rather than fighting it.

The value of failing fast. The assistant's quick probes — the import test for arctic_inference, the grep for Numba dependencies, the direct PID query — all follow the principle of failing fast and cheaply. Each negative result eliminates an entire branch of the decision tree, allowing effort to be redirected efficiently.

Pragmatic decision-making under uncertainty. The assistant consistently chooses the path with the highest probability of success given available information. When suffix decoding requires an uninstalled package, the assistant doesn't spend hours trying to install it — it pivots back to the NumPy downgrade. When MTP support exists in configuration but not in practice, the assistant judges the engineering effort against the expected return.

Conclusion

This session is a comprehensive case study in the realities of deploying large language models on multi-GPU hardware. It spans the full stack — from driver installation and CUDA toolkit management, through source code archaeology and dependency debugging, to performance optimization and production deployment. The assistant's methodical approach, combined with its willingness to read source code directly and pivot when necessary, demonstrates the kind of systematic engineering practice that separates successful ML deployments from fragile experiments.

The final outcome — a stable, optimized deployment achieving ~57 tok/s on 8 Blackwell GPUs, with a clear understanding of why speculative decoding provides limited additional benefit — represents genuine engineering progress. The session doesn't end with a magic bullet that doubles throughput, but with a deep understanding of the system's behavior, constraints, and optimization landscape. In the world of ML infrastructure, that understanding is often the most valuable output of all.