The Dry-Run That Changed Direction: How a Failed vLLM Upgrade Forced a Custom Attention Backend

In the high-stakes world of deploying cutting-edge large language models on novel hardware, the path forward is rarely a straight line. Message [msg 1718] captures a pivotal moment in a complex engineering session: the attempt to upgrade vLLM to its latest nightly build, a seemingly routine operation that instead became a dead end, redirecting the entire project toward a far more ambitious custom solution.

The Message

The message itself is deceptively simple — a single bash command executed over SSH:

ssh root@10.1.230.174 '~/.local/bin/uv pip install --python ~/ml-env/bin/python3 \
  --index-url https://wheels.vllm.ai/nightly vllm --dry-run 2>&1 | head -30'

And its result:

Using Python 3.12.3 environment at: ml-env
  × No solution found when resolving dependencies:
  ╰─▶ Because numba was not found in the package registry and numba==0.61.2
      depends on numpy>=1.24,<2.3, we can conclude that numba==0.61.2 depends
      on numpy>=1.24,<2.3.
      And because numpy was not found in the package registry and
      vllm==0.16.0rc2.dev314+gf72061a19 depends on numba==0.61.2, we can
      conclude that vllm==0.16.0rc2.dev314+gf72061a19 cannot be used.

A dependency resolution failure. The nightly vLLM wheel (0.16.0rc2.dev314, just one commit ahead of the installed dev313) required numba==0.61.2 which in turn required numpy&gt;=1.24,&lt;2.3, but neither package was found in the registry accessible to the uv resolver. The upgrade was blocked.

The Context: Why This Message Was Written

To understand why this dry-run mattered, we must step back and appreciate the predicament the team faced. They were deploying the GLM-5 model — a massive 402GB GGUF-quantized MoE (Mixture of Experts) architecture — on a machine with 8 RTX PRO 6000 Blackwell GPUs (compute capability SM120). After days of patching vLLM's GGUF loader, fixing weight mapping bugs, and resolving launch errors, they had finally gotten the model to start loading. But then they hit a wall: no attention backend supported the combination of SM120 (Blackwell), sparse MLA attention (from the DSA indexer), and qk_nope_head_dim=192.

The attention selector in vLLM systematically rejected every available backend:

What the Assistant Knew and Assumed

Before running this command, the assistant had done substantial research (<msg id=1716-1717>). Web searches had revealed:

  1. A vLLM blog post from February 3, 2026 showing GB200 Blackwell support working for DeepSeek-style models with FlashMLA
  2. A Reddit post about someone porting FlashMLA sparse to SM120
  3. A GitHub issue (#26211) describing the exact same problem — DeepSeek on RTX PRO 6000/SM120 The assistant's working assumption was that a newer nightly build might contain the SM120 patches needed. This was a reasonable hypothesis: the installed nightly was dev313, and if the upstream had merged Blackwell support in the intervening commits, a simple pip install --upgrade could solve the problem. However, the assistant also made a subtler assumption: that the nightly wheel index would resolve dependencies cleanly. The uv package manager was used with --dry-run precisely to test this assumption safely — a methodical approach that avoided breaking the working environment.

The Dependency Resolution Failure

The dry-run output revealed two intertwined problems:

First, the latest nightly (dev314) was barely ahead of the installed version (dev313). The commit hash suffix gf72061a19 confirmed it was only a handful of commits newer. This meant that even if the upgrade succeeded, it likely wouldn't contain the SM120 sparse MLA support they needed. The Blackwell support in the blog post was probably in a different branch or not yet merged into the nightly release pipeline.

Second, the dependency resolver hit a fundamental conflict. The nightly vLLM wheel pinned numba==0.61.2, which required numpy&gt;=1.24,&lt;2.3. But when using only the nightly index (https://wheels.vllm.ai/nightly), numpy and numba were not found in the registry. The uv resolver, configured with just the single index URL, could not locate these transitive dependencies and correctly reported an unsatisfiable constraint.

This was not a bug in vLLM or uv — it was a configuration issue. The nightly index only hosts vLLM wheels, not its full dependency tree. A production installation would typically combine the nightly index with the standard PyPI index (as the assistant would discover in the subsequent message [msg 1719] when adding --extra-index-url https://pypi.org/simple resolved the issue). But the dry-run as configured failed, and that failure carried decisive weight.

The Critical Decision: Abandoning the Upgrade Path

This message is the turning point. The dry-run failure, combined with the realization that dev314 was barely ahead of dev313, led the assistant to conclude that the nightly upgrade path was not viable. In the todo list updated immediately after ([msg 1726]), the status of "Update vLLM to latest nightly with SM120 Blackwell MLA support" was changed from "in_progress" to "cancelled".

Instead, the assistant pivoted to a far more ambitious strategy: writing a custom Triton MLA sparse attention backend. This was the option the user had selected from a multiple-choice question ([msg 1725]): "Patch Triton MLA to support sparse." The reasoning was sound — Triton MLA was already pure Python/Triton, worked on SM120, and the sparse integration could be modeled after the existing FlashMLA sparse backend.

The Knowledge Flow

Input knowledge required to understand this message includes:

The Thinking Process

The assistant's reasoning is visible in the structure of the command itself. The --dry-run flag reveals a cautious, methodical approach: test before committing. The 2&gt;&amp;1 | head -30 shows an expectation that the output might be long, and only the first 30 lines are needed to see the result. The use of ~/.local/bin/uv with --python pointing to the specific environment shows careful environment management — the assistant is working within a pre-existing ml-env virtual environment and ensuring the right Python is targeted.

The subsequent messages confirm this thinking. After the dry-run failure, the assistant immediately tried a variant with --extra-index-url https://pypi.org/simple and --prerelease=allow ([msg 1719]), which resolved successfully but still showed only dev314 — confirming that the upgrade wouldn't help. Then the assistant tried version constraints (&#34;vllm&gt;=0.16&#34;) to search for newer versions (<msg id=1720-1721>), which also failed, conclusively proving that no newer nightly existed on the index.

Broader Significance

In the arc of this engineering session, message [msg 1718] represents the moment when the "easy path" closed and the "hard path" opened. The failed dry-run forced the team to abandon hope of upstream support and instead build their own solution. The result was the TritonMLASparseBackend — a custom attention backend that would eventually enable the GLM-5 model to run on Blackwell hardware.

This pattern is universal in systems engineering: when a standard component cannot be upgraded to meet new requirements, the only option is to build the missing piece yourself. The dry-run was the diagnostic that revealed this truth, and the pivot it triggered would consume the next several hours of development effort. But it was the right call — and it began with a single command that returned "No solution found."