The Filename That Almost Broke the Stack: A Critical Wheel Download in the CUDA 13 Upgrade

In the course of upgrading a production ML serving stack from CUDA 12.8 to CUDA 13.0 on an 8× NVIDIA RTX PRO 6000 Blackwell system, a single bash command—ostensibly trivial—became the linchpin of an entire afternoon's debugging. Message [msg 5316] is that command: a wget invocation to download a Python wheel with a carefully corrected filename. On its surface, it is unremarkable. But to understand why this message matters, one must appreciate the chain of failures that preceded it and the delicate ABI compatibility puzzle that this download was meant to solve.

The ABI Nightmare

The story begins with the decision to upgrade to CUDA 13.0. The Blackwell GPUs (SM120 architecture) required CUDA 13 for access to Blackwell-native optimizations—specifically FlashInfer allreduce fusion and Torch symmetric memory—which had been identified as the most promising path to improve EAGLE-3 speculative decoding throughput (see [chunk 35.0]). The assistant had successfully installed CUDA 13.0.1 alongside the existing CUDA 12.8, and installed a PyTorch nightly build (2.12.0.dev20260226+cu130) from the cu130 index. But when it tried to import sgl_kernel, it crashed with an ImportError:

[sgl_kernel] CRITICAL: Could not load any common_ops library...

The root cause, uncovered through careful nm symbol inspection in [msg 5304], was a C++ ABI mismatch. The sgl_kernel shared object (common_ops.abi3.so) contained an undefined symbol _ZN3c104cuda29c10_cuda_check_implementationEiPKcS2_ib—note the trailing i (meaning the int parameter type). But the PyTorch libc10_cuda.so provided the symbol with a j suffix (unsigned int). This tiny difference in the C++ name mangling—int vs unsigned int for the error code parameter in c10_cuda_check_implementation—meant the dynamic linker could not resolve the symbol, and the entire sgl_kernel module failed to load.

This was a classic PyTorch ABI break. The sgl_kernel wheel on the cu130 index had been compiled against an older version of PyTorch (likely 2.9.x), while the installed PyTorch 2.12.0 nightly had a slightly different signature for that internal function. The wheel's use of the abi3 stable ABI tag was supposed to guarantee cross-version compatibility, but it failed here because the underlying C++ ABI of PyTorch itself had changed.

The Search for a Compatible Wheel

The assistant tried several approaches. First, it downgraded from the nightly PyTorch 2.12.0 to the stable PyTorch 2.10.0+cu130 ([msg 5302]), hoping the sgl_kernel wheel had been built against that version. Same error. Then it tried reinstalling sgl-kernel from the cu130 index with --no-cache ([msg 5306])—same binary, same checksum, same symbol mismatch. The wheel on the index was a single abi3 build that simply did not match any available PyTorch version.

The assistant then discovered nightly GitHub releases of SGLang ([msg 5308]) and checked if they contained newer sgl_kernel builds. They didn't—only the main sglang wheel was published there. Another dead end.

Then came the critical insight. The assistant found a direct GitHub release URL for a cp312-specific wheel: sgl_kernel-0.3.21%2Bcu130-cp312-abi3-manylinux2014_x86_64.whl. This was the same version (0.3.21) but with the cp312 Python version tag explicitly in the filename. The %2B in the URL was a URL-encoded + sign. The assistant downloaded it in [msg 5314]:

wget -q "https://github.com/sgl-project/whl/releases/download/v0.3.21/sgl_kernel-0.3.21%2Bcu130-cp312-abi3-manylinux2014_x86_64.whl" -O /tmp/sgl_kernel_cp312_cu130.whl

But here it made a subtle mistake: it renamed the file to sgl_kernel_cp312_cu130.whl, stripping the version string. When it tried to install this renamed file with uv pip install in [msg 5315], the tool rejected it:

error: The wheel filename "sgl_kernel_cp312_cu130.whl" is invalid: Must have a version

The Fix: Message 5316

This brings us to the subject message. The assistant realized its error and re-issued the download with the correct output filename, preserving the full version string:

[bash] ssh root@10.1.230.174 'cd /tmp && wget -q "https://github.com/sgl-project/whl/releases/download/v0.3.21/sgl_kernel-0.3.21%2Bcu130-cp312-abi3-manylinux2014_x86_64.whl" -O "sgl_kernel-0.3.21+cu130-cp312-abi3-manylinux2014_x86_64.whl" 2>&1 && ls -lh /tmp/sgl_kernel-0.3.21+cu130*'

The command is straightforward: download the wheel from GitHub Releases to /tmp, using -O to specify the output filename exactly matching the canonical wheel naming convention (package-version-python_tag-abi_tag-platform_tag.whl). The ls at the end confirms the file exists and reports its size: 376 MB.

This is a tiny message—a single bash command with a wget and an ls. But it represents the culmination of a significant debugging effort. The assistant had traversed multiple dead ends: nightly PyTorch ABI breaks, stale wheel caches, missing GitHub release assets, and now a filename parsing bug. Each failure taught something about the constraints of the environment.

Assumptions and Knowledge

The key assumption behind this message was that the cp312-specific wheel from GitHub Releases would have a different binary than the one served from the PyPI-style index at https://docs.sglang.ai/whl/cu130/. This turned out to be incorrect—as [msg 5318] would later show, the .so file inside the cp312 wheel had the exact same checksum and the same unresolved symbol. The wheel was identical; only the filename metadata differed.

The input knowledge required to understand this message includes: the wheel naming convention for Python packages (PEP 427), the fact that uv pip install parses the filename to extract metadata, the URL encoding of + as %2B, and the broader context of the ABI mismatch saga. The output knowledge created is a correctly-named wheel file sitting in /tmp, ready for installation.

The Broader Arc

What makes this message significant is not the command itself but its place in the narrative. The assistant was methodically working through a compatibility matrix: try PyTorch nightly → fails, try PyTorch 2.10.0 stable → fails, try cp312 wheel from GitHub → filename error, fix filename → still fails (same binary). Each iteration eliminated a hypothesis and narrowed the search space.

The actual solution—discovered in the messages immediately following—was to downgrade PyTorch to 2.9.1+cu130 ([msg 5319]), which matched the ABI that sgl_kernel 0.3.21 was compiled against. That worked, but then revealed a secondary issue: libnvrtc.so.13 was not on the library path ([msg 5321]), requiring a ldconfig update. After that, all three packages loaded successfully: torch 2.9.1+cu130, sgl_kernel 0.3.21, flashinfer 0.6.4.

The filename correction in message 5316 was a necessary but ultimately insufficient step. It was a bug fix within a larger debugging process—the kind of small, precise correction that experienced engineers make when they recognize a pattern (invalid wheel filename) and know exactly how to fix it. The message embodies the principle that in complex system integration work, attention to detail at every level—even the exact spelling of a filename—determines success or failure.