From CUDA Crash to Clean Benchmark: A Case Study in Infrastructure Debugging and Recovery on Blackwell GPUs

Introduction

Modern machine learning infrastructure is a layered stack of hardware, kernel modules, container runtimes, driver APIs, and application frameworks. When something breaks, the error message rarely tells you which layer is at fault. This article examines a single contiguous segment of an opencode coding session — spanning roughly 40 messages — in which an AI assistant navigates a cascade of failures that would have derailed a less methodical engineer. The session begins with a host reboot that shatters a carefully prepared GPU compute environment, and ends with the successful execution of benchmark workloads on eight NVIDIA RTX PRO 6000 Blackwell GPUs. Along the way, the assistant diagnoses CUDA kernel crashes, traces a cgroup device permission issue through strace, fixes a Proxmox LXC container configuration, recovers from a lost model download, and ultimately pivots to an entirely new model architecture.

This is not a story about a single bug fix. It is a story about the discipline of systematic debugging — the willingness to follow evidence through multiple layers of abstraction, the judgment to know when to skip a broken configuration versus when to chase a root cause, and the operational maturity to clean up corrupted state before resuming productive work.

Part I: The DDTree Crash — When Small Budgets Break on Blackwell

The session's first crisis appears in [msg 11285], where the assistant discovers that two DDTree speculative decoding configurations — budgets 8 and 12 (b8 and b12) — have returned zero throughput across all workloads. At first glance, this looks like a performance measurement: the model is simply slow with these budgets. But the assistant digs deeper, examining the server logs, and finds something far more alarming.

The DDTree verification step for budget 8 completed with zero accepted drafts: DDTREE metrics: bs=2 budget=8 avg_actual_nodes=9.00 avg_accepted_drafts=0.00 avg_commit_len=1.00. Then the server crashed with a CUDA error: an illegal instruction was encountered on the Blackwell SM120 architecture. This is not a performance problem — it is a correctness bug, and a dangerous one at that. The benchmark runner had dutifully recorded the zeros as if they were valid measurements, producing JSON results files that looked legitimate but contained garbage data.

The assistant's reasoning in [msg 11285] reveals a sophisticated diagnostic process. It considers multiple hypotheses: the flat tree structure produced by small budgets (8 children at depth 1, creating only 9 total nodes) might violate alignment constraints in the Triton attention kernel on the new SM120 architecture; the non-contiguous KV commit code might have a bug when there are zero accepted drafts to process; or the verification step might produce invalid data that corrupts the subsequent commit operation. The b12 configuration shows a slightly different failure mode — the warmup succeeds with a 16-token response, but longer 256-token requests crash mid-generation — suggesting the crash is triggered by sustained decoding rather than startup.

The assistant makes a pragmatic decision: rather than debugging the DDTree implementation for small budgets (which would require deep knowledge of Triton kernel compilation and CUDA architecture), it will skip b8 and b12 entirely and focus on the budgets known to work — b15, b16, b32, and b64. This is a risk-calibrated tradeoff that prioritizes progress over perfect understanding. The assistant also identifies a second issue: the autoregressive baseline was failing with HTTP 400 errors because the prompt generation was creating approximately 32,700 tokens, which exceeded the model's 32,768 context length limit when combined with the 256 output tokens.

These diagnoses drive a series of edits to bench_runner.py ([msg 11288] through [msg 11294]): removing b8 and b12 from the test matrix, reducing the TP1 context size from 30,000 to 25,000 tokens, and adding a server health check mechanism that detects crashes early rather than silently accumulating zero-token results. The health check addition is particularly significant — it transforms the benchmark runner from a blind executor into an adaptive system that can detect infrastructure failures and respond appropriately.

Part II: The Silent GPU — When CUDA Refuses to Initialize

With the script fixed and the crashing configurations removed, the assistant launches the first test: python3 bench_runner.py tp1. The result, shown in [msg 11297], is immediate failure:

!! SERVICE FAILED:
Failed to get device capability: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start.

This is the moment when the session pivots from benchmarking to infrastructure recovery. The assistant had spent hours recovering from a machine reboot — re-downloading the 52 GB model to /dev/shm, cleaning up corrupted benchmark results, and patching the benchmark runner. All of that work was contingent on the GPUs being accessible. Now CUDA is completely non-functional.

The assistant's diagnostic response, spanning [msg 11298] through [msg 11308], is a masterclass in systematic troubleshooting. Let us trace the chain:

Step 1 — Verify the driver is loaded. The assistant runs nvidia-smi, which works and lists all 8 GPUs with 0 MiB memory used. The kernel-mode driver is functional.

Step 2 — Enable persistence mode. The assistant runs nvidia-smi -pm 1, a common fix for post-reboot GPU issues. This doesn't help.

Step 3 — Attempt GPU reset. The assistant tries nvidia-smi --gpu-reset, which fails with "not supported" on these Blackwell GPUs.

Step 4 — Check CUDA runtime directly. Using ctypes to call the raw CUDA driver API, the assistant finds that cudaGetDeviceCount returns error code 999 (CUDA_ERROR_UNKNOWN) with count=0. This confirms the problem is at the driver level, not in PyTorch or any higher-level library.

Step 5 — Test multiple CUDA runtime versions. Both a CUDA 12.8 venv and a CUDA 13.0 venv fail identically, ruling out a version mismatch hypothesis.

Step 6 — Confirm the container environment. The assistant checks /proc/1/cgroup and finds init.scope, confirming this is an LXC container on a Proxmox host.

Step 7 — Check device file permissions. All /dev/nvidia* device nodes exist with crw-rw-rw- permissions. The assistant notes that /dev/nvidia-uvm-tools has no permissions (mode 000) and fixes it with chmod 666.

Step 8 — Trace the failing system call. This is the breakthrough. The assistant runs strace on cuInit(0) and finds the smoking gun:

openat(AT_FDCWD, "/dev/nvidia-uvm", O_RDWR|O_CLOEXEC) = -1 EPERM (Operation not permitted)

The device file /dev/nvidia-uvm has world-readable and world-writable permissions (666), yet the kernel refuses to open it. This contradiction points to a mechanism that operates above the file permission layer — a cgroup v2 device restriction.

Step 9 — Confirm the cgroup restriction. A direct Python test confirms the asymmetry: /dev/nvidiactl and /dev/nvidia0 (both major 195) open successfully, but /dev/nvidia-uvm (major 511) fails with Errno 1: Operation not permitted. The container's cgroup v2 eBPF device controller is blocking access to major number 511.

Step 10 — Read the container configuration. With host access provided by the user ([msg 11309]: "Just ssh root@10.1.2.6 -> kpro6 host"), the assistant reads /etc/pve/lxc/200.conf and finds the root cause:

lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 509:* rwm
lxc.cgroup2.devices.allow: c 226:* rwm
lxc.cgroup2.devices.allow: c 234:* rwm

The container allows devices with major numbers 195 (NVIDIA), 509, 226, and 234 — but critically, major 511 (nvidia-uvm) is absent. The config has c 509:* rwm, a leftover from a previous NVIDIA driver version where the UVM device used major 509. After the driver was updated to version 595.71.05, the UVM device's major number shifted to 511, and the container configuration was never updated.

Part III: The Fix — A Single Line in a Configuration File

The fix is elegantly simple. The assistant appends the missing line to the LXC configuration file on the Proxmox host:

echo 'lxc.cgroup2.devices.allow: c 511:* rwm' >> /etc/pve/lxc/200.conf

But the first attempt has a formatting error — the line is added without the lxc.cgroup2.devices.allow: prefix ([msg 11311]). The assistant discovers this in [msg 11312] and corrects it with sed in [msg 11313], inserting the properly formatted line after the last existing cgroup rule:

sed -i '/^c 511:\* rwm$/d' /etc/pve/lxc/200.conf
sed -i '/^lxc.cgroup2.devices.allow: c 234:\* rwm$/a lxc.cgroup2.devices.allow: c 511:* rwm' /etc/pve/lxc/200.conf

This attention to configuration hygiene — removing the malformed line before adding the correct one, and placing it in the logical section of the file — is a mark of operational maturity.

The container is then rebooted with pct reboot 200 ([msg 11314]). After just five seconds ([msg 11315]), the container is back online. The assistant runs the definitive CUDA test ([msg 11316]):

CUDA available: True
Device count: 1
Device: NVIDIA RTX PRO 6000 Blackwell Server Edition
Tensor OK: 0.0

CUDA is restored. But the model — stored in /dev/shm (tmpfs) — is gone, cleared by the reboot. The assistant launches a background download of the 52 GB Qwen3.6-27B model from Hugging Face ([msg 11317]) and monitors its progress through a polling loop ([msg 11319]), watching the directory grow from 24 GB to 37 GB to 46 GB to 52 GB over approximately 75 seconds.

Part IV: The Clean Slate — Resuming Productive Work

With the model restored, the assistant performs a deliberate act of data curation ([msg 11320]). It deletes five potentially corrupted benchmark result files — tp1-auto.json, tp1-b15.json, tp1-b16.json, tp1-b32.json, tp1-b64.json — keeping only the known-good tp1-linear.json baseline from before the crash. This is a judgment call that prioritizes data integrity over salvage: partial results from interrupted runs can contain misleading data, and the fastest path to a complete benchmark suite is to re-run from scratch rather than validate corrupted measurements.

The assistant updates its todo list ([msg 11321]) to reflect the new state: infrastructure setup and script fixes are marked completed, TP1 benchmarks are in progress. Then, finally, the first benchmark runs ([msg 11322]):

================================================================
  tp1-auto  tp=1  method=auto
================================================================
  warmup OK
  [short max_tokens=256]
       fib:    26.2 +/-   0.4 tok/s
     qsort:    26.4 +/-   0.1 tok/s
     arith:    26.4 +/-   0.1 tok/s
      json:    26.5 +/-   0.0 tok/s

The autoregressive baseline shows approximately 26.4 tok/s, consistent with user reports for Qwen3.6-27B on a single Blackwell GPU. This validation from the user ([msg 11324]: "Oh without mtp seems actually close to what users report") confirms that the infrastructure is working correctly and the benchmarks will be meaningful.

Part V: The Pivot — From Qwen3.6 to Kimi K2.6

The successful TP1 benchmarks reveal a critical insight about DDTree's behavior on the hybrid Qwen3.6 architecture. The assistant's earlier analysis ([msg 11344]) had identified that DDTree budget 15 was the optimal configuration, while higher budgets (32, 64) suffered from Mamba state leakage — a phenomenon where the model's recurrent Mamba layers corrupt their internal state when processing sibling tree nodes during sequential verification. This architectural limitation caps DDTree's potential on hybrid models.

The user then asks a pivotal question ([msg 11345]): "Would Kimi K2.6 be easier?" Kimi K2.6 is a pure attention MoE (Mixture of Experts) model — no Mamba layers, no recurrent state to leak. This makes it an ideal candidate for DDTree speculative decoding, which can exploit the model's full tree verification capacity without the state corruption penalty that plagues hybrid architectures.

The assistant pivots immediately ([msg 11348]), shifting from benchmarking Qwen3.6 to deploying and benchmarking Kimi K2.6 with DFlash and EAGLE-3 speculative decoding. This pivot involves downloading a 595 GB model, resolving a compressed-tensors library version mismatch, debugging SGLang argument validation errors, and deploying EAGLE-3 on an 8-GPU TP8 configuration. The EAGLE-3 benchmarks ultimately show a 1.6–1.7× speedup over the autoregressive baseline for single requests, though the advantage narrows at high concurrency due to PCIe AllReduce bottlenecks.

Lessons Learned

This session offers several enduring lessons for anyone working with GPU-accelerated machine learning infrastructure:

1. Error messages lie. The "CUDA unknown error" (error code 999) is a catch-all that provides no diagnostic information. PyTorch's wrapper message — suggesting an environment variable issue — is actively misleading. The only reliable approach is to trace the actual system calls with strace and see exactly which operation fails and why.

2. Know your container boundaries. LXC containers share the host kernel but have restricted device access through cgroup v2 eBPF programs. A device file with rw-rw-rw- permissions can still be inaccessible if the cgroup device controller blocks its major number. Understanding this distinction is essential for debugging GPU passthrough issues.

3. Device major numbers can change. NVIDIA driver updates can change the major number assigned to the nvidia-uvm device. Container configurations that hardcode major numbers must be updated when the driver changes. This is a known fragility in the NVIDIA driver architecture for containerized environments.

4. Verify your fixes. The assistant's first attempt to add the cgroup permission produced a malformed line. By checking the config file afterward, the assistant caught the error and corrected it before rebooting the container. This feedback loop — apply, verify, correct — is essential for reliable infrastructure management.

5. Clean up corrupted state. After the infrastructure crisis, the assistant deleted five potentially corrupted benchmark result files rather than trying to salvage them. This clean-slate approach prevents contaminated data from polluting the final analysis and is a hallmark of disciplined empirical work.

6. Know when to skip and when to chase. The assistant chose to skip the b8/b12 DDTree configurations rather than debug the CUDA kernel crash. This was a correct prioritization: the goal was to benchmark working configurations, not to fix every kernel bug encountered along the way. But the assistant also chose to chase the cgroup device permission issue to its root cause, because that was a systemic infrastructure problem that would affect all subsequent work.

Conclusion

The chunk of session examined here spans approximately 40 messages and covers two major infrastructure crises — a CUDA kernel crash on small DDTree budgets, and a complete CUDA initialization failure due to a cgroup device permission mismatch. The assistant navigates both crises with methodical discipline: forming hypotheses, designing targeted tests, interpreting evidence, and making risk-calibrated decisions about when to work around a problem versus when to fix it at the root.

The session ultimately succeeds not because the assistant avoided failures, but because it had a reliable process for recovering from them. The cgroup fix — a single line added to a configuration file — is the visible output of an invisible diagnostic chain that spanned strace analysis, cgroup theory, NVIDIA driver internals, and Proxmox LXC container management. The clean benchmark results that follow are not just measurements of model performance; they are evidence that the infrastructure is trustworthy.

In the world of large-scale ML engineering, this is the most valuable skill of all: not the ability to write code that never breaks, but the ability to systematically diagnose and recover when it inevitably does.