From GPU Race Conditions to Docker Infrastructure: A Systems Engineering Journey

Introduction

The most compelling engineering stories are often not about a single breakthrough, but about the seamless transition between fundamentally different kinds of work. This chunk of an opencode coding session captures exactly such a transition: the pivot from deep systems debugging — tracing GPU race conditions through C++ CUDA kernels, Rust FFI layers, and multi-threaded worker dispatch logic — to infrastructure engineering, where the same assistant researched build systems, read Dockerfiles, and solicited user input to architect a containerized deployment. The story spans two distinct phases, each demanding a different cognitive toolkit, yet connected by a single thread of rigorous methodology: understand the system, verify empirically, and document thoroughly.

Phase One: The Multi-GPU Fix — From Deployment to Commit

The first phase of this chunk (messages 515–535) represents the culmination of a debugging saga that had consumed multiple sessions. The CuZK proving engine, a high-performance GPU-accelerated zero-knowledge proof system, had been suffering from a subtle but devastating bug: the C++ CUDA code always routed single-circuit proofs to GPU 0, regardless of which Rust worker submitted them. On a two-GPU system, this meant workers 2 and 3 — assigned to GPU 1 by the engine — would still execute their CUDA kernels on GPU 0, creating data races and effectively wasting the second GPU entirely.

The Deployment That Proved the Fix

The assistant's first action in this chunk was to deploy the freshly compiled binary to the remote test host (cs-calib, a machine with two NVIDIA RTX A6000 GPUs). The deployment command at <msg id=515> was a carefully orchestrated sequence: stop the daemon, rsync the new binary to a temporary location, copy it to the permanent path, restart the service, wait for initialization, and verify the status. This was not a blind deployment — it was the first step in a rigorous verification chain.

The output showed the daemon active (running) with 7.1 GB of memory allocated and 328 tasks spawned. But this was only the beginning. The assistant then ran a single PoRep proof (<msg id=517>) which completed successfully in approximately 110 seconds — confirming basic functionality was intact. Then came the critical check: nvidia-smi at <msg id=518> showed both GPUs with 12,986 MiB of memory allocated. Previously, GPU 1 had only 700 MB. This was the first concrete evidence that the gpu_index parameter threading was working: both GPUs were now being initialized with the d_a_cache and SRS parameters.

The Burst Test and Log Forensics

At <msg id=519>, the assistant designed a burst test — three concurrent proof submissions on a two-GPU system — to stress-test the load balancing. The reasoning was sound: if the fix was incorrect, concurrent proofs would cause contention, data races, or OOM errors. If the fix was correct, the engine's GPU workers would distribute the work across both devices.

The results were conclusive. All three proofs completed successfully. The journalctl logs (<msg id=520>) showed workers correctly load-balanced: workers 0 and 1 on GPU 0 (6–7 jobs each), workers 2 and 3 on GPU 1 (5–6 jobs each). The d_a_cache allocation logs showed both GPUs receiving d_a_cache allocated 4096 MiB at startup. The user independently confirmed via nvtop (<msg id=526>) that both GPUs were seeing load, providing real-time visual validation that complemented the log-based analysis.

Not everything went smoothly. The assistant initially misread the gpu_tid=0 log entries, which referred to the loop index within a per-GPU thread rather than the physical GPU device. A narrow time window on the d_a_cache logs initially showed only GPU 0, requiring a broader query (10 minutes instead of 3) to reveal both GPUs. These moments of doubt — captured in messages 522–525 — are a testament to the assistant's methodological rigor: rather than accepting the first answer, it dug deeper, cross-referenced multiple data sources, and only declared success when the evidence was unambiguous.

The Commit That Sealed the Fix

With verification complete, the assistant committed the changes at <msg id=533>. The commit message is a model of engineering documentation:

Thread a gpu_index parameter through the entire proving stack (C++ -> supraseal-c2 -> bellperson -> pipeline -> engine) so that single-circuit partition proofs run on the GPU assigned to the Rust worker instead of always landing on GPU 0.

The message explicitly contrasts the old behavior ("always resolved to GPU 0 via select_gpu(0)") with the new behavior ("gpu_index >= 0 pins work to that specific GPU, while -1 preserves the original multi-GPU fan-out"). It candidly acknowledges the failed first attempt: "the original shared-mutex workaround serialized everything to one GPU." This intellectual honesty — documenting dead ends as well as solutions — is a hallmark of good engineering.

The diff touched five files with 75 insertions and 63 deletions: the C++ groth16_cuda.cu (core GPU selection logic), the Rust FFI lib.rs (parameter threading), the bellperson supraseal.rs (prover functions), the pipeline engine.rs (reverting the shared mutex hack, restoring per-GPU mutexes), and the pipeline pipeline.rs (intermediate layer). The net change was only 12 lines — a reminder that the best fixes are often small in diff but large in insight.

Phase Two: The Pivot to Docker Infrastructure

Immediately following the commit, the conversation took a sharp turn. At <msg id=536>, the user issued a request belonging to an entirely different domain of software engineering:

"Prepare a docker container with curio, cuzk, all curio dependencies, and fetching 32G PoRep and Snap (only, ideally) proving params with e.g. curio fetch-params (fetch probably as autostart script). The container should start into bash, for now, and will be published to dockerhub."

This pivot reveals something important about the user's mental model: the multi-GPU fix was never an end in itself. It was a blocker that needed to be resolved before the real goal — packaging the entire proving stack into a portable, reproducible Docker image — could be pursued. The debugging phase was necessary infrastructure work; the Docker container was the deliverable.

The Research Phase: Building a Mental Model

The assistant did not immediately start writing a Dockerfile. Instead, it initiated a structured research phase spanning messages 537–543. This is a deliberate methodological choice: before constructing, understand.

The assistant created a todo list with research items, then spawned two parallel subagent tasks (<msg id=538>) to explore the curio build system and the cuzk build requirements. These subagents ran concurrently, each producing comprehensive reports on their respective domains. The curio analysis covered existing Dockerfiles, Makefiles, build scripts, and dependency chains. The cuzk analysis covered the Cargo workspace structure, Rust version requirements, CUDA dependencies, and the supraseal C++ build process.

But the assistant did not stop at subagent summaries. It then read the primary sources directly: the existing Dockerfile (<msg id=540>) to understand the multi-stage build patterns already in use; the 00-vars.mk and 10-deps.mk Makefiles to understand build variables and dependency targets; the 30-build.mk file (<msg id=541>) to understand the curio build command itself; and the cuzk.example.toml to understand the daemon configuration. At <msg id=543>, the assistant read build-blst.sh and the first 70 lines of supraseal/build.sh to understand the C++/CUDA compilation requirements, including the GCC version constraints imposed by the CUDA toolkit.

This systematic reading reveals a clear hierarchy of concerns: first understand the containerization pattern (Dockerfile), then understand the build variable conventions (vars.mk), then understand the dependency initialization (deps.mk), then understand the specific CUDA compilation requirements (build scripts). The assistant was building a mental model before building a container.

The Deliberate Pause: Asking Before Acting

At <msg id=544>, the assistant reached a critical inflection point. It had completed its research and had everything it needed to write a Dockerfile. But instead of proceeding, it paused. The reason is captured in the first sentence: "Now I have everything I need. Let me also quickly check what curio fetch-params actually downloads and whether we can filter for just PoRep and Snap params."

The assistant then presented a structured question to the user with three configuration decisions that would fundamentally shape the Dockerfile's architecture:

Network target (calibnet vs. mainnet): The curio build system uses Go build tags to target either the test network or production. This choice affects which parameters are fetched and which contracts are compiled in.
CUDA base image version (12.8 vs. 13): The assistant had been researching CUDA 12.8 as the "latest stable" version, but the user's answer — "cuda13 is a thing now" — revealed a more recent release. The base image choice ripples through the entire Dockerfile: it determines which apt packages are available, which nvcc version is used, and which runtime libraries must be included.
Parameter fetching strategy (runtime vs. baked-in): The proving parameters for 32G PoRep and Snap total approximately 100 GB. Baking them into the Docker image would make it impractically large. Fetching them at runtime via an entrypoint script keeps the image lean (~15 GB) but requires internet access on first start. The assistant labeled one option as "Recommended" (calibnet, CUDA 12.8, fetch at runtime) but presented alternatives with clear trade-offs. This framing shows mature engineering judgment: the assistant had already evaluated the options and formed a preference, but recognized that some decisions must be informed by human authority. The user's truncated response — "mainnet, cuda13 is a thing now..." — overrode the recommendation on both the network target and the CUDA version, while presumably accepting the runtime fetching strategy. This interaction is a model of effective human-AI collaboration: the assistant researches thoroughly, identifies decision points, presents structured options, and the user provides the domain-specific knowledge that the assistant cannot infer from code alone.

The Bridge Between Two Worlds

What makes this chunk particularly compelling is not the individual accomplishments — the multi-GPU fix or the Docker research — but the seamless transition between them. The assistant demonstrated the ability to context-switch from deep systems debugging (thinking about CUDA device selection, mutex semantics, and FFI parameter threading) to infrastructure engineering (thinking about base images, multi-stage builds, and entrypoint scripts) without losing methodological rigor.

The common thread is the assistant's systematic approach: understand the system before modifying it, verify empirically rather than assuming correctness, and document decisions clearly. In the debugging phase, this meant reading code, tracing call chains, deploying to remote hosts, and cross-referencing log sources. In the infrastructure phase, this meant reading Dockerfiles, spawning research subagents, examining build scripts, and soliciting user input before writing code.

The chunk also illustrates a deeper truth about software engineering: the boundary between "debugging" and "infrastructure" is often artificial. The multi-GPU fix was necessary because the proving engine would crash on multi-GPU systems. The Docker container was necessary because the proving engine needed to be distributed to users. Both tasks were in service of the same goal — making GPU-accelerated zero-knowledge proving work reliably in production — but they demanded different skills, different tools, and different modes of thinking.

Conclusion

This chunk of the opencode session captures a complete arc of systems engineering: from the deployment and rigorous verification of a critical bug fix, through the formal commit that sealed the changes, to the pivot toward infrastructure packaging and the methodical research that preceded any construction. The assistant demonstrated thoroughness in verification, discipline in documentation, and adaptability in pivoting to a completely new task domain.

The story is not yet complete — the Dockerfile has not been written, the container has not been built — but the foundation has been laid. The research is done, the user's configuration decisions have been captured, and the assistant is poised to construct the container. What makes this chunk satisfying is not the resolution of a single problem, but the demonstration of a repeatable methodology: understand, verify, document, pivot, research, ask, build. It is a template for how complex systems engineering should be done, whether by humans or by AI.