The 4% GPU Utilization Puzzle: A Moment of Empirical Validation in the cuzk Proving Engine

Introduction

In the middle of validating a brand-new SNARK proving daemon for Filecoin's Proof-of-Replication (PoRep) protocol, a developer pauses to check GPU utilization and finds a surprising number: 4%. The message at index 225 in this coding session captures a brief but pivotal moment—a single diagnostic check that reveals the gap between architectural design and real-world performance behavior. This article examines that message in depth: why it was written, what assumptions it reveals, and how it fits into the larger narrative of building a production-grade zero-knowledge proof system.

The Message

The complete message reads:

[assistant] GPU utilization is low (4%). The proof might still be in the CPU-bound synthesis phase. Let me wait and check again: [bash] sleep 30 && nvidia-smi --query-gpu=utilization.gpu,memory.used,power.draw --format=csv 2>&1 && echo "---" && tail -3 /tmp/cuzk-daemon.log utilization.gpu [%], memory.used [MiB], power.draw [W] 4 %, 1698 MiB, 45.39 W --- [2026-02-17T15:32:12.354725Z] INFO storage_proofs_core::parameter_cache: Verify production parameters is false [2026-02-17T15:32:27.503570Z] INFO storage_proofs_core::parameter_cache: read parameters into SuprasSeal from cache "/data/zk/params/v28-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a76280..."

On its surface, this is a routine diagnostic: run nvidia-smi, observe GPU utilization, check the daemon log. But in context, it represents a critical inflection point in the development of the cuzk proving engine—the first time real hardware meets real proof work, and the developer must interpret what the machine is telling them.

Context: What Led to This Moment

To understand why this message was written, we must trace the arc of the session. The cuzk proving engine is a pipelined SNARK proving daemon designed to replace the existing ad-hoc proof generation workflow in the Curio Filecoin storage system. The developer (the assistant) had spent the previous segments designing the architecture, implementing Phase 0, and building the scaffolding: a gRPC API, a priority scheduler, a prover module wrapping filecoin-proofs-api calls, and a configuration system.

The immediate preceding messages (indices 201–224) show a tense, high-stakes validation sequence. The developer had:

  1. Built the workspace with --features cuda-supraseal to enable GPU acceleration via the SupraSeal CUDA backend, targeting an RTX 5070 Ti (Blackwell architecture) with CUDA 13.1.
  2. Started the daemon with FIL_PROOFS_PARAMETER_CACHE=/data/zk/params pointing to a pre-populated parameter cache directory containing the 45 GiB SRS (Structured Reference String) parameters required for Groth16 proving.
  3. Submitted a real proof job—a 51 MB JSON file representing the output of the C1 phase of PoRep, sent over gRPC to the daemon.
  4. Monitored the daemon logs to confirm the pipeline was executing correctly: the C1 data was deserialized, the correct proof type (StackedDrg32GiBV1_1) was registered, the parameter cache was hit, and the 45 GiB SRS was loaded into SupraSeal in approximately 15 seconds. By message 224, the daemon had entered the proving phase. The last log line showed "read parameters into SuprasSeal from cache," and then silence. No new log lines appeared, which the developer correctly interpreted as the SupraSeal prover running GPU kernels silently—the CUDA backend does not emit progress logs during computation. This is the moment message 225 is written: the proof is supposedly running on the GPU, but the developer has no visibility into whether it is actually making progress. The silence is unnerving. Is the GPU computing? Is it stuck? Is the 51 MB input still being processed, or has something gone wrong?

The Diagnostic Decision

The developer's response is to query the GPU directly using nvidia-smi. This is a pragmatic, hardware-level diagnostic that bypasses all the software layers—the gRPC server, the engine scheduler, the prover module, the filecoin-proofs-api, and the SupraSeal CUDA kernels—to ask the most fundamental question: is the GPU actually working?

The result is striking: 4% GPU utilization. The GPU is essentially idle. Memory usage is 1698 MiB—a tiny fraction of the 16,303 MiB available on the RTX 5070 Ti, and far less than the ~200 GiB peak memory the earlier analysis had predicted for the full pipeline. Power draw is a mere 45.39 watts, barely above idle.

This data point triggers an immediate hypothesis: "The proof might still be in the CPU-bound synthesis phase."

The Reasoning Process

The developer's reasoning here reveals a deep understanding of the Groth16 proof generation pipeline. The C2 phase of PoRep proving consists of several distinct stages:

  1. Deserialization — parsing the C1 output JSON and base64-decoding the inner proof data.
  2. Circuit synthesis — constructing the Rank-1 Constraint System (R1CS) circuit from the decoded witness values. This is a CPU-bound operation that involves iterating over the constraint system and populating the a/b/c vectors.
  3. GPU computation — once the circuit is synthesized, the prover transfers the data to the GPU and executes Number Theoretic Transforms (NTTs) and Multi-Scalar Multiplications (MSMs) to generate the Groth16 proof. The low GPU utilization (4%) suggests that the pipeline has not yet reached the GPU-intensive phase. The developer correctly infers that the system is still in the CPU-bound synthesis stage, where the GPU sits idle waiting for data to arrive. This reasoning is sound, but it rests on several assumptions: Assumption 1: The pipeline is sequential and synchronous. The developer assumes that circuit synthesis must complete before GPU work begins. This is true for the current architecture—the seal_commit_phase2 call is a synchronous, single-threaded operation that performs synthesis, then GPU computation, then returns the proof. There is no pipelining or overlap between CPU and GPU work in Phase 0. Assumption 2: Low GPU utilization implies CPU-bound work. This is a reasonable inference, but it is not the only possible explanation. Low GPU utilization could also indicate: - A GPU driver issue or CUDA kernel launch failure - The SupraSeal backend falling back to a CPU path - The proof being stuck on disk I/O or memory allocation - A deadlock or infinite loop in the synthesis code The developer implicitly rules these out by trusting that the earlier log lines confirm the pipeline is executing correctly. The "read parameters into SuprasSeal from cache" message is taken as evidence that the GPU backend is initialized and ready. Assumption 3: The synthesis phase is long enough to explain the delay. At the time of the first check, approximately 30–40 seconds have elapsed since the SRS was loaded. The developer assumes that synthesis for a 32 GiB sector circuit takes tens of seconds to minutes, which aligns with the earlier analysis of CPU hotpaths in the bellperson library.

The Follow-Up Check

The developer then executes a second diagnostic: wait 30 seconds and check again. This is a deliberate, structured approach to monitoring—rather than polling continuously or guessing, the developer sets a specific interval and re-checks both GPU utilization and the daemon log.

The second check shows the same result: 4% GPU utilization, 1698 MiB memory, 45.39 W power. The daemon log shows no new lines beyond the SRS loading messages. The proof is still computing, or appears to be.

This repeated check serves multiple purposes:

  1. It confirms the first observation is not a transient artifact.
  2. It establishes a baseline for how long the synthesis phase takes.
  3. It provides data for future optimization: if synthesis takes multiple minutes, that is a target for optimization.

Input Knowledge Required

To fully understand this message, a reader needs:

Knowledge of the Groth16 proving pipeline. Specifically, the distinction between CPU-bound circuit synthesis and GPU-bound NTT/MSM computation. Without this, the inference from "4% GPU utilization" to "CPU-bound synthesis" would not be obvious.

Knowledge of GPU monitoring tools. The nvidia-smi command and its output format—utilization percentage, memory usage, power draw—are standard but require familiarity with GPU hardware monitoring.

Knowledge of the cuzk architecture. The fact that the daemon uses a synchronous seal_commit_phase2 call, that SupraSeal runs CUDA kernels silently, and that the SRS was loaded successfully are all prerequisites for interpreting the diagnostic.

Knowledge of Filecoin PoRep parameters. The 32 GiB sector size, the StackedDrg32GiBV1_1 proof type, and the expected ~200 GiB peak memory footprint provide context for why the GPU memory usage (1698 MiB) seems low.

Output Knowledge Created

This message produces several valuable pieces of knowledge:

Empirical confirmation of the CPU-bound synthesis phase. The earlier analysis (in Segment 0) had identified CPU synthesis as a potential bottleneck, but this is the first real measurement. The 4% GPU utilization provides a concrete data point that synthesis is indeed the current bottleneck.

A baseline for GPU memory usage during proving. The 1698 MiB observation shows that the GPU memory footprint during the synthesis phase is minimal—the GPU is essentially waiting with only the SRS loaded. This confirms that GPU memory is not a constraint during this phase.

Evidence that the pipeline is executing correctly. The fact that the daemon is running, the GPU is responsive, and no errors have appeared in the log is itself valuable information. The developer now knows that the basic pipeline works—the proof is progressing, even if slowly.

A target for optimization. The low GPU utilization directly motivates the optimization proposals developed in later segments. If the GPU is idle 96% of the time during synthesis, then overlapping synthesis with GPU work (pipelining) or accelerating synthesis (pre-computation) could dramatically improve throughput.

The Broader Significance

This message, while brief, captures a fundamental dynamic in systems engineering: the moment when theory meets measurement. The developer had designed the cuzk engine based on an understanding of the Groth16 pipeline, but the first real proof run reveals the actual performance characteristics. The 4% GPU utilization is not a problem—it is information. It tells the developer where to focus optimization efforts.

In the subsequent messages, the developer will go on to complete the first proof (116.8 seconds), run a second proof with SRS residency (92.8 seconds, a 20.5% improvement), and then harden Phase 0 with tracing, metrics, and observability improvements. But message 225 is the moment of first contact with reality—the instant when the abstract pipeline diagram becomes a running system with measurable behavior.

The developer's response to this moment is instructive: rather than panicking or jumping to conclusions, they calmly diagnose, hypothesize, and re-check. This disciplined approach to empirical validation is the hallmark of experienced systems engineering. The 4% GPU utilization puzzle is solved not by guessing, but by understanding the pipeline well enough to know what "normal" looks like at each stage.

Conclusion

Message 225 of the cuzk coding session is a small but revealing artifact. It shows a developer in the act of learning from a running system—interpreting hardware metrics, forming hypotheses, and building an empirical understanding of performance. The message's brevity belies its significance: it marks the transition from "it compiles and the pipeline works" to "it produces real proofs with measurable performance characteristics." This is the moment when the cuzk proving engine stops being a design and starts being a real, measurable system.