The Feature Flag Epiphany: Debugging Dependency Conventions in a Multi-Crate Rust Workspace

Introduction

In the course of implementing a complex pipelined proving engine for Filecoin's Groth16 zero-knowledge proofs, an AI assistant encountered a seemingly mundane but deeply instructive build failure. The error message was terse: storage-proofs-porep did not have the cuda-supraseal feature that the assistant had just declared as a dependency. The response—a single message at index 453 in the conversation—captured a moment of realization that reveals the hidden complexity of Rust's feature flag system when working across a multi-crate workspace with conditional GPU compilation. This article examines that message in depth: what prompted it, what assumptions it corrected, and what it teaches about the gap between a developer's mental model and the actual dependency graph.

The Broader Context: Building a Pipelined Prover

The message sits within a much larger effort: Phase 2 of the "cuzk" proving engine, a pipelined architecture designed to replace the monolithic PoRep C2 prover used in Filecoin's Curio storage mining stack. The central innovation of Phase 2 was splitting the proof generation into two stages—CPU-based circuit synthesis and GPU-based proof computation—so that synthesis for one partition could overlap with GPU work for another, dramatically reducing peak memory from ~136 GiB to ~13.6 GiB per partition.

To achieve this split, the assistant had already created a new srs_manager.rs module for direct SRS (Structured Reference String) parameter loading, bypassing the private GROTH_PARAM_MEMORY_CACHE that the monolithic prover relied on. The next step was wiring up the Rust dependency graph to support the new pipeline module (pipeline.rs), which needed access to types from filecoin-proofs, storage-proofs-core, storage-proofs-porep, storage-proofs-post, storage-proofs-update, bellperson, blstrs, rayon, and ff. This required editing both the workspace-level Cargo.toml and the cuzk-core/Cargo.toml to declare these dependencies with appropriate feature flags.

The Build Failure: A Feature That Doesn't Exist

The assistant's first attempt at updating the dependencies (messages 449-450) added the new crates to the workspace manifest and to cuzk-core's manifest. The feature flag configuration was straightforward in the assistant's mind: since the entire pipeline was being built for the SupraSeal GPU backend, all the storage-proofs crates should be compiled with cuda-supraseal features enabled. The assistant wrote:

storage-proofs-porep = { version = "19.0", features = ["cuda-supraseal"] }

When the assistant ran cargo check --workspace --no-default-features to verify the build (message 451), it received a clear error:

error: failed to select a version for `storage-proofs-porep`.
...
the package `cuzk-core` depends on `storage-proofs-porep`, with features: 
`cuda-supraseal` but `storage-proofs-porep` does not have these features.

This is a classic Rust dependency resolution failure. The resolver could not find a version of storage-proofs-porep that satisfies the feature request because that crate simply does not define a cuda-supraseal feature. The error is unambiguous, but its root cause requires understanding the conventions used across the Filecoin proof library ecosystem.

The Investigation: Mapping the Feature Matrix

The assistant's response to the error (message 452) was methodical. Rather than guessing, it queried the actual feature declarations of every relevant crate by iterating over them with a shell loop:

for pkg in filecoin-proofs storage-proofs-core storage-proofs-porep \
    storage-proofs-post storage-proofs-update; do
    echo "=== $pkg ==="
    grep -A 30 '^\[features\]' ~/.cargo/registry/src/.../${pkg}-19.0.1/Cargo.toml
done

This produced a clear picture of the feature flag landscape. The output revealed a deliberate asymmetry in how the Filecoin proof libraries organize their feature flags:

The Subject Message: A Moment of Clarification

This brings us to the subject message (msg 453), the target of this analysis. The assistant wrote:

I see — only filecoin-proofs and storage-proofs-core have cuda-supraseal. The porep, post, and update crates only have cuda. And for cuda-supraseal, filecoin-proofs enables cuda (not cuda-supraseal) on porep/post/update. Let me fix the feature flags: [edit] /home/theuser/curio/extern/cuzk/cuzk-core/Cargo.toml Edit applied successfully.

The message is deceptively short. In three sentences, the assistant:

  1. Acknowledges the discovery — it now sees the actual feature flag structure.
  2. Articulates the pattern — the asymmetry between higher and lower crates.
  3. Acts on the insight — by editing the Cargo.toml to use cuda instead of cuda-supraseal for the porep/post/update dependencies. The edit itself, while not shown verbatim in the message, changed the feature flags from cuda-supraseal to cuda for storage-proofs-porep, storage-proofs-post, and storage-proofs-update, while keeping cuda-supraseal for filecoin-proofs and storage-proofs-core. This is the correct configuration: the higher crates get the SupraSeal-specific feature, and the lower crates get plain CUDA support, exactly matching how filecoin-proofs itself propagates features internally.

Assumptions Made and Corrected

The assistant's initial approach reveals several implicit assumptions:

Assumption 1: Feature naming is consistent across crates. The assistant assumed that if filecoin-proofs has a cuda-supraseal feature, then all its transitive dependencies would have the same feature name. This is a natural assumption for someone new to a codebase — feature flags often follow consistent conventions within a project.

Assumption 2: The dependency graph is flat. The assistant treated all storage-proofs crates as peers that should receive the same feature flags. In reality, the dependency graph has a hierarchy: filecoin-proofs sits above storage-proofs-core, which sits above storage-proofs-porep/post/update. Feature flags are not uniformly distributed across this hierarchy.

Assumption 3: Feature propagation is automatic. The assistant may have assumed that enabling cuda-supraseal on filecoin-proofs would automatically propagate the right features downward. But Rust's feature resolution doesn't work that way — each direct dependency must explicitly declare which features it needs. The assistant had to manually match the propagation pattern that filecoin-proofs already defines.

Assumption 4: The build error indicated a version mismatch. Initially, the error message could have been misinterpreted as a version resolution problem (the "failed to select a version" wording is misleading). The assistant correctly diagnosed it as a feature availability issue rather than a versioning issue.

The Thinking Process: From Error to Insight

The reasoning visible in messages 451-453 follows a clear diagnostic pattern:

  1. Observe the error (msg 451): cargo check fails with a feature-not-found error for storage-proofs-porep.
  2. Hypothesize the cause (msg 452): The assistant correctly hypothesizes that the issue is feature flag naming, not versioning. It states: "The issue is that storage-proofs-porep doesn't have cuda-supraseal feature directly."
  3. Gather evidence (msg 452): Rather than making assumptions, the assistant queries the actual feature declarations of all five relevant crates using a shell loop. This is the critical step — it replaces speculation with data.
  4. Analyze the evidence (msg 453): The assistant processes the output and identifies the pattern: higher crates have cuda-supraseal, lower crates only have cuda, and the higher crates propagate cuda (not cuda-supraseal) to the lower crates.
  5. Apply the fix (msg 453): The assistant edits the Cargo.toml to match the actual feature flag structure.
  6. Verify the fix (msg 454): The next cargo check succeeds, confirming the correction. This is textbook debugging: form a hypothesis, gather data, analyze, fix, verify. The assistant's use of a shell loop to inspect all crate feature declarations simultaneously is particularly elegant — it avoids the trap of checking only one crate and assuming the pattern holds.

Input Knowledge Required

To understand and appreciate this message, a reader needs:

Output Knowledge Created

This message, combined with the surrounding investigation, creates several pieces of knowledge:

Significance and Impact

While this message is only three sentences long and one file edit, it represents a critical inflection point in the Phase 2 implementation. Without this fix, the entire pipeline module would remain uncompilable. The assistant could have taken a different approach — for example, blindly trying cuda instead of cuda-supraseal on all crates — but that would have been incorrect for filecoin-proofs and storage-proofs-core, which genuinely need the SupraSeal-specific feature. The systematic investigation prevented a half-correct fix.

The message also demonstrates an important principle of working with large Rust workspaces: feature flags are not just boolean toggles but a distributed configuration language with conventions that vary across crate boundaries. What looks like a naming inconsistency is often a deliberate design choice that reflects the layered architecture of the software. The lower-level crates (porep, post, update) abstract over the GPU backend; they don't need to know whether the GPU is driven by native CUDA or SupraSeal. The higher-level crates (filecoin-proofs, storage-proofs-core) make that distinction because they orchestrate the actual proving pipeline.

The successful compilation that followed (message 454) confirmed the fix and allowed the assistant to proceed with implementing the core pipeline logic: the SynthesizedProof type, the synthesize_porep_c2_partition() function, the gpu_prove() function, and the engine refactoring to support pipeline mode. All of this depended on the dependency graph being correctly wired.

Conclusion

Message 453 is a masterclass in responding to a build error with precision. Rather than guessing or applying a superficial fix, the assistant diagnosed the root cause — a mismatch between assumed and actual feature flag conventions — by gathering empirical data across the entire dependency tree. The fix was then targeted and correct: change only the feature flags that were wrong, keep the ones that were right, and match the propagation pattern already established by the upstream crate.

In a broader sense, this message illustrates that even in AI-assisted coding, the most valuable skill is not writing code but understanding the system. The build error was a surface symptom; the real problem was a gap in the assistant's mental model of how the Filecoin proof libraries organize their conditional compilation. Closing that gap required curiosity, systematic investigation, and the willingness to let the data override assumptions. These are the same qualities that distinguish effective debugging in any programming context, whether human or AI.