Tracing the Thread: How One Read Command Uncovered the Parallelism Architecture of SUPRASEAL_C2

Introduction

In the middle of a deep-dive investigation into the SUPRASEAL_C2 Groth16 proof generation pipeline for Filecoin PoRep, a single assistant message stands out as a pivotal moment of discovery. Message index 10 in this conversation is deceptively simple on its surface: the assistant issues a read command to examine the CPU mult_pippenger implementation in pippenger.hpp. But this read operation is far from routine. It represents the culmination of a methodical, multi-step tracing effort to understand how parallelism actually flows through one of the most memory-intensive components of the Filecoin proving pipeline — the b_g2 multi-scalar multiplication (MSM) step. To appreciate why this particular read was necessary, and what it reveals, we must reconstruct the full investigative journey that led to it.

The Question That Started It All

The conversation begins with a user posing a remarkably precise research question ([msg 0]). The user wants to understand the prep_msm parallelism in groth16_cuda.cu specifically for the single-circuit case — that is, when num_circuits=1, which corresponds to the per-partition pipeline mode. This is not an idle curiosity. The SUPRASEAL_C2 pipeline is known for its ~200 GiB peak memory footprint, and understanding where parallelism is (or is not) being exploited is critical to diagnosing bottlenecks and designing memory-reduction strategies.

The user's question breaks down into four sub-questions:

1. How does the first par_map(1, ...) at lines 210-305 behave with one circuit?
2. How does the second par_map(1, ...) at lines 363-540 behave with one circuit?
3. For the b_g2 MSM at lines 553-581, does mult_pippenger use the thread pool when num_circuits=1?
4. What does thread_pool_t::par_map actually do when called with n=1? These questions are deceptively deep. They probe the boundary between two different parallelism strategies: inter-circuit parallelism (parallelizing across multiple independent proof circuits) versus intra-circuit parallelism (parallelizing within a single circuit's computation). When num_circuits=1, the inter-circuit dimension collapses, and the question becomes whether the code falls back to intra-circuit parallelism or simply runs single-threaded.

The Assistant's Investigative Method

The assistant's response to this question reveals a systematic, tool-driven investigative methodology. Rather than attempting to answer from general knowledge or static analysis alone, the assistant embarks on a live code-reading expedition across multiple files in the SUPRASEAL codebase.

Phase 1: Reading the primary source ([msg 1]). The assistant starts by reading groth16_cuda.cu at the exact line ranges the user specified. This is the obvious first step — go to the source. But the assistant doesn't just read the requested lines; it reads broader sections to understand context. The file is large and complex, and the assistant encounters the first obstacle: a glob command to find thread_pool_t.hpp fails because the working directory doesn't match the expected path.

Phase 2: Navigating the filesystem (<msg id=2-5>). The glob failure triggers a troubleshooting sequence. The assistant tries broader glob patterns, then reads directory listings to understand the project structure. It discovers that supraseal is a separate repository from supraseal-c2, and the deps directory contains sppark (Supranational's SPPark library). After navigating this structure, the assistant finally locates thread_pool_t.hpp at /home/theuser/curio/extern/supraseal/deps/sppark/util/thread_pool_t.hpp. This filesystem navigation is itself revealing: the codebase has a non-trivial dependency structure, with the thread pool living in a different repository tree than the main CUDA code.

Phase 3: Reading the thread pool implementation ([msg 6]). With the file located, the assistant reads thread_pool_t.hpp to understand par_map. This is the foundational layer — understanding how the thread pool dispatches work determines everything else. The assistant now has the par_map implementation, which will answer the user's question #4 about what happens when n=1.

Phase 4: Tracing mult_pippenger (<msg id=7-9>). The assistant now turns to the user's question #3 about mult_pippenger. A grep for mult_pippenger initially returns no results (likely a tool quirk), but a retry succeeds, revealing four matches. Crucially, the matches show two different mult_pippenger variants: one in pippenger.cuh (the GPU/CUDA version) and one in pippenger.hpp (the CPU version). The grep also reveals a comment at line 575 of groth16_cuda.cu: true, &get_groth16_pool()); // single circuit: use full thread pool. This comment is a goldmine — it explicitly documents the design intent for the single-circuit case.

The Subject Message: Reading the CPU mult_pippenger

This brings us to message 10, the subject of this article. The assistant states:

Now let me read the CPU mult_pippenger in pippenger.hpp which is the one used in the non-CUDA path

And then issues a read command for /home/theuser/curio/extern/supraseal/deps/sppark/msm/pippenger.hpp, showing lines 210-221 of the file.

Why This Read Was Necessary

The assistant could have stopped after finding the mult_pippenger declaration in pippenger.cuh (the GPU version). But it didn't. Why? Because the call site in groth16_cuda.cu at lines 553-581 is executing on the CPU, not the GPU. The b_g2 MSM is explicitly described in the code comments as running "on CPU, parallelized across circuits." The GPU mult_pippenger in pippenger.cuh is a CUDA kernel launcher — it dispatches work to the GPU. But the b_g2 MSM is a CPU-side computation, so it must be calling a different mult_pippenger overload.

The assistant's reasoning here is sophisticated: it recognizes that the same function name can refer to different implementations depending on the compilation context (CPU vs. GPU), and it correctly infers that the CPU version in pippenger.hpp is the one actually invoked from the b_g2 code path. This is a non-trivial insight that requires understanding how template libraries and header inclusion work in C++.

What the Read Revealed

The content shown in the message is only the tail end of pippenger.hpp (lines 210-221), showing:

210:             if (is_bit::set(scalar, top))
211:                 ret.add(point);
212:         }
213:     }
214: }
215: 
216: #include <util/thread_pool_t.hpp>
217: 
218: template <class bucket_t, class point_t, class scalar_t,
219:           class affine_t = class bucket_t::affine_t>
220: static void mult_pippenger(point_t& ret, const affine_t points[], size_t npoints,
221:              ...

The critical detail here is line 216: #include <util/thread_pool_t.hpp>. This include directive, placed inside the header file after some helper functions but before the mult_pippenger template definition, tells us that the CPU mult_pippenger implementation depends on the thread pool. The template signature (lines 218-220) shows it takes a point_t& ret (output reference), const affine_t points[] (input points array), size_t npoints (number of points), and then the parameter list continues beyond what was captured.

The include of thread_pool_t.hpp at line 216 is particularly significant because it means the CPU mult_pippenger implementation has direct access to the thread pool's par_map and other parallel primitives. This confirms that the CPU mult_pippenger can, and likely does, use the thread pool internally for parallelizing its own computation — which is exactly what the user's question #3 is about.

The Broader Context: What Came Before and After

Message 10 sits at a critical juncture in the investigation. Before it, the assistant has:

• Read the main CUDA file and seen the three par_map call sites
• Located and read the thread pool implementation
• Found the mult_pippenger call sites and the revealing comment about single-circuit mode After message 10, the assistant continues with:
• Message 11: Checking get_groth16_pool() to understand how the thread pool is configured (environment variable CUZK_GPU_THREADS, number of threads, etc.)
• Message 12: Reading the get_groth16_pool() implementation in detail The full picture that emerges is of a carefully designed parallelism architecture with multiple layers: 1. Inter-circuit parallelism via par_map: When num_circuits > 1, the two par_map calls (lines 215 and 363) distribute work across circuits, with each circuit's work running on a separate thread from the pool. 2. Intra-circuit parallelism via mult_pippenger: When num_circuits == 1, the par_map calls degenerate to single-threaded execution (since there's only one circuit to process), but the b_g2 MSM at line 575 explicitly passes the thread pool to mult_pippenger, allowing it to use all available threads internally. 3. The design trade-off: The comment at lines 553-555 explains the rationale: "With batch_size=N, running N single-threaded Pippengers in parallel is much faster than running N sequential thread-pooled Pippengers." This reveals a deliberate design choice — when you have multiple circuits, it's better to run one Pippenger per circuit without internal parallelism (to avoid thread contention), but when you have a single circuit, you want the Pippenger to use all available threads.

Assumptions and Their Validity

The assistant made several assumptions in message 10, all of which appear well-founded:

Assumption 1: The CPU mult_pippenger in pippenger.hpp is the one used in the non-CUDA path. This is correct — the b_g2 MSM is explicitly a CPU computation, and the GPU mult_pippenger in pippenger.cuh would not be invoked from that context.

Assumption 2: The file path /home/theuser/curio/extern/supraseal/deps/sppark/msm/pippenger.hpp is correct. This is validated by the successful read.

Assumption 3: The CPU mult_pippenger is relevant to answering the user's question. This is correct — the user asked about the b_g2 MSM at lines 553-581, which calls mult_pippenger with the thread pool parameter.

One subtle point worth noting: the assistant refers to "the non-CUDA path," but the code in groth16_cuda.cu is itself CUDA code that mixes CPU and GPU operations. The b_g2 MSM runs on the CPU even though it's in a .cu file. This is a common pattern in heterogeneous computing where a .cu file contains both host code (CPU) and device code (GPU).

Input Knowledge Required

To fully understand message 10 and its significance, a reader needs:

1. Knowledge of the SUPRASEAL_C2 project: Understanding that this is a Groth16 proof generation pipeline for Filecoin PoRep, with a ~200 GiB peak memory footprint, and that the prep_msm section is a critical code path.
2. Knowledge of Pippenger's algorithm: Multi-scalar multiplication (MSM) is the computational bottleneck in many zero-knowledge proof systems. Pippenger's algorithm (also known as the "bucket method") is an optimized approach that groups scalars by bit windows. Understanding that mult_pippenger is implementing this algorithm is essential.
3. Knowledge of C++ template metaprogramming and CUDA: The distinction between pippenger.cuh (CUDA header with GPU kernels) and pippenger.hpp (CPU implementation) is subtle but critical. The .cuh extension conventionally indicates CUDA-specific code.
4. Knowledge of thread pool parallelism patterns: Understanding par_map as a parallel-for primitive and recognizing when it provides inter-circuit vs. intra-circuit parallelism.
5. Knowledge of the Filecoin PoRep context: Understanding that Filecoin's Proof-of-Replication (PoRep) requires proving that a miner is storing a unique copy of data, and that Groth16 proofs are used to compress these proofs efficiently.

Output Knowledge Created

Message 10, combined with the surrounding investigation, creates several valuable pieces of knowledge:

1. The CPU mult_pippenger signature and location: The assistant now knows exactly where to find the CPU implementation and what its template signature looks like.
2. The dependency chain: pippenger.hpp includes thread_pool_t.hpp, confirming that the CPU Pippenger has access to thread pool parallelism.
3. The design rationale for single-circuit mode: The comment at line 575 explicitly documents that single-circuit mode should use the full thread pool, contrasting with multi-circuit mode where each circuit gets its own single-threaded Pippenger.
4. A map of the codebase architecture: The investigation has traced the relationships between groth16_cuda.cu, thread_pool_t.hpp, pippenger.cuh, and pippenger.hpp, creating a mental model of how parallelism flows through the system.

The Thinking Process Revealed

The assistant's reasoning in message 10 is visible through its actions and phrasing. The key insight is the phrase "the CPU mult_pippenger in pippenger.hpp which is the one used in the non-CUDA path." This reveals that the assistant:

1. Recognized that there are multiple mult_pippenger implementations
2. Distinguished between the GPU version (in pippenger.cuh) and the CPU version (in pippenger.hpp)
3. Correctly identified which one is relevant to the b_g2 MSM code path
4. Understood that the b_g2 MSM runs on the CPU despite being in a .cu file This is not a trivial chain of reasoning. It requires understanding how C++ name resolution works across multiple headers, how template specializations can differ between compilation units, and how heterogeneous CUDA code mixes host and device execution contexts. The assistant also demonstrates a methodical, trace-driven approach: it doesn't jump to conclusions or guess. It reads the actual code, traces the call chain, and only then forms conclusions. This is visible in the progression from the initial read of groth16_cuda.cu through the filesystem navigation to the targeted reads of specific implementation files.

Conclusion

Message 10 is a deceptively simple tool call that represents the culmination of a sophisticated investigative process. It is the moment when the assistant connects the call site in groth16_cuda.cu to the actual implementation in pippenger.hpp, confirming the dependency chain and setting the stage for answering the user's questions about parallelism behavior. The message embodies the essence of good systems research: start with a precise question, trace the code methodically, verify assumptions by reading actual implementations, and build a mental model of how the system works at every layer. For anyone seeking to understand the SUPRASEAL_C2 parallelism architecture, this message and the investigation surrounding it provide a masterclass in code reading and systems analysis.