When Automated Research Fails: A Case Study in Direct Code Investigation

Introduction

In the course of optimizing a high-performance Groth16 proof generation pipeline for Filecoin's Proof-of-Replication (PoRep) protocol, an AI assistant encountered a critical moment where its automated research infrastructure failed to deliver results. The response—a brief pair of tool calls that constitutes message [msg 2707] in the conversation—reveals a fascinating interplay between automated and direct investigation strategies, and provides a window into the assistant's problem-solving methodology when faced with incomplete information.

This article examines that single message in depth: why it was written, what assumptions drove it, the knowledge it required and produced, and the broader context of the optimization effort it served.

Context: The Phase 11 Optimization Effort

To understand message [msg 2707], one must first understand the problem it was trying to solve. The assistant had been engaged in a multi-phase optimization of the cuzk SNARK proving engine—a system that generates Groth16 proofs for Filecoin storage proofs. Through Phases 7 through 10, the assistant had progressively improved throughput by refining the per-partition dispatch architecture, implementing dual-worker GPU interlocking, optimizing PCIe transfers, and attempting (and ultimately abandoning) a two-lock GPU interlock design.

The Phase 10 failure was particularly instructive. The assistant had designed a sophisticated two-lock architecture intended to allow two GPU workers to overlap their execution, but discovered fundamental CUDA device-global synchronization conflicts that made the approach unworkable. After reverting to the Phase 9 single-lock approach and running comprehensive benchmarks, a detailed waterfall timing analysis revealed the true bottleneck: DDR5 memory bandwidth contention.

The data showed that at high concurrency (c=15 to c=20), throughput plateaued at approximately 38 seconds per proof, with both synthesis time and prep_msm time inflating significantly under load. The root cause was not raw bandwidth exhaustion—the system's DDR5 memory channels could theoretically deliver ~333 GB/s, while the pipeline demanded only ~34 GB/s. Instead, the contention manifested through three interrelated mechanisms:

1. TLB shootdown storms from concurrent munmap() calls in the async deallocation path
2. Thread pool oversubscription as two separate thread pools (Rust's rayon for synthesis and C++'s groth16_pool for prep_msm/b_g2_msm) both defaulted to 192 threads, competing for the same 12 L3 cache domains
3. Phase overlap as memory-heavy operations (synthesis SpMV, prep_msm bitmap classification, b_g2_msm Pippenger) all fired simultaneously The assistant proposed Phase 11 with three interventions: bounding async deallocation to a single thread, reducing the groth16_pool thread count, and adding a lightweight semaphore interlock to stagger memory-heavy phases. The user approved this plan but added a crucial caveat in [msg 2704]: "We should be careful to not kill parallelism btw."

The Critical Reassessment

Message [msg 2705] shows the assistant responding to this caveat with a detailed analysis of where parallelism is critical (and must not be throttled) versus where it is wasteful or harmful. Among the key insights was an observation about the groth16_pool thread pool:

groth16_pool at 192 threads — prep_msm with num_circuits=1 (per-partition mode) has very little to parallelize. The par_map(num_circuits, ...) calls with num_circuits=1 just run on 1 thread anyway. The actual parallelism within single-circuit prep_msm is in the bitmap loop, which iterates over bit_vector_size entries (~360K). Even 16-32 threads saturate this. 192 threads is wasteful for num_circuits=1; 16-32 is sufficient.

This assertion—that par_map with num_circuits=1 runs on a single thread—was critical to the optimization strategy. If true, it meant that reducing the groth16_pool thread count would have minimal impact on prep_msm latency while freeing L3 cache capacity and memory bandwidth for the synthesis workers. But the assistant needed to verify this assumption against the actual code.

To that end, the assistant spawned a subagent task (visible at the end of [msg 2705]) to research the prep_msm parallelism details, specifically asking about the par_map behavior with num_circuits=1 and the bitmap loop parallelism. This task, with ID ses_3888e88e2ffeaCMhEM1Mu68lQu, was designed to read the relevant source files and return detailed findings.

The Empty Response

Message [msg 2706] shows the assistant receiving the task result:

<task_result>

</task_result>

The task returned empty—no content whatsoever. This is a critical failure mode in the assistant's research architecture. The subagent system, which normally provides detailed analysis from reading source code, produced nothing. The reasons could be numerous: the subagent might have encountered a parsing error, timed out, failed to find the requested information, or suffered a communication breakdown in the multi-turn conversation.

What matters is not why the task failed, but how the assistant responded to that failure.

Message [msg 2707]: Direct Investigation

The subject message, [msg 2707], is the assistant's immediate response to the empty task result:

`` [assistant] [glob] **/thread_pool_t.* No files found [read] /home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu <path>/home/theuser/curio/extern/supraseal-c2/cuda/groth16_cuda.cu</path> <type>file</type> <content>360: } 361: 362: // populate bitmaps for batch additions, bases and scalars for tail msms 363: get_groth16_pool().par_map(num_circuits, [&](size_t c) { 364: auto& prover = provers[c]; 365: auto& l_bit_vector = split_vectors_l.bit_vector[c]; 366: auto& a_bit_vector = split_vectors_a.bit_vector[c]; 367: auto& b_bit_vector = sp... ``

This message contains two tool calls executed in parallel:

1. A glob search for thread_pool_t.* across the entire codebase, which returned "No files found"
2. A file read of groth16_cuda.cu starting at line 360, which returned the bitmap population code The glob search is notable because it reveals an assumption the assistant was making: that the thread pool implementation was in a file named thread_pool_t.*. This assumption was based on the code structure—the get_groth16_pool() function returns a thread_pool_t object, and the assistant expected the class definition to be in a correspondingly named file. The fact that no such file was found (the actual file was at sppark/util/thread_pool_t.hpp, which the glob pattern **/thread_pool_t.* should have matched—but the assistant was searching from the supraseal-c2 directory, and the file was in the supraseal dependencies directory, which may not have been in the search path). The file read, meanwhile, targeted the bitmap population code—the very code path the assistant needed to understand. Lines 362-367 show the par_map call that populates bitmaps for batch additions and bases and scalars for tail MSMs. This is the inner loop of prep_msm, and the assistant needed to see how par_map behaves when num_circuits=1.## Why Direct Investigation? The assistant's decision to switch from automated subagent research to direct tool calls reveals several important aspects of its reasoning architecture. First, the assistant operates under a synchronous round-based model. In each round, it can issue multiple tool calls in parallel, but it must wait for ALL results before proceeding. The subagent task (task tool) is a special case: it spawns a complete multi-turn conversation that runs to completion before returning its result. When the task returned empty, the assistant had to decide what to do next. It could have re-spawned the task with different parameters, asked the user for help, or—as it chose—investigate directly. The direct approach has a crucial advantage: parallelism. By issuing both a glob search and a file read in the same round, the assistant could gather information from two sources simultaneously. More importantly, the file read is a simple, deterministic operation that returns immediately, whereas a subagent task involves multiple rounds of reasoning and could fail again. Second, the assistant's choice reflects a pragmatic trade-off between depth and reliability. The subagent task was designed to provide comprehensive analysis—reading multiple files, tracing code paths, and synthesizing findings. But when it failed, the assistant fell back to the most reliable primitive operations: glob and read. These tools have no moving parts, no reasoning chain to go wrong, and no dependency on the subagent's conversation coherence.

Assumptions Embedded in the Message

Message [msg 2707] makes several assumptions that are worth examining:

Assumption 1: The glob pattern **/thread_pool_t.* would find the file

The assistant assumed that the thread pool implementation header would be named thread_pool_t.hpp or thread_pool_t.h and would be discoverable via a recursive glob from the current working directory. In fact, the file existed at /home/theuser/curio/extern/supraseal/deps/sppark/util/thread_pool_t.hpp—a path that the glob should have matched if the search started from the supraseal directory. The fact that "No files found" was returned suggests either the search was scoped to a different directory or the glob implementation had limitations.

This is a minor mistake—the assistant assumed a file naming convention that turned out to be correct (the file was named thread_pool_t.hpp), but the search scope was wrong. The assistant's subsequent behavior in [msg 2708] (searching with find instead of glob) confirms that it recognized the glob had failed and tried a different approach.

Assumption 2: Reading line 360 of groth16_cuda.cu would reveal the relevant code

The assistant chose to read starting at line 360, which is the bitmap population section. This was a reasonable choice based on the research prompt in [msg 2705], which asked about "the bitmap loop, which iterates over bit_vector_size entries (~360K)." However, the assistant didn't read the par_map function definition itself—it only read the call site. To fully understand whether par_map with num_circuits=1 runs on a single thread, one would need to see the implementation of par_map in the thread pool class.

Assumption 3: The code structure follows expected patterns

The assistant assumed that get_groth16_pool().par_map(...) is the primary parallelization mechanism for prep_msm, and that understanding its behavior with num_circuits=1 would be sufficient to determine the optimal thread count. This assumption was sound, but it didn't account for other parallel work happening within the same thread pool (such as the b_g2_msm Pippenger computation, which the assistant later discovered uses all 192 threads simultaneously).

Input Knowledge Required

To understand message [msg 2707], a reader needs:

1. Knowledge of the codebase structure: That groth16_cuda.cu is the main CUDA implementation file for Groth16 proof generation, and that get_groth16_pool() returns a thread pool used for CPU-side preprocessing.
2. Knowledge of the optimization context: That the assistant is investigating memory bandwidth contention in Phase 11, and specifically needs to understand the parallelism characteristics of prep_msm when operating on a single circuit.
3. Knowledge of the subagent system: That the task tool spawns a multi-turn subagent conversation, and that an empty result indicates a failure in that subagent's execution.
4. Knowledge of the glob tool: That **/thread_pool_t.* is a recursive glob pattern that should match any file named thread_pool_t.* in subdirectories.
5. Understanding of the par_map abstraction: That par_map(n, f) typically partitions n items across available threads, and that when n=1, there is only one item to process, which likely runs on a single thread regardless of pool size.

Output Knowledge Created

The message produced two pieces of knowledge:

1. Negative knowledge: The glob search confirmed that thread_pool_t.* was not found in the expected location. This is useful negative information—it tells the assistant that the thread pool implementation is either named differently, located in a different directory tree, or stored in a non-standard format.
2. Positive knowledge: The file read confirmed the exact code at the bitmap population call site (lines 362-367). This shows the par_map call structure and the lambda it executes, confirming that the bitmap population is parallelized across circuits via par_map. However, the message did not produce the knowledge the assistant was seeking. The assistant needed to know whether par_map with num_circuits=1 runs on a single thread or still spawns multiple threads. The file read only showed the call site, not the implementation of par_map. The glob search failed to find the implementation file. So the message represents an incomplete investigation—it gathered partial information but didn't answer the core question.

The Thinking Process

The assistant's reasoning in this message can be reconstructed as follows:

1. Detect failure: The subagent task returned empty. The assistant cannot proceed with the planned analysis without understanding the par_map parallelism behavior.
2. Formulate fallback strategy: Direct tool calls are more reliable than subagent tasks for simple information retrieval. Use a glob to find the thread pool implementation, and a file read to examine the relevant code section.
3. Execute in parallel: Both tools are independent—the glob doesn't depend on the file read, and vice versa. Issuing them in the same round maximizes efficiency.
4. Interpret results: The glob failed, but the file read succeeded. The assistant now has partial information. It will need to follow up (as it does in [msg 2708]) with a more targeted search. The thinking is pragmatic and adaptive. Rather than dwelling on the subagent failure or re-invoking it with different parameters, the assistant immediately shifts to a more direct approach. This reflects a design philosophy where automated reasoning is preferred for complex analysis, but simple tool primitives serve as a reliable fallback.

Broader Significance

Message [msg 2707] is a small moment in a long optimization conversation, but it illustrates a fundamental pattern in human-AI collaboration: the interaction between automated and direct investigation. The assistant's architecture includes sophisticated subagent capabilities for multi-turn research, but when those capabilities fail, the assistant falls back to the most basic operations—reading files and searching for patterns.

This pattern mirrors how human engineers work: when a complex analysis tool or script fails, they open the source file directly and look at the relevant lines. The assistant's behavior in this message demonstrates that it has learned this pattern, and that its tool-use architecture supports graceful degradation from complex to simple investigation methods.

The message also highlights the importance of redundant investigation strategies. The assistant didn't just try one approach—it tried two in parallel (glob + file read), and when the glob failed, it was positioned to try a third approach (the find command in [msg 2708]). This redundancy is essential for robust operation in complex codebases where file locations and naming conventions are not always predictable.

Conclusion

Message [msg 2707] captures a moment of adaptation in a high-stakes optimization effort. When automated research failed, the assistant pivoted to direct investigation, issuing parallel tool calls to gather information about the par_map parallelism behavior in the groth16_pool thread pool. While the message didn't fully answer the question it was pursuing, it demonstrated a robust fallback strategy and set the stage for the successful investigation that followed in subsequent messages.

The message's true value lies not in what it found, but in what it reveals about the assistant's problem-solving methodology: when complex tools fail, use simple ones; when automated reasoning stalls, read the source directly; and when one search strategy fails, try another. These principles are as applicable to human engineers as they are to AI assistants.