The Art of Diagnostic Precision: Isolating Test Pollution in a Distributed Storage System

Introduction

In the midst of a massive test coverage initiative—164 new unit tests across eight files spanning configuration loading, CIDgravity client logic, deal repair, S3 authentication, and more—a single failing test threatened to undermine the entire effort. The message at index 2440 captures a moment of quiet diagnostic rigor: the assistant, having observed a test failure that appeared and disappeared depending on how tests were run, methodically isolates the problem by running each package's tests individually. This brief but revealing message demonstrates a fundamental principle of software debugging: when a test passes in isolation but fails in a group, you are almost certainly dealing with test pollution—shared mutable state leaking across test boundaries.

The Message in Full

The assistant writes:

The test passes individually. This might be test pollution between tests. Let me run all our new tests individually:

`` cd /home/theuser/gw && go test ./configuration/ --count=1 2>&1 && echo "---CONFIG DONE---" && go test ./cidgravity/ --count=1 2>&1 && echo "---CIDG DONE---" && go test ./server/s3/ --count=1 2>&1 && echo "---S3 DONE---" ``

Result: `` ok github.com/CIDgravity/filecoin-gateway/configuration 0.010s ---CONFIG DONE--- ok github.com/CIDgravity/filecoin-gateway/cidgravity 0.267s ---CIDG DONE--- ok github.com/CIDgravity/filecoin-gateway/server/s3 0.016s ---S3 DONE--- ``

All three packages pass cleanly when tested individually. The message does not show the rbdeal package being run again (that was already confirmed passing in isolation in the preceding message), but the implication is clear: the failure only manifests when certain packages are tested together.

The Reasoning and Motivation

The message was written in response to a puzzling observation. In message 2437, the assistant ran a combined test command targeting four packages simultaneously:

go test ./configuration/ ./cidgravity/ ./rbdeal/ ./server/s3/ --count=1

The output showed FAIL for the rbdeal package, specifically TestGC_Disabled. Yet when the assistant immediately ran that same test in isolation (message 2439):

go test ./rbdeal/ -v -run TestGC_Disabled --count=1

It passed without issue. This contradictory behavior—a test that fails only when run alongside other packages' tests—is the classic signature of test pollution.

The assistant's reasoning, stated explicitly in the message, is: "This might be test pollution between tests." This is a precise diagnostic hypothesis. Test pollution occurs when one test modifies global state—environment variables, Prometheus metric registries, database connections, file system state, or singleton instances—that another test depends on. Because Go's go test command runs tests within the same process (unless explicitly using -parallel or -count flags in specific ways), shared state can persist across test functions, across files, and even across packages.

The motivation for this message is therefore twofold. First, the assistant needs to confirm the hypothesis by demonstrating that each package passes individually. Second, and more importantly, the assistant is gathering evidence to narrow down which package's tests are causing the pollution. By running each package separately and confirming all pass, the assistant establishes that no package has an inherently broken test. The pollution must arise from an interaction between packages—likely from global state initialized in one package's test setup that corrupts the state expected by another package's tests.## The Context: A Massive Test Coverage Initiative

To understand the significance of this diagnostic moment, one must appreciate the context. The assistant had just completed a systematic, multi-phase test coverage initiative, creating 164 new unit tests across eight new test files. The testing plan, documented in testing-plan.md, spanned multiple phases:

Phase 1.1: Configuration tests (14 tests) — verifying default values, environment variable overrides, and YAML loading for the RibsConfig structure.
Phase 1.2: CIDgravity client tests (15 tests) — testing the GetBestAvailableProviders API client with mock HTTP servers, including error handling, empty responses, and fallback logic.
Phase 1.3: Fixing skipped tests in rbstor/basic_test.go — enabling integration tests that required YugabyteDB connectivity.
Phase 2.1: Deal making tests — covering parseFallbackProviders, deal state management, and configuration defaults.
Phase 2.3: Deal repair tests (26 tests) — testing the repair worker's configuration, HTTP client behavior, and error handling.
Phase 4.2: S3 authentication tests (42 tests) — comprehensive AWS SigV4 signature validation, covering valid requests, invalid signatures, malformed headers, access key mismatches, and disabled auth modes. The failing test, TestGC_Disabled, was not part of the new tests. It was an existing test in the rbdeal package that the assistant had not modified. Its sudden failure when run alongside the new configuration or CIDgravity tests was a red flag—it suggested that the new tests might be introducing global state changes that interfere with pre-existing test expectations.

The Diagnostic Method

The assistant's approach in this message exemplifies several best practices for diagnosing test pollution:

1. Reproduce the Failure Consistently

Before the message shown, the assistant had already confirmed that the test passed in isolation (go test ./rbdeal/ -v -run TestGC_Disabled --count=1). The next step was to see whether the failure was specific to a particular ordering or combination of packages. Running all four packages together had produced the failure; running them individually did not.

2. Isolate by Package Boundary

By running each package's test suite individually with --count=1 (which disables test caching and ensures a fresh run), the assistant confirmed that the configuration, cidgravity, and server/s3 packages all pass cleanly on their own. This rules out the possibility that any single package contains a fundamentally broken test. The pollution must arise from an interaction—most likely from global state set up during the initialization of one package's tests that persists into another package's test execution.

3. Use Explicit Separators

The command uses echo "---CONFIG DONE---" between package runs. This is a small but important detail: it ensures that if any package's test output is truncated or mixed with another's, the boundaries remain clear. In a shell pipeline, this makes it easy to see where one package's output ends and another's begins.

4. Document the Hypothesis Explicitly

The assistant states "This might be test pollution between tests" before running the diagnostic command. This is not merely a comment—it is a documented hypothesis that frames the subsequent investigation. By making the hypothesis explicit, the assistant creates a clear chain of reasoning that can be reviewed later.

What the Message Does Not Show

The message ends with the three packages passing individually. The assistant does not yet identify the specific source of pollution. That work happens in subsequent messages (not shown in this segment). However, the groundwork laid here is essential: the assistant has proven that the pollution is cross-package, which narrows the search dramatically. Potential sources of cross-package test pollution in a Go project like this include:

Prometheus metric registration: If tests register metrics with the same name in different packages, the second registration can cause a panic or duplicate registration error. This is a known issue in Go projects using Prometheus, and indeed the assistant later discovered and fixed duplicate Prometheus metrics registration in index_metered.go and gc.go.
Environment variables: If one test sets an environment variable that another test reads, the order of test execution can produce different results.
Global singletons: Database connections, HTTP clients, or configuration objects stored in global variables can be modified by one test and read by another.
File system state: Tests that create temporary files or directories without proper cleanup can affect subsequent tests.
Logging or metrics exporters: Global loggers or metrics exporters initialized in one package's TestMain can affect the output or behavior of another package's tests.

Assumptions and Input Knowledge

To fully understand this message, the reader needs several pieces of input knowledge:

Go testing semantics: The reader must understand that go test runs all tests in the specified packages within the same process (unless parallel execution is explicitly configured). This means global state is shared across packages.
The --count=1 flag: This Go test flag disables test result caching, ensuring that each run is a fresh execution. Without this, Go might return cached results from a previous run, masking the pollution.
The concept of test pollution: The reader must recognize that tests can interfere with each other through shared mutable state, and that a test passing in isolation but failing in a group is a telltale sign of such interference.
The project architecture: The configuration, cidgravity, rbdeal, and server/s3 packages are separate Go packages within the same module. They share access to the same global state (Prometheus default registry, environment variables, etc.) even though they are logically separate.

Output Knowledge Created

This message produces several important outputs:

Confirmed diagnostic evidence: The assistant now has concrete proof that the failure is cross-package, not intra-package. This eliminates the need to debug individual test logic within rbdeal.
A documented hypothesis: The phrase "test pollution between tests" becomes the guiding theory for subsequent investigation. Future debugging efforts will focus on finding shared mutable state rather than incorrect test assertions.
Reproducible commands: The shell commands shown can be re-run by any developer to verify the behavior. This is a form of executable documentation.
A boundary for the search space: The assistant now knows that the pollution involves at least two packages. The next step would be to run pairs of packages together (e.g., configuration + rbdeal, cidgravity + rbdeal) to identify which combination triggers the failure.

The Deeper Lesson: Why Test Pollution Matters

Test pollution is not merely a nuisance—it is a threat to the reliability of the entire test suite. When tests pass or fail based on execution order rather than actual correctness, developers lose trust in the test results. A flaky test that fails intermittently due to pollution can mask real bugs or, conversely, cause developers to waste time investigating phantom failures.

In distributed storage systems like the Filecoin Gateway project, where correctness is paramount, test reliability is especially critical. The system being tested manages deal making with storage providers, S3-compatible object storage, garbage collection, and repair workers—all operations that, if incorrect, could result in data loss or financial penalties. A test suite that cannot be trusted is worse than no test suite at all.

The assistant's careful diagnostic work in this message demonstrates a commitment to test suite integrity. Rather than dismissing the failure as a fluke or blindly re-running tests until they pass, the assistant pauses to understand the root cause. This is the mindset of an engineer who values reliability over speed.

Conclusion

The message at index 2440 is a small but revealing window into the practice of disciplined debugging. In just a few lines of shell commands and a concise hypothesis, the assistant transforms a confusing test failure into a well-defined diagnostic problem. The message teaches us that when tests behave inconsistently, the first step is not to change the code but to understand the conditions under which the failure occurs. By isolating the failure to cross-package interactions, the assistant sets the stage for identifying and fixing the root cause—in this case, duplicate Prometheus metric registrations that would later be resolved.

This message is a testament to the value of methodical thinking in software engineering. It reminds us that the most powerful debugging tool is not a debugger or a log statement, but a clear hypothesis and the discipline to test it systematically.