When fields lack consensus standard methods and accessible ground truths, reproducibility can be more of an ideal than a reality. Such has been the case for functional neuroimaging, where there exists a sprawling space of tools and processing pipelines. We provide a critical evaluation of the impact of differences across five independently developed minimal preprocessing pipelines for functional magnetic resonance imaging. We show that, even when handling identical data, interpipeline agreement was only moderate, critically shedding light on a factor that limits cross-study reproducibility. We show that low interpipeline agreement can go unrecognized until the reliability of the underlying data is high, which is increasingly the case as the field progresses. Crucially we show that, when interpipeline agreement is compromised, so too is the consistency of insights from brain-wide association studies. We highlight the importance of comparing analytic configurations, because both widely discussed and commonly overlooked decisions can lead to marked variation.