Persistent experimenters, stopping rules, and statistical inference

This paper considers a key point of contention between classical and Bayesian statistics that is brought to the fore when examining so-called 'persistent experimenters'-the issue of stopping rules, or more accurately, outcome spaces, and their influence on statistical analysis. First, a working definition of classical and Bayesian statistical tests is given, which makes clear that (1) once an experimental outcome is recorded, other possible outcomes matter only for classical inference, and (2) full outcome spaces are nevertheless relevant to both the classical and Bayesian approaches, when it comes to planning/choosing a test. The latter point is shown to have important repercussions. Here we argue that it undermines what Bayesians may admit to be a compelling argument against their approach-the Bayesian indifference to persistent experimenters and their optional stopping rules. We acknowledge the prima facie appeal of the pro-classical 'optional stopping intuition', even for those who ordinarily have Bayesian sympathies. The final section of the paper, however, provides three error theories that may assist a Bayesian in explaining away the apparent anomaly in their reasoning.