Replication Data for: The Credibility Crisis in Science

In The Credibility Crisis in Science, we argue that the most important fraudulent strategy is crucially underappreciated. While data fabrication and manipulation are widely recognized as fraudulent, “tweaks”—the intentional selection of research designs and model specifications based on the results they give—are not. The authors contend that the term “scientific fraud” must include tweaks. Tweakers, like other fraudsters, deceive readers by concealing their manipulation of empirical results and they do so to further their own interests.

We show how easily observational data analyses, experimental designs, and causal models are tweaked in ways that are extremely difficult, often impossible, to detect. As a consequence, the credibility crisis in science is even more severe than both scientists and the public believe.

We argue that conventional strategies to deter, prevent, and detect fraud will not work for tweaks and we put forth two potential solutions: first, a classification system that categorizes data based on its susceptibility to manipulation and the probability of such manipulation being identified, and second, the proposal that journal editors and reviewers, rather than authors, select robustness tests.

Table of Contents:

Part I: Is Science Broken?

1 Scientific Integrity and Its Discontents 2 Fraudsters’ Motives 3 The Tip of the Tip of the Iceberg 4 Causes of Detection

Part II: Researcher Degrees of Freedom

5 Tweaking in Observational Data Analyses 6 Tweaking in Experimental Research Designs 7 Tweaking in Causal Models

Part III: Countering Fraud

8 Deterrence Policies 9 Prevention Plans 10 Detection Strategies 11 Two Steps Forward 12 And One Step Back?