Statistical Significance, False Positives, and Flexibility in Data Analysis (Researcher Degrees of Freedom)
September 6 @ 1:00 pm - 4:00 pm
This workshop is a discussion of the important article “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant” by Simmons, Nelson and Simonsohn (2012), which discusses how seemingly benign decisions during data collection and analysis, or researcher degrees of freedom, make false positives much more likely. Their simulations show how a combination of a few of these decisions can increase the false-positive rate to above 50%. When researcher degrees of freedom are not disclosed in the reporting of the study, reported p-values are not reflective of the flexibility required to achieve those results and will tend to be overconfident. Solutions for both researchers and journal reviewers will be proposed.