In this article, I present a spreadsheet that demonstrates how researcher degrees of freedom (RDoF) increase type 1 errors in scientific research. RDoF refer to the flexibility in analyzing and reporting data. The overarching goal is to instill good research practices in students through awareness of the problems associated with RDoF and the mechanisms through which specific types of RDoF increase type 1 error rates. To accomplish this goal, the spreadsheet is organized into four modules. The first three modules use Monte Carlo simulations to demonstrate common examples of RDoF—dichotomization, optional stopping and multiple testing. The modules allow students to manipulate factors that control the type 1 error rate. The fourth module demonstrates how multiple types of RDoF combine in practice to produce high type 1 error rates. The article concludes with a set of pedagogic questions that instructors may use to teach core concepts associated with RDoF.
Fisher, Christopher R.
A Pedagogic Exploration of Researcher Degrees of Freedom,
Spreadsheets in Education (eJSiE):
1, Article 1.
Available at: https://epublications.bond.edu.au/ejsie/vol7/iss1/1