We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers’ decision making. Participants were presented with scenarios (testing a central hypothesis with p = .06 or p = .04) in a random order and surveyed about what they would do in each scenario. Participants showed significant bias in deciding to randomize additional participants when p = .06 (24.5%) compared to when p = .04 (6.5%), χ2 = 19.86, odds ratio (OR) = 13.5. Based on this bias to randomize additional participants, we present statistical simulations showing the cumulative negative effect this bias has on the false positive rate for a field. While it is important for researchers to have flexibility in their data-processing decisions, there is a cost/benefit trade-off to experimenter degrees of freedom and it is important that these decisions be unbiased. These data highlight the importance of a priori analysis plans in research.
|Number of pages||9|
|Journal||Measurement in Physical Education and Exercise Science|
|State||Published - Jul 2 2016|
- false positives
- research methods