Researchers’ Perceptions of Statistical Significance Contribute to Bias in Health and Exercise Science

Taylor L. Buchanan, Keith R. Lohse

Research output: Contribution to journalArticlepeer-review

5 Scopus citations

Abstract

We surveyed researchers in the health and exercise sciences to explore different areas and magnitudes of bias in researchers’ decision making. Participants were presented with scenarios (testing a central hypothesis with p = .06 or p = .04) in a random order and surveyed about what they would do in each scenario. Participants showed significant bias in deciding to randomize additional participants when p = .06 (24.5%) compared to when p = .04 (6.5%), χ2 = 19.86, odds ratio (OR) = 13.5. Based on this bias to randomize additional participants, we present statistical simulations showing the cumulative negative effect this bias has on the false positive rate for a field. While it is important for researchers to have flexibility in their data-processing decisions, there is a cost/benefit trade-off to experimenter degrees of freedom and it is important that these decisions be unbiased. These data highlight the importance of a priori analysis plans in research.

Original languageEnglish
Pages (from-to)131-139
Number of pages9
JournalMeasurement in Physical Education and Exercise Science
Volume20
Issue number3
DOIs
StatePublished - Jul 2 2016

Keywords

  • bias
  • false positives
  • randomization
  • research methods

Fingerprint

Dive into the research topics of 'Researchers’ Perceptions of Statistical Significance Contribute to Bias in Health and Exercise Science'. Together they form a unique fingerprint.

Cite this