Small Sample Failure of Random Assignment

When small samples of participants are randomly assigned to experimental conditions, it is not usual for the resulting groups to be unequal in their distributions of individual differences (e.g., some groups having more men than women). The resulting correlation between individual-difference variables and treatment makes it possible for individual differences to masquerade as treatment effects. The conditions under which erroneous inferences are most likely to occur, however, either are rare (e.g., very high correlations between the confounding variable and the outcome) or involve samples so small that the resulting low power prevents the biased effect from being statistically significant. Furthermore, any bias is random and cancels across studies. When confounding variables are likely, it is better to control them more aggressively (e.g., with matched random assignment) and to include them explicitly in the statistical analyses so their effects can be separated from those due to treatment.