Small Sample Failure of Random Assignment: A Further Examination

The purpose of random assignment is to produce equivalence on nuisance variables across experimental groups, even when the nuisance variables are unmeasured or of uncertain number. The effectiveness of random assignment, however, depends on sample size; as sample size increases, the likelihood of equivalence also increases. This article demonstrates that, although nonequivalence on a nuisance variable may be quite likely for small samples, the probability is quite low that nonequivalence will produce erroneous inferences. In fact, the probability of an erroneous inference in the absence of a true treatment effect is generally no greater than the nominal Type I error rate. Accordingly, it is unlikely that small samples have biased inferences drawn from past psychotherapy outcome research. However, small samples cause other problems that argue against their routine use.