TY - JOUR
T1 - The effect of study design on gain in evaluations of new treatments in medicine and surgery
AU - Colditz, Graham A.
AU - Miller, James N.
AU - Mosteller, Frederick
PY - 1988/7
Y1 - 1988/7
N2 - We analyzed the results of 128 comparisons of an innovation with a standard treatment in medicine, and 221 comparisons in surgery, to relate features of study design to the magnitude of gain. The mean gain (measured by the Mann- Whitney statistic) for the innovation over standard therapy was relatively constant across study designs, except for nonrandom trials with sequential assignment to therapy. These trials showed a significantly higher likelihood that a patient would do better on the innovation than on standard therapy. In medical trials that used sequential assignment of refractory patients, the mean gain (measured as the Mann — Whitney statistic) was 0.94, compared to the mean gain for randomized controlled trials of 0.62 (p <. 01). In surgery, the mean Mann- Whitney statistic for nonrandom sequential studies evaluating primary treatments was 0.78, compared to 0.56 for randomized controlled trials (p<. 01). In the evaluation of both medical and surgical therapies, randomized control trials that used a placebo control were significantly more likely to produce a gain for the innovation. In medicine, the Mann- Whitney statistic was 0.72 for placebo controlled trials and 0.61 for nonplacebo controlled trials (p=0.04). In surgery, the average Mann-Whitney statistic was 0.60 for 12 double-blind trials that used placebos, and 0.52 for 10 double-blind trials that did not. When interpreting an evaluation of a new therapy, we may consider adjusting the results of studies that have used sequential assignment so that the average bias, as reported in this article, may be taken into account. Likewise, in studies that have used a placebo control, the Mann- Whitney statistic could also be adjusted downward by.10 if a standard treatment is available. These adjustments provide a more realistic appraisal of the new treatment until stronger studies supersede them.
AB - We analyzed the results of 128 comparisons of an innovation with a standard treatment in medicine, and 221 comparisons in surgery, to relate features of study design to the magnitude of gain. The mean gain (measured by the Mann- Whitney statistic) for the innovation over standard therapy was relatively constant across study designs, except for nonrandom trials with sequential assignment to therapy. These trials showed a significantly higher likelihood that a patient would do better on the innovation than on standard therapy. In medical trials that used sequential assignment of refractory patients, the mean gain (measured as the Mann — Whitney statistic) was 0.94, compared to the mean gain for randomized controlled trials of 0.62 (p <. 01). In surgery, the mean Mann- Whitney statistic for nonrandom sequential studies evaluating primary treatments was 0.78, compared to 0.56 for randomized controlled trials (p<. 01). In the evaluation of both medical and surgical therapies, randomized control trials that used a placebo control were significantly more likely to produce a gain for the innovation. In medicine, the Mann- Whitney statistic was 0.72 for placebo controlled trials and 0.61 for nonplacebo controlled trials (p=0.04). In surgery, the average Mann-Whitney statistic was 0.60 for 12 double-blind trials that used placebos, and 0.52 for 10 double-blind trials that did not. When interpreting an evaluation of a new therapy, we may consider adjusting the results of studies that have used sequential assignment so that the average bias, as reported in this article, may be taken into account. Likewise, in studies that have used a placebo control, the Mann- Whitney statistic could also be adjusted downward by.10 if a standard treatment is available. These adjustments provide a more realistic appraisal of the new treatment until stronger studies supersede them.
KW - Bias
KW - Blinding
KW - Clinical studies
KW - Comparative studies
KW - Meta-analysis
KW - Observational studies
KW - Placebo control
KW - Randomized controlled trials
KW - Research synthesis
UR - http://www.scopus.com/inward/record.url?scp=84973703136&partnerID=8YFLogxK
U2 - 10.1177/009286158802200307
DO - 10.1177/009286158802200307
M3 - Article
AN - SCOPUS:84973703136
SN - 2168-4790
VL - 22
SP - 343
EP - 352
JO - Therapeutic Innovation & Regulatory Science
JF - Therapeutic Innovation & Regulatory Science
IS - 3
ER -