The effect of study design on gain in evaluations of new treatments in medicine and surgery

Graham A. Colditz, James N. Miller, Frederick Mosteller

Research output: Contribution to journalArticle

10 Scopus citations

Abstract

We analyzed the results of 128 comparisons of an innovation with a standard treatment in medicine, and 221 comparisons in surgery, to relate features of study design to the magnitude of gain. The mean gain (measured by the Mann- Whitney statistic) for the innovation over standard therapy was relatively constant across study designs, except for nonrandom trials with sequential assignment to therapy. These trials showed a significantly higher likelihood that a patient would do better on the innovation than on standard therapy. In medical trials that used sequential assignment of refractory patients, the mean gain (measured as the Mann — Whitney statistic) was 0.94, compared to the mean gain for randomized controlled trials of 0.62 (p <. 01). In surgery, the mean Mann- Whitney statistic for nonrandom sequential studies evaluating primary treatments was 0.78, compared to 0.56 for randomized controlled trials (p<. 01). In the evaluation of both medical and surgical therapies, randomized control trials that used a placebo control were significantly more likely to produce a gain for the innovation. In medicine, the Mann- Whitney statistic was 0.72 for placebo controlled trials and 0.61 for nonplacebo controlled trials (p=0.04). In surgery, the average Mann-Whitney statistic was 0.60 for 12 double-blind trials that used placebos, and 0.52 for 10 double-blind trials that did not. When interpreting an evaluation of a new therapy, we may consider adjusting the results of studies that have used sequential assignment so that the average bias, as reported in this article, may be taken into account. Likewise, in studies that have used a placebo control, the Mann- Whitney statistic could also be adjusted downward by.10 if a standard treatment is available. These adjustments provide a more realistic appraisal of the new treatment until stronger studies supersede them.

Original languageEnglish
Pages (from-to)343-352
Number of pages10
JournalTherapeutic Innovation & Regulatory Science
Volume22
Issue number3
DOIs
StatePublished - Jul 1988
Externally publishedYes

Keywords

  • Bias
  • Blinding
  • Clinical studies
  • Comparative studies
  • Meta-analysis
  • Observational studies
  • Placebo control
  • Randomized controlled trials
  • Research synthesis

Fingerprint Dive into the research topics of 'The effect of study design on gain in evaluations of new treatments in medicine and surgery'. Together they form a unique fingerprint.

  • Cite this