TY - JOUR
T1 - Reproducibility of cognitive endpoints in clinical trials
T2 - lessons from neurofibromatosis type 1
AU - the NF Clinical Trials Consortium
AU - Payne, Jonathan M.
AU - Hearps, Stephen J.C.
AU - Walsh, Karin S.
AU - Paltin, Iris
AU - Barton, Belinda
AU - Ullrich, Nicole J.
AU - Haebich, Kristina M.
AU - Coghill, David
AU - Gioia, Gerard A.
AU - Cantor, Alan
AU - Cutter, Gary
AU - Tonsgard, James H.
AU - Viskochil, David
AU - Rey-Casserly, Celiane
AU - Schorry, Elizabeth K.
AU - Ackerson, Joseph D.
AU - Klesse, Laura
AU - Fisher, Michael J.
AU - Gutmann, David H.
AU - Rosser, Tena
AU - Packer, Roger J.
AU - Korf, Bruce
AU - Acosta, Maria T.
AU - North, Kathryn N.
N1 - Funding Information:
This work was supported by the United States Army Medical Research and Materiel Command, Office of the Congressionally Directed Medical Research Programs, Department of Defense Neurofibromatosis Research Program, Grant Number W81XWH‐05‐1‐0615.
Funding Information:
JMP reports grants from the US Department of Defense and is supported by a Murdoch Children’s Research Institute Clinician‐Scientist Fellowship. IP reports grants from the US Department of Defense. DC reports grants and personal fees from Shire; personal fees from Eli Lilly, Medice, Novatis, Oxford University Press, Servier; grants from Vifor. GAG is a coauthor of the Behavior Rating Inventory of Executive Function (BRIEF) measure, published by Psychological Assessment Resources, Inc. (PAR), for which he receives royalties. GC reports personal fees (Data and Safety Monitoring Boards) from AMO Pharmaceuticals, Biolinerx, Horizon Pharmaceuticals, Merck, Merck/Pfizer, Opko Biologics, Neurim, Ophazyme, Sanofi‐Aventis, Reata Pharmaceuticals, Receptos/Celgene, Teva pharmaceuticals, NHLBI (Protocol Review Committee), NICHD (OPRU oversight committee). He also reports personal fees (Consulting or Advisory Boards) from Atara Biotherapeutics, Axon, Biogen, Biotherapeutics, Argenix, Brainstorm Cell Therapeutics, Charleston Labs Inc, Click Therapeutics, Genzyme, Genentech, GW Pharma, Klein‐Buendel Incorporated, Medimmune, Medday, Novartis, Roche, Scifluor, Somahlution, Teva pharmaceuticals, TG Therapeutics, UT Houston. GC is also President of Pythagoras Inc., a private consulting company located in Birmingham AL. CR‐C reports grants from the Children’s Tumor Foundation during the conduct of the study. MJF reports grants from the US Department of Defense. DHG reports a patent Neurofibromatosis Gene with royalties paid, a patent Neurofibromin pathway modulators issued, and a patent Neurofibromin/dopamine signaling as a biomarker for cognitive and behavioral problems in children with neurofibromatosis type 1 pending.
Publisher Copyright:
© 2019 The Authors. Annals of Clinical and Translational Neurology published by Wiley Periodicals, Inc on behalf of American Neurological Association.
PY - 2019/12/1
Y1 - 2019/12/1
N2 - Objective: Rapid developments in understanding the molecular mechanisms underlying cognitive deficits in neurodevelopmental disorders have increased expectations for targeted, mechanism-based treatments. However, translation from preclinical models to human clinical trials has proven challenging. Poor reproducibility of cognitive endpoints may provide one explanation for this finding. We examined the suitability of cognitive outcomes for clinical trials in children with neurofibromatosis type 1 (NF1) by examining test-retest reliability of the measures and the application of data reduction techniques to improve reproducibility. Methods: Data were analyzed from the STARS clinical trial (n = 146), a multi-center double-blind placebo-controlled phase II trial of lovastatin, conducted by the NF Clinical Trials Consortium. Intra-class correlation coefficients were generated between pre- and post-performances (16-week interval) on neuropsychological endpoints in the placebo group to determine test-retest reliabilities. Confirmatory factor analysis was used to reduce data into cognitive domains and account for measurement error. Results: Test-retest reliabilities were highly variable, with most endpoints demonstrating unacceptably low reproducibility. Data reduction confirmed four distinct neuropsychological domains: executive functioning/attention, visuospatial ability, memory, and behavior. Test-retest reliabilities of latent factors improved to acceptable levels for clinical trials. Applicability and utility of our model was demonstrated by homogeneous effect sizes in the reanalyzed efficacy data. Interpretation: These data demonstrate that single observed endpoints are not appropriate to determine efficacy, partly accounting for the poor test-retest reliability of cognitive outcomes in clinical trials in neurodevelopmental disorders. Recommendations to improve reproducibility are outlined to guide future trial design.
AB - Objective: Rapid developments in understanding the molecular mechanisms underlying cognitive deficits in neurodevelopmental disorders have increased expectations for targeted, mechanism-based treatments. However, translation from preclinical models to human clinical trials has proven challenging. Poor reproducibility of cognitive endpoints may provide one explanation for this finding. We examined the suitability of cognitive outcomes for clinical trials in children with neurofibromatosis type 1 (NF1) by examining test-retest reliability of the measures and the application of data reduction techniques to improve reproducibility. Methods: Data were analyzed from the STARS clinical trial (n = 146), a multi-center double-blind placebo-controlled phase II trial of lovastatin, conducted by the NF Clinical Trials Consortium. Intra-class correlation coefficients were generated between pre- and post-performances (16-week interval) on neuropsychological endpoints in the placebo group to determine test-retest reliabilities. Confirmatory factor analysis was used to reduce data into cognitive domains and account for measurement error. Results: Test-retest reliabilities were highly variable, with most endpoints demonstrating unacceptably low reproducibility. Data reduction confirmed four distinct neuropsychological domains: executive functioning/attention, visuospatial ability, memory, and behavior. Test-retest reliabilities of latent factors improved to acceptable levels for clinical trials. Applicability and utility of our model was demonstrated by homogeneous effect sizes in the reanalyzed efficacy data. Interpretation: These data demonstrate that single observed endpoints are not appropriate to determine efficacy, partly accounting for the poor test-retest reliability of cognitive outcomes in clinical trials in neurodevelopmental disorders. Recommendations to improve reproducibility are outlined to guide future trial design.
UR - http://www.scopus.com/inward/record.url?scp=85076084771&partnerID=8YFLogxK
U2 - 10.1002/acn3.50952
DO - 10.1002/acn3.50952
M3 - Article
C2 - 31797581
AN - SCOPUS:85076084771
SN - 2328-9503
VL - 6
SP - 2555
EP - 2565
JO - Annals of Clinical and Translational Neurology
JF - Annals of Clinical and Translational Neurology
IS - 12
ER -