Interrater Reliability of the Nancy Histologic Index in Assessing Histologic Remission in Treated Ulcerative Colitis Biopsies: A Multi-institutional Experience Among Gastrointestinal Pathologists in the United States

Background and objectives: Histologic remission is recommended as an adjunctive treatment target in ulcerative colitis, and scoring systems have been proposed to enhance reproducibility. The Nancy Histologic Index (NHI) is increasingly used in clinical trials; however, its performance in real-world settings is not fully established. This study aimed to assess the interrater reliability (IRR) of the NHI among gastrointestinal pathologists in the United States. Methods: Thirty-seven whole-slide images of colorectal biopsies from 34 treated ulcerative colitis patients enrolled in a multicenter adult cohort were independently reviewed by 12 gastrointestinal pathologists. Each biopsy was reviewed twice, five months apart, and graded using the NHI. Prior to the second review, pathologists completed an online tutorial on the NHI. Results: The NHI showed substantial IRR in both reviews [intraclass correlation coefficient (ICC) = 0.79; 95% confidence interval (CI), 0.70–0.87 at Review 1; ICC = 0.78; 95% CI, 0.69–0.86 at Review 2]. However, considerable variability was observed in individual grade assignments, with the lowest IRR for Grade 2 (ICC = 0.24; 95% CI, 0.15–0.37; P < 0.001, and ICC = 0.23; 95% CI, 0.14–0.36; P < 0.001 for Reviews 1 and 2, respectively), followed by Grade 4 (ICC = 0.41; 95% CI, 0.29–0.55; P < 0.001, and ICC = 0.47; 95% CI, 0.35–0.61; P < 0.001). Grade 1 showed the highest IRR (ICC = 0.79; 95% CI, 0.70–0.87; P < 0.001, and ICC = 0.78; 95% CI, 0.69–0.86; P < 0.001). When Grades 2, 3, and 4 (i.e., active disease) were grouped together, the IRR remained substantial across both reviews (ICC = 0.76; 95% CI, 0.66–0.85; P < 0.001). Conclusions: While the substantial IRR for active disease (Grades ≥ 2) in this study underscores the clinical utility of the NHI, refinement of criteria for Grades 2, 3, and 4 will be crucial in reducing variability among observers and enabling more accurate monitoring of treatment endpoints.