Measuring legal change: The reliability and validity of Shepard's Citations

With few exceptions, scholars have generally relied on judges' final votes on the merits as the primary indicator of judicial outcomes. Yet, to fully understand judicial decision-making we think it imperative that research also focus on the interpretation of precedent and legal change. To do so, it is necessary to develop measures of legal change and the treatment of precedent over time. Scholars have begun doing so by using Shepard's Citations, a legal citation index. One of the most important features of Shepard's is its list of all opinions that legally treat a previously decided case, as well as its characterization of the nature of that legal treatment. Yet, the reliability and validity of Shepard's is unknown, and we should therefore be appropriately skeptical of it. This article empirically tests the reliability of Shepard's and discusses the validity of its coding protocols. Our analysis demonstrates that Shepard's coding of legal treatment is quite reliable, though there is some notable variance across Shepard's treatment categories. We also point out several features of Shepard's that could potentially affect the validity of a measure derived from it. We conclude that, as long as scholars keep these validity issues in mind, Shepard's can be a highly appropriate data source.