A CROC stronger than ROC: Measuring, visualizing and optimizing early retrieval

S. Joshua Swamidass, Chloé Agathe Azencott, Kenny Daily, Pierre Baldi

Research output: Contribution to journalArticlepeer-review

77 Scopus citations

Abstract

Motivation: The performance of classifiers is often assessed using Receiver Operating Characteristic ROC [or (AC) accumulation curve or enrichment curve] curves and the corresponding areas under the curves (AUCs). However, in many fundamental problems ranging from information retrieval to drug discovery, only the very top of the ranked list of predictions is of any interest and ROCs and AUCs are not very useful. New metrics, visualizations and optimization tools are needed to address this 'early retrieval' problem. Results: To address the early retrieval problem, we develop the general concentrated ROC (CROC) framework. In this framework, any relevant portion of the ROC (or AC) curve is magnified smoothly by an appropriate continuous transformation of the coordinates with a corresponding magnification factor. Appropriate families of magnification functions confined to the unit square are derived and their properties are analyzed together with the resulting CROC curves. The area under the CROC curve (AUC[CROC]) can be used to assess early retrieval. The general framework is demonstrated on a drug discovery problem and used to discriminate more accurately the early retrieval performance of five different predictors. From this framework, we propose a novel metric and visualization-the CROC(exp), an exponential transform of the ROC curve-as an alternative to other methods. The CROC(exp) provides a principled, flexible and effective way for measuring and visualizing early retrieval performance with excellent statistical power. Corresponding methods for optimizing early retrieval are also described in the Appendix. Availability: Datasets are publicly available. Python code and command-line utilities implementing CROC curves and metrics are available at http://pypi.python.org/pypi/CROC/. Contact: pfbaldi@ics.uci.edu.

Original languageEnglish
Article numberbtq140
Pages (from-to)1348-1356
Number of pages9
JournalBioinformatics
Volume26
Issue number10
DOIs
StatePublished - Apr 7 2010

Fingerprint

Dive into the research topics of 'A CROC stronger than ROC: Measuring, visualizing and optimizing early retrieval'. Together they form a unique fingerprint.

Cite this