TY - JOUR
T1 - Statistical Process Control Methods for Expert System Performance Monitoring
AU - Kahn, Michael G.
AU - Bailey, Thomas C.
AU - Steib, Sherry A.
AU - Fraser, Victoria J.
AU - Dunagan, William Claiborne
PY - 1996
Y1 - 1996
N2 - The literature on the performance evaluation of medical expert systems is extensive, yet most of the techniques used in the early stages of system development are inappropriate for deployed expert systems. Because extensive clinical and informatics expertise and resources are required to perform evaluations, efficient yet effective methods of monitoring performance during the long-term maintenance phase of the expert system life cycle must be devised. Statistical process control techniques provide a well-established methodology that can be used to define policies and procedures for continuous, concurrent performance evaluation. Although the field of statistical process control has been developed for monitoring industrial processes, its tools, techniques, and theory are easily transferred to the evaluation of expert systems. Statistical process tools provide convenient visual methods and heuristic guidelines for detecting meaningful changes in expert system performance. The underlying statistical theory provides estimates of the detection capabilities of alternative evaluation strategies. This paper describes a set of statistical process control tools that can be used to monitor the performance of a number of deployed medical expert systems. It describes how p-charts are used in practice to monitor the GermWatcher expert system. The case volume and error rate of GermWatcher are then used to demonstrate how different inspection strategies would perform.
AB - The literature on the performance evaluation of medical expert systems is extensive, yet most of the techniques used in the early stages of system development are inappropriate for deployed expert systems. Because extensive clinical and informatics expertise and resources are required to perform evaluations, efficient yet effective methods of monitoring performance during the long-term maintenance phase of the expert system life cycle must be devised. Statistical process control techniques provide a well-established methodology that can be used to define policies and procedures for continuous, concurrent performance evaluation. Although the field of statistical process control has been developed for monitoring industrial processes, its tools, techniques, and theory are easily transferred to the evaluation of expert systems. Statistical process tools provide convenient visual methods and heuristic guidelines for detecting meaningful changes in expert system performance. The underlying statistical theory provides estimates of the detection capabilities of alternative evaluation strategies. This paper describes a set of statistical process control tools that can be used to monitor the performance of a number of deployed medical expert systems. It describes how p-charts are used in practice to monitor the GermWatcher expert system. The case volume and error rate of GermWatcher are then used to demonstrate how different inspection strategies would perform.
UR - http://www.scopus.com/inward/record.url?scp=57349200746&partnerID=8YFLogxK
U2 - 10.1136/jamia.1996.96413133
DO - 10.1136/jamia.1996.96413133
M3 - Review article
C2 - 8816348
AN - SCOPUS:57349200746
SN - 1080-6040
VL - 3
SP - 258
EP - 269
JO - Emerging Infectious Diseases
JF - Emerging Infectious Diseases
IS - 4
ER -