Statistics and Data Science Seminar
Prof. Bruce D. Spencer
Northwestern University
Estimating the Accuracy of Verdicts in Criminal Trials when Truth Is Unknown
Abstract: Criminal trials may be viewed as complex classification procedures where the verdict represents classification as guilty or not guilty. Assessing the accuracy of verdicts is difficult because the "true" state of the defendant typically is unknown, and those cases where it is known are atypical. Yet, average accuracy of verdicts in criminal cases can be studied systematically and empirically provided we can obtain a second (or even a third) rating of the verdict. For example, in a jury trial the judge can also be asked for a verdict, as in the National Center for State Courts (NCSC) study of criminal cases from four jurisdictions in 2000-01. That study, like the famous Kalven-Zeisel study of the 1950s, showed only modest agreement between the judge and jury. Estimates of overall accuracy of verdicts are easily developed from the judge-jury agreement rate, and under plausible conditions the estimates of accuracy are optimistic. Estimates of false conviction rates and false acquittal rates, are more challenging, and are developed for the NCSC data with the use of log-linear latent class models. Those models, as well as models based on more than two raters, depend on stronger assumptions than the estimates of overall accuracy based on agreement rates. Numerical estimates of verdict accuracy are presented for the NCSC data and sources of uncertainty in the estimates are discussed, with particular attention to the effect of invalidity of the latent class. The estimates of the false conviction rates and false acquittal rates lead to questions about the appropriate balance of errors. Limitations of statistical decision theory for finding an optimal balance will be discussed.
Wednesday April 28, 2010 at 3:00 PM in SEO 636