How Powerful Is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence

Arguably the most risky kind of evidence is a large effect size gleaned from a statistically underpowered study. When we see findings like this, our reaction should probably be one of “organized skepticism” (Merton, 1973)—where caution is exercised and a call for replication is made before we get too excited about the single study (see also Kulig, Pratt, and Cullen, 2017; Pratt, Turanovic, and Cullen, 2016). But demonstrating such caution is not something humans are either wired for or socialized to do very well (Kahneman, 2011). And as a discipline, criminology has a sketchy track record with demonstrating caution in the wake of a splashy new finding.
Take, for example, the recent treatment—both in policy and empirical circles—of the correctional intervention known as Project HOPE. Nearly a decade ago, a single evaluation (one with a relatively low level of statistical power) of a correctional program in Hawaii found that threatening offenders on probation with a short but immediate stint in jail for even minor technical violations substantially reduced recidivism (Hawken and Kleiman, 2009). On the coattails of this single small study, HOPE-like programs spread rapidly to over a hundred locations in dozens of states, and has even gone international (Bartels, 2017). But then replications of programs based on the HOPE model started to trickle in—studies with the kind of statistical power the original evaluation lacked (see, e.g., Hamilton et al., 2016)—and the results were unequivocal: HOPE programs had no appreciable effect on recidivism (Lattimore et al., 2016; O’Connell, Brent, and Visher, 2016). That thousands of offenders, victims, and community members have been subjected to the consequences of what now appears to be a failed program underscores the importance of exercising caution when low-powered studies reveal startling results (Cullen, Pratt, and Turanovic, 2016). This is the very thing that an evidence-based policymaking philosophy should seek to avoid.