What’s in Your File Drawer? The Case of the Missing Null in Criminology and Criminal Justice

Analysis of scholarship in the physical, biological, and social sciences has discovered that peer-reviewed journals publish a much larger proportion of articles with statistically significant findings compared to articles with null results. Publication bias in criminology and criminal justice (CCJ) has received very little attention, however. The current study is an exploratory analysis of research in leading CCJ journals across 2 years to determine the current state of null findings in contemporary CCJ scholarship. Our findings are consistent with studies in other disciplines; null results are rare in leading CCJ journals. We explore the context of our findings, outline the importance of examining publication bias to improve CCJ research and better inform policy, and discuss the limitations of our approach.
**
Project Hope, a correctional intervention in Hawaii started in 2004 that used the threat of an immediate jail sentence to deter probationers from violating the conditions of their supervision, provides an example of the importance of publishing null findings (Barnes et al., 2020). One study (Hawken & Kleiman, 2009) reporting a statistically significant relationship led to HOPE-like programs spread across 160 jurisdictions in over 30 states (Bartels, 2017; Cullen et al., 2016). In 2012, National Institute of Justice’s CrimeSolutions.gov rated HOPE as “Promising” based on this one study. Later, five of six replication studies using statistical techniques the original evaluation failed to use found largely null effects—or no statistically significant difference in rearrests between those who received HOPE probation and those who did not (Cook, 2016; Nagin, 2016). Had the replication studies that found null effects not been published, academics and policy makers would likely have continued to promote and endorse an intervention that had no impact on recidivism. Meanwhile, resources were expended and hundreds, if not thousands, of persons were potentially put at risk by an ineffective program. In 2020, CrimeSolutions.gov reversed its rating of HOPE to “No Effect.” Determining the prevalence of null findings within the CCJ research literature can provide researchers and policy makers an opportunity to consider whether publication bias might be impacting policy and practice.