Impact Over Orthodoxy

Research, policy solutions, and what counts as sufficient evidence in transforming the criminal legal system.
Nicholas Turner President & Director
Dec 06, 2023

At the Vera Institute of Justice, we drive policy change to advance safety and justice and to measurably improve communities and people’s lives. We work to transform the United States’ excessively punitive criminal legal system and to challenge the country’s long-standing complacency with the harms of mass incarceration. This involves piloting solutions in difficult settings—including jails and prisons—and developing evidence to show what does and does not work. Evidence has always been Vera’s cornerstone for our 62 years of existence, and we have a reasoned perspective on it.

Two weeks ago, Jennifer Doleac, an economist and Arnold Ventures’s new executive vice president in charge of its criminal justice portfolio, took to X (formerly Twitter) to openly disparage recent Vera research on college-in-prison programs in New York State. The Vera study she attacked found that incarcerated students who enroll in college are 66 percent less likely to be reconvicted of a new offense following release. Her characterization was surprisingly blithe: “This is research malpractice, y’all.” Rendered so cavalierly by a powerful and putative ally at one of Vera’s closest and deepest philanthropic partners, this mischaracterization necessitates a thoughtful response.

To reach our findings in the New York study, Vera’s researchers selected the best available method for assessing causation—in this case, a design known as propensity score matching. We matched incarcerated students who enrolled in college with incarcerated individuals with similar backgrounds who did not enroll, based on a plethora of attributes that are associated in the literature with selection into education and recidivism (e.g., age at admission to prison, gender, race, prior education history, criminal history, and an array of other characteristics). Those interested in learning more can read the interim study report.

Propensity score matching is a widely-used, accepted, and credible research method in the field. The premise of Doleac’s “malpractice” characterization is that propensity score matching is susceptible to selection bias—in other words, that there is a possibility that people who enroll in prison education have lower rates of conviction for a new offense because of some intrinsic characteristic that is not accounted for in the study design (e.g., “focus” or “motivation to change”). She went on to assert that the field needs to conduct studies that use a handful of narrowly defined methods, including randomized controlled trials (RCTs), considered the “gold standard” of impact evaluations, to gain sufficient insight.

RCTs, a form of research that Vera uses when appropriate for the setting and context, are often painstakingly slow, and, by design, only a small number of subjects can participate in the intervention being tested. Vera does not blindly privilege “gold standard” experimental research for its own sake over that which is sufficiently robust and defensible to advance smart, impactful solutions and policies that improve people’s lives. Vera’s perspective is that when a solid evidence base for positive impact already exists, and when there are 767,000 people in prison who can benefit from a college program, waiting for the results of an RCT is, to use Doleac’s phrasing, malpractice—which has significant and indefensible human costs.

By privileging a rigid research standard and creating divisions over methodological differences, Doleac is giving comfort to the opponents of reform. Policymakers who oppose the mission to end mass incarceration will happily use this as an excuse to continue with the “tough-on-crime” status quo. As it stands, political will is not exactly on our side. In short, Doleac’s perfect—as she defines it—is the enemy of the good, and the sad irony is that, in stark contrast, the status quo we seek to undo has little to no basis in evidence.

The point of responding publicly is not simply to defend Vera’s reputation and research. It is to supply needed perspective. It is to state plainly that people’s lives and liberty, as well as the success of good policy, are at stake. In the case of college-in-prison programs, it is critical that we maintain forward momentum after a historic win in 2020 that repealed a 26-year ban on Pell Grant funding for incarcerated students. We have made—and are making—great strides. Let’s keep our eyes on the prize.

Related