It’s easier to change their environment.
Megan Stevenson documents that randomized controlled trials (RCTs) of crime policy interventions typically fail to show significant effects and those that do often fail to replicate. I find myself largely in agreement with Stevenson’s perspective. I too am not enamored of the “economist as engineer” metaphor. The engineer thinks he can plan other people’s lives, a conceit that is both perilous and offensive. Indeed, there is a long history running from Adam Smith to Friedrich Hayek and beyond that takes as one core lesson of economics the rejection of the engineer’s mindset. Adam Smith, for example, wrote in his 1759 Theory of Moral Sentiments:
The man of system sees the world as a large chess board with pieces which can be moved around at the will of the legislator. The man of humanity and benevolence sees that every chess piece has a principle of motion of its own, altogether different from that which the legislature might choose to impress upon it.
Similarly, Megan Stevenson’s critique of “evidence-based” criminal justice reform offers one broad reason why the engineer often fails:
People have already fought to create the best lives they could for themselves given the circumstances. Any barriers to success that were readily moveable had already been moved—by people themselves and their communities. In econ-speak, people had maximized their utility subject to constraints.
Yet, there is a puzzle. Don’t some programs change constraints? And isn’t Adam Smith’s second book, “The Wealth of Nations,” all about how incentives matter? Amazon and Google hire economist-engineers, literally called mechanism designers, and they pay them handsomely. Would they do so if the engineers failed to increase profits? I think these questions force us to make distinctions and provide a clue as to why many randomized controlled trials fail. Many of the programs that Stevenson discusses try to change preferences rather than incentives or constraints. Consider the list Stevenson offers:
- Counseling/therapy programs
- Criminal legal supervision, including intensive probation
- Scared-straight programs
- Work/job-training programs
- Drug testing, substance abuse counseling and drug court
- Juvenile diversion
- Policing “hot spots”
- Boot camps
Of these, counseling/therapy and scared-straight programs are clearly attempts to change preferences, but preference-changing is also a major component of many of the other programs on the list, including boot camps, juvenile diversion and drug courts. Intensive probation might sound like a change in constraints, but one source explains it as follows:
Intensive Probation Supervision works with supporting agencies to ensure the probationer is not only monitored, but taking positive steps toward rehabilitation. These agencies may include:
- Mental health organizations
- Counseling services
- Social services
- Community treatment programs
- Addiction resources
- Relapse prevention programs
- Educational and vocational training programs
There is some incentive and constraint changing here, especially while the probationer is being monitored (fail a drug test and go back to jail) but there is also a lot of attempted preference-changing! Moreover, to the extent that these programs hope to be effective after supervision ends, it’s almost entirely through preference-changing.
Don’t some programs change constraints? And isn’t Adam Smith’s second book, “The Wealth of Nations,” all about how incentives matter?
One could argue that offering free counseling services, addiction support, educational and vocational training and so forth changes constraints due to the high market cost of these services. This assumes, however, that individuals would opt for such services if they had greater income. Providing free cod liver oil, however, does not lift a constraint for those of us who would never buy it. It’s a common belief that success in counseling and training programs hinges on the individual’s desire to change. Yet, my contention goes further: If an individual wishes to change, these programs aren’t necessary. True, these programs may offer some help for those who wish to change, but the point is made that most of these programs involve a substantial amount of attempted preference-changing, and the constraint-lifting portion is weak even as the programs are expensive.
Some other programs that Stevenson mentions elsewhere are also not predominantly constraint- or incentive-changing. Take, for example, the many papers estimating the effect of imprisonment on the post-release behavior of criminal defendants via the random selection of less and more lenient judges. At first, it may seem absurd to say that imprisonment is not about incentives. Isn’t deterrence the ne plus ultra of incentives? Yes, but the economic theory of deterrence, so-called general deterrence, is rooted in the anticipation of consequences — the odds before the crime. By the sentencing stage, we’re merely observing where the roulette wheel stopped. Criminals factor in the likelihood of capture as just another cost of doing business. Thus, the economic theory of deterrence predicts high rates of recidivism, as the calculus that justified the initial crime remains unchanged after punishment. To be sure, imprisonment might change behavior for all kinds of reasons. Maybe inmates learn that they underestimated the unpleasantness of prison, but perhaps they improve their criminal skills while in prison or join a gang, or perhaps the stain of a criminal record reduces the prospect of legitimate employment. Thus, the study of imprisonment’s effects on criminal defendants is intriguing, but it’s not testing deterrence or incapacitation, on which we have built a body of work with clear predictions.
Indeed, on Stevenson’s list only hot-spot policing is a clear example of changing constraints. It is perhaps not coincidental that hot-spot policing is one of the few interventions that Stevenson acknowledges “leads to a small but statistically significant decrease in reported crime in the areas with increased policing.” While I do not begrudge Stevenson her interpretation, other people shade the total evidence differently. Here, for example, is the Center for Evidence-Based Crime Policy, in my experience a rather tough-minded and empirically rigorous organization not easily swayed by compelling narratives:
As the National Research Council review of police effectiveness noted, “studies that focused police resources on crime hot spots provided the strongest collective evidence of police effectiveness that is now available.” A Campbell systematic review by Braga et al. comes to a similar conclusion; although not every hot spots study has shown statistically significant findings, the vast majority of such studies have (20 of 25 tests from 19 experimental or quasi-experimental evaluations reported noteworthy crime or disorder reductions), suggesting that when police focus in on crime hot spots, they can have a significant beneficial impact on crime in these areas. As Braga concluded, “extant evaluation research seems to provide fairly robust evidence that hot spots policing is an effective crime prevention strategy.”
More generally, the non-RCT, causal-inference evidence on policing is quite strong. Having contributed to the literature, I am biased, but I am nevertheless taken by the fact that, as shown in Table 1, many studies by different authors in different times and places find that more police reduce crime, with similar-sized effects even in different contexts. Effects of this size suggest that the United States is drastically under-policed and that a doubling of police on the street would not be unwarranted. Moreover, this is a case where better methods overturned previous null effects and converged on a finding that fits theory, intuition and the anecdotal evidence coming from “natural experiments” such as police strikes.
The other tried-and-true crime-reducing strategy is incapacitation. There is plenty of evidence that incapacitation works. El Salvador’s massive increase in incarceration since 2021-2022 appears to have dropped the murder rate by 70% or more. Amazingly, in 2006 the Italian government released one-third of its prison population. Crime jumped and within a few years incarceration rates returned to their previous levels.
We don’t have many RCTs that study incapacitation by randomly imprisoning people, but that does not mean we have no RCTs on incapacitation! I am thinking of summer jobs programs for teenagers — a kind of voluntary incapacitation. And what does Stevenson say about these programs?
One of the few interventions in which success has been replicated is city sponsored summer job programs for teens. Numerous RCTs have found that summer employment reduces criminal justice involvement.
As Stevenson implies, most of the effect of these programs seems to come from incapacitation — that is, the effect generally lasts only for the summer, suggesting teens are simply being kept away from trouble rather than undergoing a long-lasting change in preferences which, if it exists, is icing on the cake.
It’s decidedly tough to change preferences and many criminology and social science programs aim to change preferences.
Conclusions
To be clear, I am not arguing that it’s easy to change behavior by changing constraints or fiddling with incentive design — assumptions may be false, theories may be wrong, consequences may be unintended. Nor is it easy to test such theories — file drawers, small sample sizes, motivated reasoning, forking paths and so on apply to all such tests.
Nor am I saying that changing preferences is impossible. Propaganda and marketing change preferences, although in both cases it’s much easier to convince people that a particular brand satisfies preexisting preferences than to create entirely new preferences. I am saying that it’s decidedly tough to change preferences and that many criminology and social science programs aim to change preferences. Given this foundation, it’s hardly surprising that many RCTs fail. The criminological programs that RCTs test are typically inspired more by compelling narratives than solid theory and cognate evidence. Human beings are great storytellers — but most of the stories we tell are false. Acknowledging this could significantly change how we choose which programs to fund and test.