Evidence to Support CED programs: The Growing Talk About RCTs

Community and Economic Development – Blog by UNC School of Government

https://ced.sog.unc.edu

Evidence to Support CED programs: The Growing Talk About RCTs

By Maureen Berner

Published October 19, 2021

Whether you work in community or economic development, there has always been pressure to measure your success. CED professionals are very familiar with having to explain what type of impact they hope to have with their initiatives.  It is even harder to document success. Evidence-based decision-making is front and center in conversations on good governance. But what does this mean in practice? What serves as evidence? In the eyes of a growing group of CED policymakers, the best evidence comes from randomized controlled trials (RCTs).

RCTs are the gold standard in research. But as with any gold standard, it’s very hard to achieve in practice. RCTs are still new to local, state or even federal government. The concept behind RCTs is simple. If you randomly assign potential program participants to one of two groups, where one group actually participates in the program but the other group doesn’t, you should be able to see if the program is effective. For example, if a group of job-seekers is randomly assigned to two groups – one taking part in a job training program and the other group not, you can look at how successful the two groups are in obtaining jobs. If those with the training find betters jobs, or find jobs more easily, then the training could be the reason.

The North Carolina Office of Strategic Partnerships (OSP), part of the North Carolina Office of State Management and Budget, has been moving toward using RTC’s to understand program impact. The NC OSP work was highlighted recently by Pew Charitable Trusts, particularly for developing partnerships with NC local governments and non-profits, colleges and universities, and other experts in their work.

Theoretically, RCTs are simple and elegant. However, real people and real CED programs are not the same as a controlled laboratory experiment. So why do it? The benefit of the strong evidence that can come out of an RTC may be worth the work to try to adapt the simple model to a real setting.

Where Can You Learn More?

One of the best, practical, local-government focused resources on RCTs is from MIT’s Poverty Action Lab North America (J-PAL). Anti-poverty programs can be some of the most difficult to assess for short- or long-term impact. The J-PAL report, Implementing Randomized Evaluations in Government: Lessons from the J-PAL State and Local Innovation Initiative, however, is a great place for local CED professionals to start.

J-PAL sees using RCTs for local government in the same way scientists do.  They look look for opportunities to use RCTs (knowing it is not feasible or practical in every circumstance); try to get the best, most valid, reliable and objective results; and share the knowledge – the actual program results and best practices of how to do RCTs — with other practitioners. They include case studies in Philadelphia, PA, Rochester, NY, and South Carolina.  RCTs are a hot topic in local government — they were also featured earlier this year by Johns Hopkins University’s Bloomberg Cities Network.

The J-PAL report and Bloomberg Cities Network show it can be done. The NC OSP has identified RCTs as a focus for future state evidence-based decision-making efforts. It has also shown how RCTs can be done with limited resources. Partnering with others who are familiar with the RTC approach can increase the capacity of the public sector partners, helping practitioners and researchers at the same time. That’s success.