The Future of Effective Government: Use Evidence, Build Evidence, Repeat
At the beginning of President Obama’s first term, Congress created a $650 million grant program for the U.S. Department of Education (DOE) known as Investing in Innovation (I3).1 This program offered an unprecedented opportunity to re-imagine how federal funding competitions could promote the use of existing evidence to improve important student outcomes (including achievement or student growth, high school graduation rates, and college enrollment and completion rates) while simultaneously generating new knowledge about how to move the needle in K-12 education.2 The brief statute stipulated that funds must flow to entities with a track record of closing the achievement gap for disadvantaged students. However, it left the executive branch considerable flexibility to design a program that would advance evidence-based decision-making at multiple government levels.
To design the new program, the Office of Management and Budget worked with two DOE offices that had limited experience collaborating with each other: the program office, which was responsible for administering new grants, and the Institute of Education Sciences (IES), which includes research experts who manage impact evaluations. Together, these teams created a three-tiered program with evidence built into its DNA. Under the tiered structure, the largest grants supported strategies that were backed by the strongest evidence of impact — including from large-sample, multi-site randomized controlled trials (RCTs) — and were considered ready for scale-up. Medium-sized grants were awarded to support validation of promising strategies that were backed by less rigorous evidence from quasi-experimental evaluations, and small grants were awarded to evaluate innovative or untested strategies with a strong theory of change. In every tier, applicants were required to demonstrate how they used relevant research to inform program design and to provide a plan for rigorously evaluating the impact of their interventions.
The I3 program became a blueprint for other tiered-evidence initiatives launched by the Obama administration in education, in workforce training, and through the Social Innovation Fund. It also served as a guide for evidence-based initiatives launched at the state and local levels. Many of these recent initiatives embody the same characteristics as the I3 program: They are co-created by government and researchers; they focus on using and building evidence of effectiveness in real-world settings; and they employ rigorous evaluation to determine a program’s impact.
The growing enthusiasm for data-driven, outcomes-focused public policies is promising. However, to make meaningful progress in addressing society’s biggest challenges — things like joblessness, health disparities, and recidivism — the process for learning about which strategies measurably improve people’s lives must be accelerated. We believe this will occur only if federal, state, and local governments commit to an iterative, evidence-building approach like the one I3 embodies. This must include program designs that are informed by research, ongoing randomized experimentation, and a commitment to drive resources toward solutions that demonstrate results.
This highlights an important, yet often overlooked, feature of evidence-based policymaking: It is a dynamic, long-term pursuit of outcomes that requires sustained focus on using data and evaluation to learn and continuously improve approaches that address important problems. Put another way, employing evidence-based policy goes beyond simply using prior evidence to figure out what to do next. Making informed decisions based on the best evidence available at the time is important, but without a commitment to generating new knowledge, progress will be unacceptably slow and episodic. Instead, when governments try new programs, they should incorporate rigorous evaluation to monitor whether they deliver the intended results. A commitment to doing so is the only way to ensure that good ideas are adopted and scaled and that we can learn from and improve upon less successful efforts.
Unfortunately, I3’s incorporation of evidence building is the exception rather than the rule among government programs. Engaging in evidence-based decision-making will require time, resources, and a culture shift in the public sector, as many real and perceived challenges prevent government officials from routinely implementing this approach. However, there are also a number of solutions that can help governments overcome these obstacles.
One of the most frequently cited challenges is that data can be expensive, messy, and difficult to access. Although collecting new data can certainly be costly, governments already collect a wide range of data, referred to as “administrative data.” These data can be an excellent source of information for performance management, observational research, and impact evaluation.3 And while administrative records can be messy and not always aligned with government’s policy interests, the investment required to clean these data and align collection with outcomes of interest is minimal relative to the total amount spent on the delivery of government programs. It can also pay huge dividends for many years to come. Research-practice partnerships, including the Houston Education Research Consortium, have been successful in combining local, state, and national data and analyzing that data to inform policymakers about important decisions, including whether to continue teacher incentive and supplementary reading programs.4
A second set of challenges stems from the fact that administrative data are often disjointed, housed in multiple agencies, and protected by important privacy laws. Given that individuals often enroll in multiple government programs, data sets must be linked together to allow researchers and policymakers to track outcomes across programs and identify comprehensive solutions to challenging social problems. While collecting and linking data is often time-consuming and generally done on an ad-hoc basis, the development of Integrated Data Systems (IDS) has helped governments overcome these challenges by routinely linking databases, streamlining data access, and protecting privacy.5 The state of South Carolina is leveraging its IDS, which links multiple-state data sets, to conduct a large-scale, low-cost, randomized evaluation of a home visitation program as it is expanded to serve more families across the state. Policymakers will use the evaluation findings to determine the program’s cost-effectiveness and make future programmatic adjustments or improvements.
Governments also face difficulties in identifying evidence-based solutions for the problems they are trying to solve. In many areas of public policy, we simply don’t know much about what works, for whom, and under what circumstances. But this only underscores the need for the approach outlined above. Governments must build evidence rather than simply consume it. Although a small number of interventions have promising prior research evidence, including programs like those found in IES’s What Works Clearinghouse and those funded by I3’s large grants, we don’t know how effective those solutions will be when they are scaled and replicated in different settings. Sometimes the hoped-for effect is achieved, as occurred when KIPP expanded its network of middle schools and improved reading and math achievement among students.6 Other times, however, the result differs from what is expected. Recent findings from a randomized evaluation of Success for All, a promising school-wide reform program for high-poverty schools, showed that the program did not improve reading fluency or comprehension among students.7 These results were considerably weaker than those reported in an earlier randomized evaluation, which found significant positive impacts.8 This example shows that there is no guarantee of success when programs that initially demonstrate promising results are replicated on a larger scale, with new target populations, or in a different context. So although governments should use research and evidence to design programs and policies to ensure the best chance of success, they must also rigorously evaluate those programs in order to build additional evidence and advance our knowledge of “what works” over time.
Yet another challenge is the fact that employing evidence-based policy requires, among other things, technical capacity to collect, store, and manage large amounts of data; the ability to rigorously analyze that data; and the social-science expertise to set up and execute prospective randomized program evaluations. In addition to using existing resources, including administrative data, evidence repositories, and natural opportunities for experimentation (i.e., oversubscribed programs), governments must pursue partnerships with organizations that can provide these essential knowledge-building skills. One of the most promising models is a strategic partnership with a university which can help governments develop robust data systems, embed rigorous evaluation into the policymaking process, and support innovation. Examples include the Government Performance Lab at the Harvard Kennedy School, which deploys teams of highly qualified fellows to state and local government agencies to help them develop results-driven contracting processes and tie spending to outcomes; the Rhode Island Innovative Policy Lab at Brown University, which helps the state design policy experiments; and JPAL North America, which helps governments implement randomized evaluations.
To make real, measurable progress in solving today’s big social problems, we must embrace a culture of innovation and learning. We should be humble about what we know and hungry for more evidence about approaches that will deliver better outcomes for those in need. The pace of learning and discovery dramatically increased when we began to apply scientific method in such fields as medicine and the hard sciences; the social sciences are long overdue to follow suit. By focusing on both the use and generation of evidence, we believe that the public sector is poised to rapidly accelerate the pace at which we learn what solutions work, for whom, and why.
1 U.S. Department of Education, “U.S. Secretary of Education Announces National Competition to Invest in Innovation” (2009), available at https://www.ed.gov/news/press-releases/us-secretary-education-announces-national-competition-invest-innovation.
2 U.S. Department of Education, “Programs, Investing in Innovation Fund i3,” available at https://www2.ed.gov/programs/innovation/index.html?exp=0.
3 J-PAL North America Resources on How to Obtain and Use Nonpublic Administrative Data, “Using Administrative Data for Randomized Evaluations,” available at https://www.povertyactionlab.org/admindata.
4 Dara Shifrer, “Houston Independent School District’s Aspire Program: Estimated Effects of Receiving Financial Awards 2009–10 Aspire Program,” The Houston Education Research Consortium (HERC) Kinder Institute for Urban Research Rice University (2013), available at https://kinder.rice.edu/uploadedFiles/Kinder_Institute_for_Urban_Research/Programs/HERC/2013V1|1.SHIFRER_ASPIRE20092010[3].pdf; Daniel Bowen, “An Evaluation of the Houston Independent School District’s Secondary Reading Initiative: First Year Student Effects,” The Houston Education Research Consortium (HERC) Kinder Institute for Urban Research Rice University (2014), available at https://kinder.rice.edu/uploadedFiles/Kinder_Institute_for_Urban_Research/Programs/HERC/2014V2I2.BOWEN_SRI1[3].pdf.
5 Actionable Intelligence for Social Policy, “Establishing an IDS,” available at http://www.aisp.upenn.edu/integrated-data-systems/establishing-an-ids.
6 Christina Clark Tuttle et al., “Understanding the Effect of KIPP as it Scales: Volume I, Impacts on Achievement and Other Outcomes (Executive Summary),” Mathematica Policy Research (2015), available at https://www.mathematica-mpr.com/our-publications-and-findings/publications/executive-summary-understanding-the-effect-of-kipp-as-it-scales-volume-i-impacts-on-achievement.
7 Janet Quint et al., “Scaling Up the Success for All Model of School Reform: Final Report from the Investing in Innovation (i3) Evaluation,” MDRC (2015), available at http://www.mdrc.org/publication/scaling-success-all-model-school-reform.
8 Geoffrey D. Borman et al., “Final Reading Outcomes of the National Randomized Field Trial of Success for All,” American Educational Research Journal (2007), available at http://aer.sagepub.com/content/44/3/701.