A New Normal for Evidence Building: Five Tips for Funders
Twenty-five years ago, I worked for a large anti-poverty nonprofit in Boston.
We wanted to evaluate our programs — to measure what was working and improve what wasn’t — but persuading foundations to fund this evaluation proved challenging. Some felt that evaluation wouldn’t make a big enough difference, while others felt that the government should support such efforts.
Foundations were reluctant to pay for evidence building. However, when we asked for program funding, they demanded evidence of impact — evidence that we couldn’t build without the funding they were hesitant to provide.
Twenty-five years later, although the ecosystem has grown significantly healthier, this fundamental obstacle remains.
TODAY’S CONTEXT
More and more, philanthropy and impact investors are using data and evidence to inform their funding decisions. Fully 96 percent of nonprofits report that they have at least one donor who requires impact-data reporting. Over 33 percent of foundations have a dedicated evaluation unit.1 But the quality of impact data and evidence varies greatly.
Contrary to conventional wisdom, nonprofits want to build evidence and evaluate not because they want to meet grant requirements (only eight percent of the nonprofits we surveyed indicated this was highly important to them) or to remain accountable to external stakeholders (18 percent), but to improve performance (90 percent) and to verify intended impact (96 percent).2
Yet, despite increased demand from all sides, the dominant approach to evaluation remains slow, expensive, and insufficient. There are many reasons for this.
Evaluations can leave nonprofits no closer to showing evidence of impact or with any more valuable organizational learning than they had in the first place. Premature and underpowered evaluations can leave otherwise effective programs with no evidence of impact. Most evaluations, regardless of findings, leave no roadmap for improvement. “Many evaluators,” as one nonprofit leader we spoke to in 2016 said, “are interested in the science of program evaluation but lack the perspective of providers looking to improve impact. Sometimes they use unrealistic instruments or models.”
Policymakers and funders who demand evidence tend neither to understand nonprofits’ operational and financial realities, nor to use evidence consistently to set policy or award contracts. As a result, the return on evidence-building investments is not always clear to nonprofits or their boards. Funders can shift this dynamic and call for a better balance of evaluation for learning and improvement as well as impact.
Funding for evidence building (and for the infrastructure that allows it) is limited and too often in the hands of third parties. Rather than following consistent business practice, or being empowered to pursue a strategic learning agenda, nonprofits are sometimes forced into ad hoc evaluations, driven by the agendas of others versus longer-range plans guided by emerging evidence of implementation and impact.
In short, funders, evaluators, and nonprofits all have slightly different goals. Funders want proof that their money is making a difference, evaluators want answers to questions of interest to the field, and nonprofits want to improve their programs. As a result of this misalignment, the ecosystem severely limits the pipeline of evidence-based programs ready for investment. Funders seek programs that “work” for investment purposes, when, in fact, they also need to seek opportunities to build the evidence and improve programs’ ability to deliver outcomes to communities.
OPPORTUNITY FOR FUNDERS
Funders are in a unique position to manage conflicting incentives, hold both nonprofits and researchers accountable for learning, improvement, and proof objectives, and make sure that evaluation is in the best interest of everyone involved — nonprofits, evaluators, funders, and, most important, the communities we are all trying to serve.
Below are five ways that funders can use the resources at their disposal to create a new normal for evidence building.
Invite and Support Grantees to Lead
If philanthropy is designed to support grantees in helping their communities, it stands to reason that nonprofits desiring a bigger impact should lead their own evaluation agendas. But this is not how the current ecosystem is set up: Instead of stemming organically from organizational strategy, evidence building is usually conducted to answer questions researchers are interested in or to “prove” to funders that programs are effective. Nonprofits are too often the “caboose” of the evidence-building train; they should be part of the engine.
As an alternative, we encourage funders to support nonprofits in leading the evaluation process. We encourage a new starting point for building evidence that is based on grantees’ learning agenda and priorities. Once this is established, clarity regarding prior research for the specific target population served — questions whose answers will make programs more effective — should be integrated. Finally, by using methodologies appropriate to the questions asked, the available data, and the stage of the organization, we can help nonprofits build the evidence they need to make a bigger difference in their communities. Nonprofits, rather than researchers or funders, should lead this process; it is their responsibility to know whether their program is on track, as well as what’s needed to guide it toward even better performance.
A series of smaller-scale internal and external studies — to inform program implementation and impact over time — allows for a more continuous, more cost-effective, and lower-risk approach. In fact, this is how the much-lauded Nurse-Family Partnership (NFP), which provides maternity and prenatal care to poor mothers, built its evidence base. NFP was grounded in theory and epidemiology, formatively developed with a set of pilot families, and then tested in a sequential series of randomized controlled trials to examine its impact with different populations living in different contexts before it was offered for public investment.3 At every step of this process, data on implementation and outcomes were used to refine the program model and replication process.4
Move From Judgment to Development
“Many funders,” said one nonprofit leader, “want to fund the type of research that would lead to a third-party rigorous evaluation that ‘proves’ program impact. However, just as important is funding the steps leading to this type of rigorous evaluation work.”5
There is no such thing as a one-study wonder that makes a program perfect. The road to higher impact is paved with incremental improvements.
To help grantees follow this road, funders will have to shift from a more judgmental stance to a more developmental one based on learning and organizational development in service of improvement. Our mutual goals call for joint problem-solving, rigorous experimentation, and a willingness to evolve, none of which is possible in a pure thumbs-up/thumbs-down context. By viewing evidence building not as an event but as a cycle — and supporting nonprofits to pursue rigorous, mixed methods that fit their organizational needs — we can improve outcomes not just once but continually.
Don’t Stop at Research: Support Learning and Implementation
Evidence building alone will have limited utility if nonprofits don’t have sufficient capability to interpret the evidence and make adjustments based on what they learn. As part of the move from judgment to development, therefore, funders must go beyond simply supporting research studies to supporting learning, cost-benefit analyses, and implementation.
Funders can also help connect research to policy and practice, as is clear from the success of research-practice partnerships like California’s Fostering Connection to Success legislation, which gave foster care youth in California the right to care until the age of 20. Philanthropy played an important role in this legislation by funding the research, as well as stakeholder engagement to connect these efforts. Legislation like this shows what is possible when multiple stakeholders come together to implement solutions using evidence connected to policy and practice.6
Ask For and Fund Outcomes
Too often foundations and policymakers focus on compliance over impact. This can make both funders and government seem more interested in checking boxes than in making a real difference in nonprofits’ communities. Further, it can breed a thumbs-up/thumbs-down culture, at the expense of learning, which can impede the ability to understand both if and how programs work so we can focus on improvement.
This will change, however, if foundations develop a stronger understanding of different levels of evidence and what it takes to build evidence. While ensuring compliance, it will allow them to send more resources to opportunities that build and sustain stronger outcomes over time. Specifically, funders should make greater use of the following:
- Staged funding to allow for learning, failure, and adjustment over time; continuous improvement requires the resources and space for experimentation.
- Entry-evidence reviews to understand the pre-existing evidence base and best practices for a given program and to identify gaps in critical research.
- The many clearinghouses to understand the evidence for given interventions.
- Data and evidence standards aligned with one another and with federal standards to enable a more coherent and accessible ecosystem.
Embrace Data and Technology
Funders have been reluctant to embrace data and technology. As a result, foundation staff struggle to gain strategic insights from evaluations, while nonprofits struggle to secure evaluations that generate useful insights. Solving these twin problems will require new capacity and investment in tools, technology, and talent, as well as a fresh approach to evaluation design predicated on questions relevant to nonprofits.
Anu Malipatil, education director at Overdeck Family Foundation, said, “As a learning organization, in the spirit of continuous improvement, we knew it was critical that we built strong data systems that would enable us to ask and answer questions about our efficiency and effectiveness on a variety of workstreams, including programmatic grantmaking, and operations. More importantly, we are making it a priority to support our grantees to be learning organizations too. We strive to model what we hope to see in the field more broadly.”7
By building or buying evaluation and learning capacity, as well as data and technology infrastructure, for themselves and their grantees, foundations can accelerate change, reduce costs, and remain helpful to grantees in the emerging ecosystem. This is especially important as developments in big data, predictive analytics, and machine learning (among other developments) shape the world around us.
THE NEXT CHAPTER
Increasingly, nonprofits see evidence building and learning as core to their work, but the funding markets are not keeping pace. As one leader said, “Nonprofits need the runway to solve hard problems. Multi-year support is beneficial, and funding for data and evidence infrastructure would be huge!”8 Funders play a critical role in promoting a culture of learning and evidence building in the social sector — not just for themselves, their evaluators, and their grantees, but also for the communities that will benefit from programs with stronger outcomes.
1 Center for Effective Philanthropy and Center for Evaluation Innovation, “Benchmarking Foundation Evaluation Practices” (2016); Open/Impact, “The Giving Code: Silicon Valley Nonprofits and Philanthropy” (2016); Innovation Network, “The State of Evaluation 2016” (2016).
2 All data are from Project Evident’s fall 2016 market survey of nonprofit organizations; 51 responses were received from 100 surveys; 80 percent of responses were from organizations with revenues greater than $5 million and serving more than 1,000 clients annually; 41 percent of the responders’ organizations focus on education.
3 David L. Olds, “Prenatal and Infancy Home Visiting by Nurses: From Randomized Trials to Community Replication,” Prevention Science 3 (3) (2002): 153–72.
4 David L. Olds, “Improving the Life Chances of Vulnerable Children and Families with Prenatal and Infancy Support of Parents: The Nurse-Family Partnership,” Psychosocial Intervention 21 (2) (2012): 129–43.
5 From Project Evident Planning Interviews.
6 For more details, see Mark E. Courtney and Jennifer E. Mosley, “Partnership and the Politics of Care: Advocates’ Role in Passing and Implementing California’s Law to Extend Foster Care,” Chapin Hall at the University of Chicago (2012).
7 From Project Evident planning interviews.
8 Ibid.