At BELL, we recognize that our ability to measure performance has a direct impact on our scholars, our educators, our programs, and ultimately, our organizational sustainability. The ability to illustrate how and why programs work is a prerequisite to achievement of any nonprofit’s mission. But rarely is measurement of social impact a perfect science.
In BELL’s history, we have undertaken two random assignment studies, considered the “gold standard” of assessment. These studies, one of which is ongoing, were designed to provide third-party, independent proof of BELL’s impact. Unfortunately, all evaluative measures, particularly these types of gold-standard studies, require a good amount of gold. As shown in the CEP study, Room for Improvement, it is difficult for nonprofits to raise operating funds to deliver programs, and even more challenging to generate additional funds to assess impact. Throw in the cost of a multi-site random assignment study by a well-respected evaluator, and it’s not surprising that most nonprofits are unable to generate the types of rigorous evidence that foundations seek.
Through BELL’s current study, we are incredibly fortunate to have the support of a dedicated group of funders, including the Social Innovation Fund, the Edna McConnell Clark Foundation, The Wallace Foundation, and The JPB Foundation, who serve as both financial and strategic partners in assessing the impact of our work. While the findings of the CEP study do point to a need for more foundation support of assessment activities, cost is only one hurdle that nonprofits face, particularly in the case of rigorous, independent research. Additional hurdles include:
Capacity: BELL is fortunate to have infrastructure and personnel to support rigorous assessment. Thanks to philanthropic support, we have invested in a full-time Manager of Evaluation who has significant experience in educational research, and have built the technological and data-collection capacity to support our assessment and evaluation needs.
Yet rigorous, independent studies have a way of stretching an organization’s capacity in every way imaginable. The (not so) simple act of designing our current study required a full year of planning, not to mention the time dedicated to developing partnerships, arranging logistics, recruiting participants, and supporting the day-to-day execution of the study. Many nonprofits lack the built-in capacity to properly support such research.
As nonprofits begin to think about undertaking these types of studies, they should work with their foundation partners to conduct a needs assessment and determine what built-in capacity is available.
Required Partnerships: There can often be an assumption on the part of foundations that nonprofits can find willing partners who are enthusiastic to support our evaluation efforts.
In BELL’s case, that assumption states that superintendents and principals are ready and willing to join a rigorous study during the summer months. However, oftentimes community partners can have different, and sometimes competing, priorities when it comes to assessment. In the case of our current study, schools and districts needed to be willing to actively support our research measures through a range of activities, from setting enrollment eligibility to facilitating parent outreach to coordinating our assessment with their own (often mandated) assessments.
When designing any evaluation measure, nonprofits should remember that finding willing partners can have significant implications for success.
Human Element: Random assignment studies can take a toll on participants. In BELL’s case, this includes our scholars and parents. Through implementation of a random-assignment study, students who need support are sometimes unable to obtain it, as they land in the control group. This can be a painful consequence for families, who are forced to wait until the last possible moment to be told whether they will receive services (treatment group) or if they will become a member of the control group, and thus not receive services.
A small consolation for us at BELL was that we were able to reach a point of oversubscription before moving forward with our study. This is not always the case with these types of studies; where participants can be turned away in an effort to fill control groups.
Despite the value that data obtained through rigorous studies can bring to the education field, we as nonprofit and foundation leaders must remember not to lose sight of the human element involved in ‘testing’ those we serve. It can lead to disappointment for some, and that disappointment can be blamed on the nonprofit as the source of both the opportunity found and opportunity lost.
Receptiveness: Throughout the process of designing a random assignment study we have learned that it is not just the cost of funding the actual study that is important. At the end of the day, we need to be able to do something with the data, beyond utilizing it as a tool to grow our revenue base.
These types of evaluations are most valuable when we can utilize them to better understand what is working well and what needs improvement within our organization. This knowledge provides a platform for us to innovate upon and increases the effectiveness and impact of our work.
Evaluations are also incredibly valuable when they are used to impact the larger community dialogue. In the case of our current study, the evidence that we hope to achieve—that BELL improves educational outcomes for scholars—needs to find its way into the hands of policy makers, district leaders, and community members throughout the country to impact education practice on a broader scale. The receptivity of the policy community to this type of change can be a hurdle for many nonprofits. Foundations can play a key role by supporting nonprofits to prove that our solutions work and disseminating our results to a larger audience, with the ultimate goal of utilizing results to create public policy changes.
Imperfect Science: Once a nonprofit has accepted the human impact of an evaluation, dealt with the lack of capacity, assuaged its partners, and found ways to cover costs—which always end up higher than planned—there is one more hurdle to consider. For the most part, nonprofits are not being asked by foundations to simply count the number of clients served. We must measure far less tangible outcomes, which are highly influenced by the external landscape.
Think about measuring outcomes for a new pharmaceutical drug: measurement is precise, many trial studies are run, and official approvals are made by the FDA. For the most part, these studies are conclusive—the drug either cured the problem or it did not. In the nonprofit field, our ability to measure performance is often not as precise, particularly when multiple levels of rigor are added. Even with the most well-designed studies, the results can be unsatisfying. In fact, the more rigorous the assessment—the more qualifiers that exist, the more hurdles the treatment/control group have to jump over, the more internal structures that need to be in place—the less likely that results will be as strong as the researchers hope.
Undertaking rigorous assessments can be a risky proposition for a nonprofit, as the data can be more reflective of the constraints of the study itself than of the nonprofit’s actual performance. This can be a dangerous paradox for nonprofits, as they run the risk of appearing less effective than their peers who may pursue less rigorous approaches. Foundations need to remember that measuring social impact is an imperfect science, even when you have the smartest researchers using the best research design possible, and that they should support those organizations that are willing to take risks to achieve the most accurate results.
Before proceeding with any assessment tool, foundations and nonprofits must develop a shared understanding of the hurdles that will arise in measuring performance. Most importantly, they must keep the end goals of the assessment in mind. Assessment for the sake of assessment, or for the sake of justifying philanthropic investments, misses the main point.
We must ensure that evaluation results have an impact on our programs and organizational practices, and that the results are broadly shared to impact standard practice within the field. In our case, we will consider our evaluation a success if it leads to the broader adoption of a comprehensive summer learning model as an important element of school turnaround plans.
Foundations are a powerful force in helping to achieve this goal, as they have the ability to transform nonprofit solutions from good ideas to viable turnaround strategies for communities in need throughout our country. We all must be willing to see research through, to jump over hurdles and pitfalls, and to provide actionable data that makes a difference.
Join the conversation about the findings featured in Room for Improvement: Foundations’ Support of Nonprofit Performance Assessment on Twitter using the hashtag #granteevoice.