Posts Tagged ‘performance assessment’

The Effect of Response Bias: Who Completes our Surveys?

Monday, September 12th, 2011

The topics about which the Center for Effective Philanthropy (CEP) surveys foundation leaders can be controversial and challenging. As a result, we never know exactly who will respond to a particular survey or why. Yet the issue of who responds and who does not should always be considered by critical readers of research based on survey data.

For example, in 2006, we sent surveys to 163 CEOs of foundations that had used CEP’s Grantee Perception Report, asking about the types of support they provide to grantees, and why: 48 percent responded. We got about the same response rate, 49 percent, when we sent surveys to CEOs of foundations with $100 million or more in assets about their provision of assistance beyond the grant in 2008.

But when we surveyed essentially the same population in 2009 about the concept of strategy and their approaches to it, only 23 percent responded. We knew our survey in this case was long, and the questions were more complex than the 2006 and 2008 surveys.  Perhaps that contributed to the lower response rate. But, another potential explanation is that those who were not interested in strategy, or those who were not yet using strategy, were less likely to respond.

For our recently released research report, The State of Foundation Performance: A Survey of Foundation CEOs, we sent surveys early this year to the 537 CEOs of foundations giving $5 million or more annually in grants. Thirty-two percent responded, for a total of 173 CEOs.

As you can see, response rates of foundation leaders to the surveys we have fielded over the years differ quite a bit. Response rates are one indication of how representative our survey data might be. But whenever we close a survey, and before we analyze the data, a big question for us is: What kind of response bias do we have?

While we can never fully know the answer to that question, we do know a few things about our latest sample. Foundations from which CEOs did and did not respond looked the same in terms of asset size. The giving of these two groups of foundations differed only slightly, with CEOs of foundations giving more being slightly more likely to respond. So we know that, on these basic dimensions, we don’t appear to have a bias issue.

But we did see one important difference: CEOs of foundations that had used CEP’s Grantee Perception Report were quite a bit more likely to have responded to the survey than CEOs of foundations that had not.

When it comes to response bias, a key concern is how the respondents differ from non-respondents on the principal variables the survey seeks to measure. Our main variables in this survey were foundation CEOs’ attitudes and practices regarding foundation performance assessment. It is likely that those less focused on assessment were less likely to respond to the survey. Put another way, perhaps members of the proverbial choir to which CEP preaches – those who are already thinking about, and perhaps working on, assessment – are over-represented among our survey respondents.

Though we have no systematic data with which to test this hypothesis, one email we received while fielding the survey indicates we should at least consider this possibility. The note came from a CEO who told us that she is usually open to doing surveys, but, “I got halfway through and it was seriously raising my anxiety about all the things we aren’t doing to evaluate our work in a systematic way. Then I closed the window and went back to my other work.”

So, we do need to be careful about generalizing too broadly from these results.

But we also believe that responses from 173 CEOs of the largest foundations in the country form a solid sample to give us a sense of CEOs’ attitudes and practices regarding performance assessment.  We also conducted a similar survey almost a decade ago, with a 34 percent response rate – which for that survey represented 77 CEOs – and there’s no reason to believe the response biases would be different this time than last time. That’s why we’re comfortable making some statements about how practices appear to have changed.

But readers of the report, and for that matter any report that’s based on a survey, should always consider the important question of response bias.

My hope is that those who read this latest report take a look at our findings as well as our methodology.

Ellie Buteau, Ph.D., is Vice President – Research at the Center for Effective Philanthropy.

 

 

What Foundations are Doing to Understand their Effectiveness

Wednesday, September 7th, 2011

One of the first research projects undertaken by the Center for Effective Philanthropy (CEP) after it was founded 10 years ago was an effort to understand what foundations were doing to assess how effective they are.

What we learned in 2001-2002 was that foundation leaders were frustrated by how limited their ability was to determine how well the foundations they lead were doing. The data they collected was most likely to speak to the financial performance of the foundation or to the success of individual grants. But it was hard to determine a meaningful answer to the bigger question: how a particular foundation is doing in work designed to make a difference in the world.

CEP’s 10th anniversary created an opportunity to take a fresh look at the current state of performance assessment at large foundations in the United States. What our new research determined is that, in brief, progress is real, but much remains to be done.

As my colleague Phil Buchanan said at our 2011 conference, “… even though we can point to many compelling examples of foundations making an impact, there is far too much ineffective philanthropy–foundations funding programs that don’t work or having the opposite of the intended effect. That’s why assessments are so important.”

Today, CEP is releasing new research about foundation CEOs’ attitudes toward assessment and what the foundations they lead are doing to measure performance. Our  new report, The State of Foundation Performance: A Survey of Foundation CEOs, draws on the responses of 173 CEOs who head foundations making $5 million or more in grants annually. Responses to the survey provide a snapshot of a movement in philanthropy that remains controversial and complex.

The majority of CEOs we surveyed believe that foundations have made great strides in being able to assess their effectiveness. A majority also believes that too few foundations understand their overall performance. Clearly, the issue carries weight. Fully 72 percent of CEOs say assessing their foundation’s performance is a high priority for them.

The search for useful metrics has expanded between our first and second reports. Our data indicate that foundations are using a broader range of information to understand their effectiveness than they were a decade ago.

  • With regard to foundation operations and finances, almost every foundation looks at investment performance and administrative costs. But the kind of information that foundations are least likely to look at is data about people working at the foundation: diversity of staff, staff retention rates and what staff say it is like to work at the foundation.
  • With regard to programmatic work, almost every foundation uses anecdotal feedback, grantee reports, site visits and evaluations. But the information that foundations are least likely to turn to for understanding their program work is feedback from their ultimate beneficiaries – those who are receiving the services or programs that foundations are funding.
  • More than 90 percent of foundation CEOs report they conduct formal evaluations of their work, and almost 50 percent of CEOs combine information across multiple functions to generate a foundation-wide assessment.

These findings seem to support CEOs’ perceptions that there is still a ways to go when it comes to understanding their foundations’ performance, and offer foundations potential paths to follow to get closer to that understanding.

Perhaps one of the most sobering findings from this survey is CEOs’ continued desire for more involvement by foundation boards of directors in the important work of performance assessment – a key aspect of any board’s oversight responsibilities. Over the years this finding has repeatedly arisen in CEP’s research. In this new report, we gleaned more insight into what is getting in the way. The most frequent reason CEOs cited is that the board does not have a deep enough understating of the issue areas in which the foundation funds – a reason that can certainly be addressed by any foundation.

As with all of CEP’s research, we hope this report will spur conversations in the staff meetings and board rooms of those who read it.

Ellie Buteau is Vice-President—Research at the Center for Effective Philanthropy.

The State of Performance Assessment, 10 Years Later

Tuesday, September 6th, 2011

A decade ago, I started my job at CEP and we launched a study on performance assessment at large foundations. It’s not as if we were the first to think the question of whether a foundation was achieving its desired results was important. The earliest major American philanthropists cared deeply about results and some foundations, like Robert Wood Johnson Foundation, had been putting major resources into evaluation for decades.

But what was lacking, at that time, was a look across the large foundations to understand both practices and attitudes when it came to performance assessment.  So we undertook that study with limited resources — $345,000 in grants from the Atlantic Philanthropies, Packard, and Surdna –  and a small staff (three of us worked on the project, and that was CEP’s sole effort in its first year).  We conducted a broad-based survey of CEOs and a series of qualitative interviews.

What we found was clear – and sobering.  CEOs didn’t feel particularly satisfied with the information they could tap to understand how their foundations were doing. They weren’t utilizing a broad set of indicators, tending instead to rely on evaluations that told them more about a particular grant or set of grants than about the overall effectiveness of the foundation they led. And many of them told us they wanted more timely indicators to help guide and inform their work.

A lot has changed in the past decade. The Bill & Melinda Gates Foundation and other new foundations have burst onto the scene. A whole set of infrastructure organizations, from CEP to GEO to Bridgespan to Guidestar, have either been founded or grown substantially. And the media has paid more attention to philanthropy, raising questions about how it operates and what it has achieved.

So we thought it made sense to take stock, again, of the state of performance assessment at large foundations. This time, we surveyed CEOs of foundations with at least $5 million in annual grantmaking.  We’ll release results of that study tomorrow at noon, EST.  Check this blog and our Web site to find out how performance assessment has changed and what the attitudes and practices are of foundation CEOs today.

Avoiding a False Choice (Why We Need Both ‘Morality’ and Assessment)

Thursday, July 21st, 2011

I am speaking today on a panel at the Hudson Institute in Washington, D.C. on “Reclaiming the Moral Life of Philanthropy.”  The panel includes Gara LaMarche, president of Atlantic Philanthropies, and is centered on a talk he gave at MIT in which he suggested he has:

“A disquiet about the way we in the foundation world, along with the organisations we support and the infrastructure many of us have helped to build, have mirrored trends in the political world to talk about what we do and why we are doing it in ways that have strayed too far from first principles. We have become more about the fix, the intervention – to use a horribly dominant word in the field that calls to mind invading armies – than about the reasons for doing or caring about it. In marching under the flag of what works, and in particular what can be proven or demonstrated through the rigours of evidence, we risk straying too far from what is right. I think it is time to strike a better balance.”

I am one of three responding to Gara’s comments – along with Indiana University’s Leslie Lenkowsky, Maya Wiley of the Center for Social Inclusion.  William Schambra will moderate.

Here is what I intend to say (my actual comments may vary a bit of course): 

There is much about Gara LaMarche’s speech at MIT to be admired – and that I agree with.

For example, I agree with him that our political discourse, today, is often focused on poll-driven pragmatism at the expense of moral clarity and moral courage. Whether the issue is climate change or health care or immigration – all examples he cites – we hear too much about what is feasible, or politically possible, and not enough about what, simply, is right. (Although one could easily argue that, given what is going on in Washington right now, a little more focus on what is feasible with respect to the debt ceiling would be helpful! Still, I tend to agree with Gara’s general point here.)

Where I part company with him is when he asserts that this problem infects philanthropy. Or when he suggests that there is some disconnect in philanthropy between the moral case for what we do and the quest to understand what works. Or when he seems to imply that the fact that too many foundations are entrenched in “fixed and safe positions,” which I would agree with, is attributable to a focus on effectiveness – when, in fact, I believe the opposite is the case.

Gara says:

“In marching under the flag of what works, and in particular what can be proven or demonstrated through the rigors of evidence, we risk straying too far from what is right.”

I say: we stray from what is right when we do not assess.  This is, after all, about the people we seek to help – and if we don’t do the necessary work to confirm that we are, in fact, helping, we are falling short of our moral obligation. 

  • As Mario Morino of Venture Philanthropy Partners argues in his important new book, Leap of Reason, “Every ounce of our effort on assessing social outcomes should be with one end in mind: helping nonprofits deliver greater benefits to those they serve.” 
  • It is nothing less than a moral outrage when programs like DARE and Scared Straight receive massive funding only to realize, after years and millions of dollars, that they are having no effect, or, worse, the opposite of the intended effect. Mario’s book has a powerful example of such an instance here in Washington, D.C., at the Latin American Youth Center. That organization sought to change attitudes about domestic violence – but it found its efforts were having the opposite of the intended effect.  Fortunately, the Latin American Youth Center’s metrics allowed it to learn that early, and retool the program.
  • Or consider the durable programmatic infestation of abstinence-only education, propelled by only fervent belief in the face of clear evidence that it fails to protect young people from unintended pregnancy and exposure to disease.

We are all painfully aware that resources are limited. That means ineffective or counterproductive strategies deflect attention and waste time and money urgently needed to support and expand strategies that are effective.

Gara says,

“The effectiveness movement is now finding, I believe, that there is no real constituency for effectiveness as such. … because it is values that move people.”

I say, whatever happened to doing what is right because it is right, rather than worrying about whether an idea has a “constituency?” In fact, isn’t this the very point you led off, with, Gara, in your talk at MIT? 

I would say it is morally right to learn as much as possible about whether what you are doing is having the desired effect, whether it is politically popular or not. 

And let’s take on the task of building the constituency for effectiveness. My experience at the Center for Effective Philanthropy convinces me that there is one.  And I think it is growing.  A survey we conducted this year of foundation CEOs showed that the overwhelming majority say assessment is among their highest priorities. And the constituency for effectiveness has deep roots: it is as old as philanthropy itself. Bill Schambra so often reminds us of this in discussing the “mania to measure” – although to him it is a lament – that goes back to the days of Rockefeller and Carnegie.

The constituency for effectiveness is made up of all those who are not content to assume that they are doing as well as they could be doing – those who are not content to take for granted that they are not unintentionally doing harm.  Those who want to learn and improve. Those who reject ignorance and seek knowledge.  Those, among foundations, who want to break out of the bubble of isolation and positive feedback that affiliation with large endowments inevitably creates, so they can learn how they’re really doing – and how they are really perceived. It is a growing constituency of those who believe effectiveness is a moral imperative.

So here is my central point: Assessment and morality are not in tension, never have been.  On the contrary. We are morally obliged to seek to know how we are doing, and how we can improve.  

And I would argue that the impulse to understand the efficacy of what we are doing stems exactly from our reasons for caring about it.

  • If we really care about improving the lives of former foster kids, than we must track, as the Stuart Foundation in California and its grantees do, the numbers of those with lifelong connections to caring adults and the numbers graduating from college.
  • If we really care about civil rights for gays and lesbians, as the Gill Foundation and its grantees do, than we must track in which states gays and lesbians enjoy the same legal protections – whether in marriage or the workplace – that heterosexuals do.
  • If we really care, as the Robert Wood Johnson Foundation and its grantees do, about tackling the obesity epidemic, than we must track the relevant data – including data on the aspects of the strategies employed that we believe to be crucial, just as that Foundation did in its successful work on tobacco use.
  • If we really care, as the Wilburforce Foundation and its grantees do, about preservation of habitats for wildlife, than we must be data-driven in our tracking of endangered species and seek to strengthen the organizations on the ground doing that hard work; in seeking to strengthen those organizations, the Wilburforce Foundation also seeks to find out whether its help is really deemed to be helpful by its grantees – in part by working with CEP to get candid and comparative data on that question.

These organizations assess because they bring passion to important issues.  Caring and knowledge of efficacy are not in tension, these are not qualities needing to be held in “balance,” as Gara argues.  They form, instead, a virtuous cycle.

Just as we should reject the caricatures and labels – and polarization – that cheapen our political discourse, so should we reject them in philanthropy. 

Invocation of “morality” without regard to effectiveness is often mere ideology. 

We work in the nonprofit sector because of our desire to make communities stronger, lives better, air cleaner, whatever the goal is that you or your organization fight for.  That’s why we do the work.  For that reason, we want to know – most of us deeply, desperately want to know – whether our work is making a difference. 

Of course, it’s hard to know, and some of the attempts to answer those questions are misguided or ill-designed. I would be the first to acknowledge that some assessment is done badly.  And I’d also concede that some make the foolish choice to let ease of measurement drive their selection of goals.

But let’s not let those errors in thinking and approach taint our view of the importance of assessment. 

The fact is, the alternative to measurement and assessment is flying blind.  Doing nothing, or worse than nothing, when we mean to be doing good. 

Failing to know what really helps people. 

Failing to direct resources in ways that result in real improvement in peoples’ lives. 

Failing to be our very best.

Assessment and morality are not in tension, never have been.  On the contrary. We are morally obliged to seek to know how we are doing, and how we can improve.  

Grantee Feedback Fuels Changes at Foundations

Tuesday, May 10th, 2011

Collecting feedback from grantees about a foundation’s performance seems like a good idea, but does it lead to any change?

“Since we began [the Grantee Perception Report®] eight years ago, a big question is whether collecting feedback from grantees that is candid, confidential and comparative in nature leads to any change,” said Center for Effective Philanthropy (CEP) President Phil Buchanan at CEP’s Better Philanthropy: From Data to Impact pre-conference seminar on May 9.

At the meeting, CEP released a new research report that indicates that repeated use of the Grantee Perception Report (GPR) by foundations does appear to be contributing to changes in foundation practices that benefit grantees. The GPR provides provides funders with comparative, candid feedback based on grantee perceptions. A CEP analysis of 59 foundations that have undertaken the GPR at least twice showed that on a number of dimensions in the grantee survey there were statistically meaningful improvements for those funders.

Overall, 80 percent of funders repeating the GPR see some level of positive improvement on the impact that grantees perceive them to be having on their organization, and nearly one-third see statistically significant increases, said Ellie Buteau, CEP Vice-President—Research. Only three percent saw their average rating statistically decline.

“The good news is that funders that are committed to getting improved ratings from grantees are seeing improvement,” Buteau said.

The largest improvements were seen on questions such as:

  • Grantees’ perceptions of foundations’ understanding of, and impact on, their organizations;
  • Helpfulness of a funder’s selection process;
  • Impact on and understanding of grantees’ fields.

Repeated users of the GPR saw the least improvement on community-focused measures, such as understanding of grantees’ local communities. CEP found no improvement on making an impact on local communities.

The findings are particularly noteworthy because CEP found that grantees’ perceptions of foundations in general have not improved in the past eight years, a finding that surprised CEP staff.  CEP hypothesized that given the number of “infrastructure” organizations that have sprung up in this field in the past decade that focus on the experiences grantees have with their funders —including CEP—that CEP would see higher ratings of foundations for first-time GPR users in recent years than it did in early days.

“The disappointing news is that we don’t see recent first-time GPR users rating funders any different than first-time GPR users did eight years ago,” Buteau said.

Those findings make the improvements of the 59 foundations that repeated the GPR particularly striking, she said.

Making changes at foundations, particularly after receiving poor ratings, however, isn’t easy. 

Mary Vallier-Kaplan, vice president and chief operating officer of the New Hampshire-based Endowment for Health, said that when the foundation received its first GPR, it ranked in the 25th percentile of funders on several significant indicators.

“It really felt like a punch in the stomach,” she said at the CEP meeting. “We had worked so hard and we thought we had done so well. It took a while not to deny [the findings], question the methodology or say, ‘Well, we’re a new foundation.’”

After Vallier-Kaplan, staff, and board members absorbed the news, the Endowment for Health undertook several steps to improve its ratings, she said.

“We had been so focused on doing a good job and being fair and using evaluation that we had forgot some of the human side of what this business is about,” Vallier-Kaplan said. “Much of what we did to improve was to add the human dimension.”

As a result, when it undertook the GPR again three years later, the Endowment for Health saw greatly improved ratings. On many of the dimensions where the Endowment had lagged behind most other funders, it was now among the leaders.

CEP’s report, Can Feedback Fuel Change at Foundations? An Analysis of the Grantee Perception Report will be released on its website the week of May 16. The report includes:

  • Detailed findings from the CEP research;
  • Stories of the steps that the Endowment for Health and other foundations took to make improvements based on GPR findings;
  • Four common characteristics of foundations that drive real improvement in GPR results.

Susan Parker is owner of Clear Thinking Communications