The topics about which the Center for Effective Philanthropy (CEP) surveys foundation leaders can be controversial and challenging. As a result, we never know exactly who will respond to a particular survey or why. Yet the issue of who responds and who does not should always be considered by critical readers of research based on survey data.

For example, in 2006, we sent surveys to 163 CEOs of foundations that had used CEP’s Grantee Perception Report, asking about the types of support they provide to grantees, and why: 48 percent responded. We got about the same response rate, 49 percent, when we sent surveys to CEOs of foundations with $100 million or more in assets about their provision of assistance beyond the grant in 2008.

But when we surveyed essentially the same population in 2009 about the concept of strategy and their approaches to it, only 23 percent responded. We knew our survey in this case was long, and the questions were more complex than the 2006 and 2008 surveys.  Perhaps that contributed to the lower response rate. But, another potential explanation is that those who were not interested in strategy, or those who were not yet using strategy, were less likely to respond.

For our recently released research report, The State of Foundation Performance: A Survey of Foundation CEOs, we sent surveys early this year to the 537 CEOs of foundations giving $5 million or more annually in grants. Thirty-two percent responded, for a total of 173 CEOs.

As you can see, response rates of foundation leaders to the surveys we have fielded over the years differ quite a bit. Response rates are one indication of how representative our survey data might be. But whenever we close a survey, and before we analyze the data, a big question for us is: What kind of response bias do we have?

While we can never fully know the answer to that question, we do know a few things about our latest sample. Foundations from which CEOs did and did not respond looked the same in terms of asset size. The giving of these two groups of foundations differed only slightly, with CEOs of foundations giving more being slightly more likely to respond. So we know that, on these basic dimensions, we don’t appear to have a bias issue.

But we did see one important difference: CEOs of foundations that had used CEP’s Grantee Perception Report were quite a bit more likely to have responded to the survey than CEOs of foundations that had not.

When it comes to response bias, a key concern is how the respondents differ from non-respondents on the principal variables the survey seeks to measure. Our main variables in this survey were foundation CEOs’ attitudes and practices regarding foundation performance assessment. It is likely that those less focused on assessment were less likely to respond to the survey. Put another way, perhaps members of the proverbial choir to which CEP preaches – those who are already thinking about, and perhaps working on, assessment – are over-represented among our survey respondents.

Though we have no systematic data with which to test this hypothesis, one email we received while fielding the survey indicates we should at least consider this possibility. The note came from a CEO who told us that she is usually open to doing surveys, but, “I got halfway through and it was seriously raising my anxiety about all the things we aren’t doing to evaluate our work in a systematic way. Then I closed the window and went back to my other work.”

So, we do need to be careful about generalizing too broadly from these results.

But we also believe that responses from 173 CEOs of the largest foundations in the country form a solid sample to give us a sense of CEOs’ attitudes and practices regarding performance assessment.  We also conducted a similar survey almost a decade ago, with a 34 percent response rate – which for that survey represented 77 CEOs – and there’s no reason to believe the response biases would be different this time than last time. That’s why we’re comfortable making some statements about how practices appear to have changed.

But readers of the report, and for that matter any report that’s based on a survey, should always consider the important question of response bias.

My hope is that those who read this latest report take a look at our findings as well as our methodology.

Ellie Buteau, Ph.D., is Vice President – Research at the Center for Effective Philanthropy.