Posts Tagged ‘Grantee Perception Report’

The Unique and Urgent ‘Business’ of Philanthropy

Thursday, December 22nd, 2011

The following blog post is adapted from a talk presented at a meeting of the Grantmakers of Oregon and Southwest Washington.

As Suk Rhee mentioned in her introduction, I am a native Oregonian, so I know that there is no better place than the Pacific Northwest. And I am so happy to be here with you – funders who are committed to making the region even better;

  • More economically prosperous;
  • with better opportunities for all children and stronger schools in which they can learn and grow;
  • with its natural beauty protected from pollution and destruction;
  • with habitats for wildlife preserved;
  • with fewer homeless and abused;
  • with fewer struggling with mental illness and disease;
  • with discrimination – whether based on gender, race, sexual orientation, or disability – stamped out.

These are among the very toughest challenges. Yet, if you pick up the paper or read some of the increasing number of books on philanthropy, you might think it’s not so hard.  Just apply “business thinking” or “market-based solutions,” and we’re all set.

Problems solved.

In their influential 2008 book, Philanthrocapitalism: How the Rich Can Save the World, Matthew Bishop of the Economist and his co-author Michael Green argue that a new breed of “philanthrocapitalists” are working to “apply the secrets behind their money-making success to their giving.” (As if this idea never occurred to Carnegie and Rockefeller.)

They write:

“While some are skeptical about the invasion of the M.B.A.-enabled executives in suits into the Birkenstock world of charity, many philanthrocapitalists believe that the world of giving could benefit at least as much as business from a bigger role for professional intermediaries and advisors, and from the sort of transparency and accountability that exists in financial markets.”

Did I mention that this book came out in the fall of 2008? Where is Lehman Brothers when you need them? Bishop and Green thought they were really on to something, and they’re thoughtful, smart guys who meant well.

But you know charity is working on the toughest problems. If business and government could solve the problems I listed at the start, they wouldn’t exist. You know, because you do this work every day, that no sector owns the concept of effectiveness, and that there is much work in the nonprofit sector that is strategic, data-driven, and rigorous, whether those doing it are wearing Birkenstocks or loafers.

You know, because you are living it, that though people look to you and your resources jealously, they pale in comparison to the challenges you confront.  Here is some context: Annual giving in this country is $300 billion. Total state and federal spending is an estimated $5 trillion.

You know all too well, because you are experiencing it right now that you cannot possibly fill the gaps created by slashed government spending and increased need. But you also know the nonprofit sector has a crucial role to play – one that is separate and distinct from corporations and government.

And, you know, because you live on this earth – and perhaps because, like me, you are a customer of a big national bank (and wondering why), or of the local cable company – that corporations have no greater claim on effectiveness than anyone else.

As Jim Collins, author of Built To Last and Good To Great,  has said, “We must reject the idea – well-intentioned, but dead wrong – that the primary path to greatness in the social sectors is to become ‘more like a business.’”

So let’s cut through the cloud of rhetoric that has descended on philanthropy like ash from Mount St. Helens. Let’s talk about how hard it is to be effective in philanthropy. And let’s talk about what it takes to be effective in philanthropy.

I don’t have the secret formula. No one does.

What I can offer you today may well even sound like what you already know.  (I promise to try not to pretend otherwise!) But I hope that I can at least help you understand more about why doing what we already know we need to be doing is so hard.

So let’s talk about what foundation effectiveness looks like. If you want to create the most positive impact with the resources you have – and I am going to go on the assumption that if you are here, you do – then how do you make that happen?

I think it requires four things:

  • Clear goals
  • Coherent strategies to achieve those goals
  • Disciplined implementation of those strategies
  • Relevant performance indicators to assess progress

Again, I want to repeat that these are not original thoughts nor fancy new ideas – and the challenge is not in understanding that they are necessary for effectiveness, it is that they are so hard to do right.

 

Clear goals

Defining clear, specific goals is tough.  If you’re Howard Schultz, and you’re running Starbucks, the task is clear: make a profit selling coffee.  Execution might be hard, but defining the goal is not.  Sell as much coffee as you can.  Package it up in different sizes and flavors and make people feel they have to have it. You might, I hope, increase the degree of difficulty by saying you want to do it in a responsible way, with good labor practices and sourcing, and so on. But, still, the goal is straight forward.

In foundationland, it’s much tougher. How do you choose goals?

It’s subjective, after all. Values-laden.

Plus, how do you choose just one or two goals or even several, when there are so many pressing social problems, and when those problems are so interrelated?

A wonderful former foundation staff member who I had lunch with a year or so before he died once recounted to me the story of being asked for advice by the new CEO of the foundation where he worked.

“What is the greatest mistake you fear I will make?” the new CEO asked.

The answer my lunch companion gave, at least as he recounted the story, was this:

“The greatest mistake you will make is that you will be drinking your coffee and reading the paper, and you will be deeply moved by something you read. And you’ll walk into the foundation offices and you’ll say, ‘Let’s investigate this issue. We have tremendous resources. There must be something we can do.’ And you will set in motion a process that leads to the creation of a new programmatic area at the foundation.

And, then, six months later, you will be shaving, thinking about another pressing social problem.  And you’ll do the same thing.

And then you’ll look out over the foundation five years from now, or maybe ten, and there will be so many different programs, in pursuit of so many different goals, you will realize that you have squandered the opportunity that your scale offered – to really make a difference on an issue.”

And that is precisely what occurred at this particular foundation. The CEO made exactly the mistake my former colleague predicted he would.

It’s hard not to.

When Jim Collins spoke at our conference in Los Angeles a couple of years ago, he said this:

“Disciplined action begins with piercing clarity about what you choose to not do.  In a world awash with opportunity for contribution, it’s what we choose not to do – because there is so much to do.”

So ask yourself:

  • Are your goals specific, focused, and clear?
  • How many are there?
  • Are they goals that are reasonable for you to believe you can make a difference toward, given your resources?
  • How aligned are your board and staff?
  • If each person associated with the foundation wrote down the goals, would they be the same?

It seems simple.  But all too often, the alignment just isn’t there. Our research shows that a CEO’s perception that there is goal alignment among staff and board is crucial.

So, whether your foundation has one staff member or 30, job one is to get clear on your goals.

Coherent strategies

If clear goals are the “what,” strategy is the “how.”

Everyone loves “strategy.”  Everyone has one.  But do they, really?

First, let me say that strategy plays out differently in philanthropy than in business, because there are no competitive dynamics for private foundations. In business, it’s all about “unique positioning.” But if I can make tremendous impact at my foundation by doing exactly what your foundation is doing, why wouldn’t I?  I wouldn’t demur, by saying, “No, I don’t want to fund those vaccinations because it’s not our ‘distinctive position’ to do that work.”

So how to define strategy in philanthropy?

We at CEP define strategy as:

A framework for decision-making that is

1) focused on the external context in which the foundation works, and

2) includes a hypothesized causal connection between use of foundation resources and goal achievement.

But in our research, we see a disconnect between the rhetorical embrace of strategy and the reality of its actual use. This disconnect exists at both private foundations and community foundations.

Strategy in philanthropy requires a relentless focus on the logic of how you will achieve your goals.  It is about data-driven decision making, rooted in analysis and a theory of how the foundation’s efforts can contribute to the desired change. Wherever possible, it’s informed by evidence of what works and what doesn’t, the more rigorous the better. It’s also influenced by feedback loops so you can constantly iterate and improve your strategy based on a changing context.

Strategy isn’t about deciding what works on high, about being arrogant or top-down. The best strategists are always questioning assumptions – theirs and others – and getting feedback.

Take the example of the Stuart Foundation, with roughly $300 million in assets, making grants in California and the Pacific Northwest.  In their child welfare work, the foundation focused on changing outcomes for former foster kids in California.

The conventional wisdom was that older foster kids couldn’t get adopted:  couldn’t find loving families or people who would assume some responsibility. The folks at Stuart consulted widely with those on the ground, including grantees, government officials, funders and foster kids themselves. They found reason to believe this assumption might be flawed, and tested a strategy that led to older foster kids getting adopted or connected to a caring adult. The data showed that the strategy worked, so they expanded the work.

The Foundation, joined by other foundations, also invested in the kind of supports at state universities that other kids would take for granted as coming from their families. Graduation rates in some cases for kids in these programs were better than the general student population.  The Foundation also recognized that better data systems were needed to track these kids and figure out what is working. So it did something too few foundations do: it invested in a data system.

The Stuart approach became a national model.

That’s good strategy. It’s incredibly difficult, but it can produce remarkable results.

 

Disciplined implementation

But Stuart didn’t just have a good strategy.  It had good implementation.

This is where so much falls apart.

We at CEP see this all the time. Over the past nine years we have surveyed 40,000 grantees of more than 250 foundations, developing a huge comparative dataset that we can mine for our research and for the Grantee Perception Reports we provide to individual foundations, such as Northwest Health Foundation and MJ Murdock Charitable Trust, which have both participated recently.

We see foundations that work very productively with their grantees to achieve shared goals, implementing their strategies well. But we also see foundations that undermine their effectiveness by operating in ways that are not supportive of grantees;

  • they compromise the relationship and then, inevitably, their ability to hear what is really going on;
  • they place requirements on grantees that don’t serve a purpose;
  • they fail to learn from those doing the work on the ground;
  • they fail to communicate clearly about their goals and strategies, and then somehow expect grantees to be able to implement against them.

We see foundations that say they provide capacity building assistance to nonprofits, that that is crucial to their strategy, but they don’t do it in the ways our research shows are required to make a difference.  Rather than doing the kind of comprehensive or field-focused assistance that actually helps grantees, they settle for “drive-by assistance” that may do as much harm as good.

More broadly, foundations frequently don’t commit the resources that would be necessary for implementation of their strategies.  In our research, this emerges as the biggest perceived barrier to strategy implementation.  The most effective foundations recognize that major change requires major resources. They also understand that it almost always requires resources beyond what any one foundation possesses.

Lately there has been a lot of chatter about the concept of collective impact: I got an email promoting a webinar (cost: $49) that read:

“Recently published in the Stanford Social Innovation Review …. Collective Impact is an approach to solving social problems that’s based on the idea that no organization acting alone can solve large-scale issues.”

Wait, didn’t we know this already?

Effective foundations have recognized for decades that achieving significant change requires working together to make changes with other funders and with the grantees on the ground. All the great examples of impact that foundations can lay some real claim to, from the Green Revolution to reducing tobacco use to the end of Don’t Ask, Don’t Tell, involve many institutions, funders, nonprofits, sometimes other actors including companies and government, working together, under the banner of a shared strategy, coordinating implementation.

There is nothing new here.

That’s why CEP and the Monitor Institute are working together to provide a tool that Monitor Institute created, called the Strategy Landscape Tool, to groups of funders who are working toward the same goal.  This tool allows these funders to see who is funding what, by strategy, rather than by the less helpful categories by which grantmaking is typically grouped. We’re providing these for foundations working toward shared goals in specific fields or, sometimes, as for funders in Detroit, for those pursuing improvement in a specific community.

Steve Schroeder, the former CEO of the Robert Wood Johnson Foundation, has argued that “Execution trumps strategy.” I think what he means is that strategy is meaningless if it isn’t well-implemented.

 

Relevant performance indicators to assess progress.

There has been real progress on this front in the past decade.

We see, in a survey we conducted this year of CEOs of foundations that make more than $5 million in grants annually, that assessment of foundation performance is a high priority to CEOs.  They believe much progress has been made in the last decade but that more needs to be done.  We also see that foundations are using a broader range of indicators than they were when we first studied this issue a decade ago.

This is good news.

Still, there is so much confusion about assessment. Lately, there has been a bit of a backlash against experimental design and randomized control trials. But rigorous experimental design is a powerful evaluation approach when applied well. If you are going to put major funding behind a particular strategy then you should know whether it works or not. If it’s new, then support finding out whether it works.

If something has already been shown to work, then don’t assess it all over again. Take the New York State Health Foundation, whose CEO, Jim Knickman, is on CEP’s Board of Directors. The Foundation focuses on improving clinical care and patient outcomes for diabetes – doing so by supporting the proliferation of treatment approaches that have already been proven effective. Rather than funding their own massive evaluation, they have relied on the work of others who have already shown what works – and their assessment now focuses on their ability to spread what works.

So you don’t all have to fund massive evaluative studies, but it’s irresponsible not to pay attention to what is known, and not known, about what works – and to act accordingly.

Another problem is that so much assessment today puts the foundation’s needs in front of the nonprofit grantee’s needs.  But assessment should also support the work of those on the ground, giving both you and them the information to assess and improve.

In his important new book, Leap of Reason, Mario Morino writes:

“I know many nonprofit leaders who are not managing to outcomes but are strongly predisposed to do so.  They inherently know what their outcomes are and very much want to assess and manage to them.  But they are severely hamstrung by the lack of available funding to do this hard work. … At minimum funders should be supporting efforts to help nonprofits to …. (a) track the outcomes of those served; (b) undertake at least basic analysis of this information; and (c) identify how they can use the information to learn and improve their programs over time.”

When it comes to assessment, I think the first step a foundation can take is to open itself up to feedback from the outside,  recognizing that it is surrounded by those who are predisposed to say what they think foundation staff want to hear.  To have any meaning whatsoever, this feedback must be collected by a third-party that is recognized and trusted for its independence so that the feedback is candid; and it must be put in a comparative context.  Without comparative data, it is impossible to make sense of what is a good result, what are relative strengths and weaknesses.

At the Center for Effective Philanthropy, we have worked hard to create these feedback loops.  Our Grantee Perception Report has been used by foundations as big as Gates as well as those with a couple of million dollars in annual giving. Our Applicant Perception Report captures the feedback of those who were declined funding. Our Donor Perception Report gives community foundations the opportunity to understand the views of their donors. Our Stakeholder Assessment Report taps into the perspectives of policy makers and field and community leaders. Our YouthTruth project taps into the voices of those who should matter most – the people whose lives a foundation seeks to improve. YouthTruth does this for education funders through surveys of students in high schools and, soon, middle schools – and its made a powerful difference to schools, districts, and funders.

In every case, the perceptual data is put in a comparative context to make it meaningful.

And we see that foundations are improving, to a statistically meaningful degree, when they repeat tools like the Grantee Perception Report.

As you assess, remember that what you learn often has broader relevance. If you know something about what works, or what doesn’t – share it, so others can learn from it. In my view, it’s morally indefensible not to.  Again, there is no competitive dynamic here – we’re all trying to make a positive difference.

Clear goals. Coherent strategies. Disciplined implementation. Relevant performance indicators.

Not radical concepts.

But incredibly difficult – it is much, much harder to be effective in philanthropy than it is to be successful in business.  Just ask some of the folks in Silicon Valley who have recently made the transition from business to philanthropy.

Not radical concepts – not new concepts either.  In their very good new book, GiveSmart, Tom Tierney and Joel Fleishman imagine Andrew Carnegie and Bill Gates having dinner, writing, “They would quickly discover how much they had in common.”

The authors speculate that as

“rigorous, disciplined, and deeply strategic” men, “the industrial baron and the software tycoon would be highly compatible.”

They write,

“If Gates were to mention ‘strategic philanthropy,’ ‘social entrepreneurs,’ or ‘scaling what works’ in the course of the conversation, Carnegie might not recognize the phrases but he would immediately understand the concepts.”

So the concepts are neither new nor radical, but they’re incredibly hard to act on, each day.

Each one of you here today has a role in making decisions about how resources are used to influence change for the better in this amazing region we call the Pacific Northwest.  How you do that work, each day, the discipline and clarity you bring to the task…it matters.

Foundations play a role other actors in our society cannot, or will not.

When they do it well, the results can be stunning.

So do it well.

 

Phil Buchanan is President of the Center for Effective Philanthropy.

Guest Post: How the Robert Wood Johnson Foundation Seeks To Improve

Monday, October 3rd, 2011
At the Center for Effective Philanthropy, we believe that improved performance of philanthropic funders can have a positive impact on nonprofit organizations and the people and communities they serve. As part of our work, we aim to highlight stories from funders who share that vision and who value the role of data and assessment in efforts to increase their impact.

 

PROCESS PROGRESS: HOW RWJF STRIVES TO IMPROVE
by Robin Mockenhaupt, Ph.D., M.P.H., Chief of Staff,
Dee Colello, Senior Manager, Program Operations, and
David Adler, M.P.A., Communications Officer
of the Robert Wood Johnson Foundation

 

They must often change, who would be constant in happiness or wisdom. ~Confucius

Change and continual improvement is a valued part of the Robert Wood Johnson Foundation’s culture. It’s in our guiding principles, which state, “We must commit ourselves to lifelong learning and continual improvement.” In addition to change and improvement, RWJF is also committed to transparency and peer learning, and it is in that vein that we are sharing our progress on quality improvement since 2004.

As part of our 2004 annual organizational assessment, we commissioned the Center for Effective Philanthropy’s Grantee Perception Report (GPR). We learned that while our grantees rated us comparatively well in several areas, we were using up a lot of grantee time meeting our administrative demands and we weren’t moving as fast as we needed to in processing grants. We also learned we needed better clarity in our communications of goals and strategies.

A 2011 CEP case study, Frequent Checkups Make for Healthier Funding Relationships, illustrated that we changed. We wanted to share one of the ways we took that advice to heart.

With the findings from the GPR, input from an all-staff retreat, and a focus group of grantees, we developed our first Foundation-wide Quality Improvement (QI) initiative. Our process for implementing this QI project can be broken down to five steps.

  • We set the right tone. The all-staff retreat and the announcement of the QI process by our president and CEO, Risa Lavizzo-Mourey, M.D., M.B.A., helped create widespread buy-in among staff. Any organization considering quality improvement projects should recognize that having the buy-in and public support of the CEO is a crucial first step for setting the appropriate environment for change.
  • We started with something manageable. The initial focus of our quality improvement work was on a category of grants that accounts for about 20 percent of our grantmaking. We wanted to start with something manageable to see how it worked before rolling out broader efforts.
  • We gathered the troops. After the announcement, staff interested in quality improvement convened to map out current grantmaking processes; shortly after that, a smaller core staff team was chartered, supported by a group of project sponsors from senior management.
  • We jumped in. The team designed a pilot. After testing and implementation, the project moved into control (maintenance) phase. Our first QI project resulted in sequencing and prioritizing steps in our grantmaking process, as well as launching the Foundation’s Program Information Management System (PIMS).  After our first QI project, two other projects were designed and implemented, using a similar structure and process. In addition, three smaller projects were led by staff trained in the QI process.
  • We are monitoring ongoing progress. We needed a way to monitor how we were doing and for identifying new ideas for improvement. We organized a standing staff group called the Process Improvement Group to help track metrics for our grantmaking and to initiate new quality improvement initiatives. Additionally, other units within RWJF have taken up their own quality improvement initiatives.

What did we learn?

  • Communications is a key component to implementing quality initiatives and staff behavior change.
  • Staff like being involved in cross-functional improvement projects
    when they see the need for change and can be a part of the solution.
    It’s also an opportunity to involve staff at every level of the
    organization.
  • Staff need dedicated time for QI work, as opposed to trying to “fit it in” around other responsibilities.
  • Over time, managers learned better how to scope and implement QI projects.
  • The automation of our grantmaking process (which initially was
    paper) allowed for project milestones and timelines to be standardized
    and to become transparent to all staff.

 

We’re pleased that more recent CEP reports have concluded our grantee perceptions have gotten better over time, and we believe our quality improvement efforts were a factor in this change. Our responsiveness measures are now higher as well as our quality of interactions. With that said, all our quality improvement efforts were not successful and we are receptive to revising any processes that may have missed the mark. For example, we are still working toward reducing the amount of time in both selection process and improving our ongoing monitoring and reporting.

We’re happy to share additional information about our QI process and are eager to hear how our colleagues are approaching this as well.

The Effect of Response Bias: Who Completes our Surveys?

Monday, September 12th, 2011

The topics about which the Center for Effective Philanthropy (CEP) surveys foundation leaders can be controversial and challenging. As a result, we never know exactly who will respond to a particular survey or why. Yet the issue of who responds and who does not should always be considered by critical readers of research based on survey data.

For example, in 2006, we sent surveys to 163 CEOs of foundations that had used CEP’s Grantee Perception Report, asking about the types of support they provide to grantees, and why: 48 percent responded. We got about the same response rate, 49 percent, when we sent surveys to CEOs of foundations with $100 million or more in assets about their provision of assistance beyond the grant in 2008.

But when we surveyed essentially the same population in 2009 about the concept of strategy and their approaches to it, only 23 percent responded. We knew our survey in this case was long, and the questions were more complex than the 2006 and 2008 surveys.  Perhaps that contributed to the lower response rate. But, another potential explanation is that those who were not interested in strategy, or those who were not yet using strategy, were less likely to respond.

For our recently released research report, The State of Foundation Performance: A Survey of Foundation CEOs, we sent surveys early this year to the 537 CEOs of foundations giving $5 million or more annually in grants. Thirty-two percent responded, for a total of 173 CEOs.

As you can see, response rates of foundation leaders to the surveys we have fielded over the years differ quite a bit. Response rates are one indication of how representative our survey data might be. But whenever we close a survey, and before we analyze the data, a big question for us is: What kind of response bias do we have?

While we can never fully know the answer to that question, we do know a few things about our latest sample. Foundations from which CEOs did and did not respond looked the same in terms of asset size. The giving of these two groups of foundations differed only slightly, with CEOs of foundations giving more being slightly more likely to respond. So we know that, on these basic dimensions, we don’t appear to have a bias issue. 

But we did see one important difference: CEOs of foundations that had used CEP’s Grantee Perception Report were quite a bit more likely to have responded to the survey than CEOs of foundations that had not.

When it comes to response bias, a key concern is how the respondents differ from non-respondents on the principal variables the survey seeks to measure. Our main variables in this survey were foundation CEOs’ attitudes and practices regarding foundation performance assessment. It is likely that those less focused on assessment were less likely to respond to the survey. Put another way, perhaps members of the proverbial choir to which CEP preaches – those who are already thinking about, and perhaps working on, assessment – are over-represented among our survey respondents.

Though we have no systematic data with which to test this hypothesis, one email we received while fielding the survey indicates we should at least consider this possibility. The note came from a CEO who told us that she is usually open to doing surveys, but, “I got halfway through and it was seriously raising my anxiety about all the things we aren’t doing to evaluate our work in a systematic way. Then I closed the window and went back to my other work.”

So, we do need to be careful about generalizing too broadly from these results. 

But we also believe that responses from 173 CEOs of the largest foundations in the country form a solid sample to give us a sense of CEOs’ attitudes and practices regarding performance assessment.  We also conducted a similar survey almost a decade ago, with a 34 percent response rate – which for that survey represented 77 CEOs – and there’s no reason to believe the response biases would be different this time than last time. That’s why we’re comfortable making some statements about how practices appear to have changed.

But readers of the report, and for that matter any report that’s based on a survey, should always consider the important question of response bias. 

My hope is that those who read this latest report take a look at our findings as well as our methodology.

Ellie Buteau, Ph.D., is Vice President – Research at the Center for Effective Philanthropy.

 

 

Grantee Feedback Fuels Changes at Foundations

Tuesday, May 10th, 2011

Collecting feedback from grantees about a foundation’s performance seems like a good idea, but does it lead to any change?

“Since we began [the Grantee Perception Report®] eight years ago, a big question is whether collecting feedback from grantees that is candid, confidential and comparative in nature leads to any change,” said Center for Effective Philanthropy (CEP) President Phil Buchanan at CEP’s Better Philanthropy: From Data to Impact pre-conference seminar on May 9.

At the meeting, CEP released a new research report that indicates that repeated use of the Grantee Perception Report (GPR) by foundations does appear to be contributing to changes in foundation practices that benefit grantees. The GPR provides provides funders with comparative, candid feedback based on grantee perceptions. A CEP analysis of 59 foundations that have undertaken the GPR at least twice showed that on a number of dimensions in the grantee survey there were statistically meaningful improvements for those funders.

Overall, 80 percent of funders repeating the GPR see some level of positive improvement on the impact that grantees perceive them to be having on their organization, and nearly one-third see statistically significant increases, said Ellie Buteau, CEP Vice-President—Research. Only three percent saw their average rating statistically decline.

“The good news is that funders that are committed to getting improved ratings from grantees are seeing improvement,” Buteau said.

The largest improvements were seen on questions such as:

  • Grantees’ perceptions of foundations’ understanding of, and impact on, their organizations;
  • Helpfulness of a funder’s selection process;
  • Impact on and understanding of grantees’ fields.

Repeated users of the GPR saw the least improvement on community-focused measures, such as understanding of grantees’ local communities. CEP found no improvement on making an impact on local communities.

The findings are particularly noteworthy because CEP found that grantees’ perceptions of foundations in general have not improved in the past eight years, a finding that surprised CEP staff.  CEP hypothesized that given the number of “infrastructure” organizations that have sprung up in this field in the past decade that focus on the experiences grantees have with their funders —including CEP—that CEP would see higher ratings of foundations for first-time GPR users in recent years than it did in early days.

“The disappointing news is that we don’t see recent first-time GPR users rating funders any different than first-time GPR users did eight years ago,” Buteau said.

Those findings make the improvements of the 59 foundations that repeated the GPR particularly striking, she said.

Making changes at foundations, particularly after receiving poor ratings, however, isn’t easy. 

Mary Vallier-Kaplan, vice president and chief operating officer of the New Hampshire-based Endowment for Health, said that when the foundation received its first GPR, it ranked in the 25th percentile of funders on several significant indicators.

“It really felt like a punch in the stomach,” she said at the CEP meeting. “We had worked so hard and we thought we had done so well. It took a while not to deny [the findings], question the methodology or say, ‘Well, we’re a new foundation.’”

After Vallier-Kaplan, staff, and board members absorbed the news, the Endowment for Health undertook several steps to improve its ratings, she said.

“We had been so focused on doing a good job and being fair and using evaluation that we had forgot some of the human side of what this business is about,” Vallier-Kaplan said. “Much of what we did to improve was to add the human dimension.”

As a result, when it undertook the GPR again three years later, the Endowment for Health saw greatly improved ratings. On many of the dimensions where the Endowment had lagged behind most other funders, it was now among the leaders.

CEP’s report, Can Feedback Fuel Change at Foundations? An Analysis of the Grantee Perception Report will be released on its website the week of May 16. The report includes:

  • Detailed findings from the CEP research;
  • Stories of the steps that the Endowment for Health and other foundations took to make improvements based on GPR findings;
  • Four common characteristics of foundations that drive real improvement in GPR results.

Susan Parker is owner of Clear Thinking Communications

Keepin’ It Real with GPR Results

Wednesday, September 8th, 2010

Before I get started, I’d like to thank the folks at CEP for asking me to contribute to their blog. I’m looking forward to the conversation that will unfold here.

I’m a person who appreciates candor in my relationships. I think it runs in my family. I don’t know too many Ragins who will hesitate to tell you what they think about anything — always with tact and respect, of course. As such, I was excited (and admittedly a bit nervous) when the Hewlett Foundation commissioned CEP to conduct its third Grantee Perception Report (GPR) in 2009. Having previously worked at CEP, it was odd to be on the receiving end of grantee feedback for the first time, but I was anxious to learn what Hewlett’s grantees thought of the performance of the Performing Arts Program, where I serve as associate program officer.

So what did we learn? First, the good news. Compared to other foundations, grantees rated our program quite positively for its impact on and understanding of the field, local community, and organizations that we support. Given that the Performing Arts Program is field-specific as well as regionally focused, we were pleased with this result and felt affirmed in the grantmaking approaches we’d chosen to take. Though we don’t have direct measures of impact (we’re still working on it), grantee perceptions therefore are important performance indicators, as these organizations have on-the-ground knowledge upon which to draw.

However, we received less positive ratings on other important measures, including the clarity of communications of our goals and strategy and the helpfulness of our selection process in strengthening grantee organizations. In response to open-ended questions, grantees expressed a desire to better understand the Program’s strategy and to have it communicated more consistently. They also wanted to know how their work fits into our strategies. With regard to the selection process, grantees told us something we certainly know — relative to other arts funders, our application is long and difficult. We intend the rigorous thinking that is required to complete our application to be of benefit to the organizations, and while many responded that they found the “extra” application elements to be useful, we recognize that improvements could be made.

As a program officer, it’s easy to get trapped in a bubble, deprived of the critical feedback necessary to improve practice over time. This is why GPR results are so helpful. They give us a necessary reality check that we may not otherwise receive.

So what did we do? After a number of internal conversations, as well as some Program-specific follow-up sessions with CEP staff, we made a few changes that we think will help the Performing Arts Program move in the right direction. We now reinforce and communicate to grantees the Program’s goals and strategies as frequently as possible, particularly when having in-person meetings. We placed our Program’s logic model on the Hewlett Foundation website and refer grantees to it, as well as other funders and interested parties. Three times a year, we host a series of grant seeker workshops, during which the Program staff review goals and strategies with grantees, walk through the more challenging application elements, and field questions. We also revised our application, identifying and removing some redundant elements.

I’d be curious to know what other folks have done to improve communications about goals and strategy and to make their selection processes more helpful in strengthening grantees. Any good stories or suggestions? CEP has published a number of reports about ways in which foundations have made improvements in response to grantee feedback. Check them out, if you haven’t already.

Ron Ragin is an associate program officer in the Performing Arts Program at Hewlett Foundation.  The views expressed are his own.