Posts Tagged ‘foundation performance’

« Older Entries

Making Transparency Matter

Thursday, December 8th, 2011

There has been lots of talk lately about foundations and “transparency.” Perhaps the most prominent foundation transparency initiative is a website called Glass Pockets, launched about a year ago by Foundation Center. Its tagline? “Bringing transparency to the world of philanthropy.”

CEP worked as one of a number of partners to Foundation Center as it designed this effort, and we were pleased and proud to do so. We favor the general idea of foundations being more open about what they do and how they do it, and I agree with Foundation Center President Brad Smith that the record of foundations overall is not so hot when it comes to transparency. So I applaud the effort Foundation Center is making, under Brad’s leadership, to promote transparency.

In fact, as I prepared for the event on “Transparency and Effectiveness” that Foundation Center and CEP co-hosted in San Francisco this week, I started to worry about whether we’re going nearly far enough. I started to worry about whether we are settling for superficial transparency, about whether the inevitable push to create simple check-lists is resulting in good information – or misinformation.

I worry that, in the worst case, we could witness the kind of counterproductive behavior that we see in the world of colleges and universities, where preoccupation with placement on the U.S. News and World Report rankings has led some institutions to take steps to improve their standing – to game the system, really – that are counter to what is best for their students. I worry that we’re not discussing critically the information foundations are being transparent about – and I am not sure there is value in transparency without engagement. And I worry that we’re overemphasizing transparency about trivial matters and under-emphasizing transparency that really facilitates greater effectiveness.

Which brings me back to Glass Pockets. On the Glass Pockets website, 32 foundations to date are listed under the question “who has glass pockets?” because they have agreed to be assessed on whether they disclose information in categories such as “Basic Contact Information,” “Governance Policies & Information,” “HR/Staffing Policies & Information,” “Financial Information,” “Grantmaking Information,” and “Performance Measurement.”

Some don’t do too well – turns out some of their pockets may be more of a tinted glass.

But what about those who do perform well? What about those receiving magnifying glasses or bullhorns on the site (the icons Foundation Center uses to give credit, and which link to the supporting material) in the various categories as indications of their transparency? What does it take, exactly, to get credit in the various areas?

I don’t see the categories as even close to equally important. So I was particularly interested, given CEP’s focus, in foundations that received magnifying glasses on the four dimensions of the “Performance Measurement” category. The specific dimensions in that category are: “assessment of overall foundation performance,” “knowledge center,” “grantee feedback mechanism,” and “grantee surveys.”

Much of what I found was less than inspiring. One foundation receives a magnifying glass for “overall foundation performance assessment” for a consultant’s report based on interviews with leaders in the community – with absolutely no information about methodology or number of interviews conducted. Another foundation links only to the Grantee Perception Report (GPR) provided by CEP. While I think the GPR can be an important component of an overall assessment, should it be enough, on its own, to qualify as an “overall foundation performance assessment?” I am not sure.

One foundation that gets credit on Glass Pockets for conducting a grantee survey does so because it self-administered a survey several years ago and got a 22 percent response rate – and what that corresponds to in terms of number of responses is not disclosed. The questions are poorly constructed, with response option choices that virtually guarantee positive results – and there is no comparative data. Yet among the “findings” are that there are “very favorable views of [the foundation] in all areas” and that the donor “would be proud!” I wonder. (I am not exactly objective, here, I realize. We have our own approach to grantee surveys and a particular point of view about what constitutes a rigorous survey effort.)

Another foundation gets credit for having a “grantee feedback mechanism” simply because it posts contact information for a staff member that grantees can reach out to.

Some of the foundations that do well on Glass Pockets absolutely deserve to. There are funders with impressive supporting materials in the “Performance Measurement” category. But it’s impossible to distinguish between them and others whose commitment seems more cursory based only on what’s listed on Glass Pockets. You have to go to the foundations’ web sites to figure it out, and I am not sure how many of the visitors to the web site will take the time to do that. I think the natural tendency will be to assume that those receiving credit for doing performance measurement, and being public about it, are doing something really meaningful.

So what should be done? A few ideas:

  • I would like to see Glass Pockets add a space for comments on the quality of what foundations have been transparent about. If we don’t engage with what foundations are making public, if it just sits on websites, then how valuable is it?
  • I would like to see more discussions about the substance – both online and face-to-face at meetings such as the one we held in San Francisco. Is transparency a good in its own right? Or is it a means to an end of greater effectiveness? Are transparency and effectiveness ever in tension?
  • I’d like to see Glass Pockets do more to push for openness about programmatic effectiveness – so foundations are encouraged to disseminate information about what works and what doesn’t. That seems far more important to me than criteria such as whether a foundation has a staff list on its Web site, or a newsletter, or a blog.
  • I also wonder about the potential importance of the Charting Impact initiative developed by Independent Sector, BBB Wise Giving Alliance, and Guidestar USA. This effort encourages organizations to publicly answer five “deceptively simple questions,” including “What is your organization aiming to accomplish?, What are your strategies for making this happen?, and How will your organization know if you are making progress?” Only one foundation, the William and Flora Hewlett Foundation, is listed on the IS Web site as having completed a Charting Impact Report. (Full disclosure/hypocrisy check: CEP has not yet done this, although our answers to essentially the same questions can be found in our publicly available strategic plan.) Transparency (and clarity) about the answers to these questions on the part of all major foundations would be a step forward.

I think the Glass Pockets effort is a promising one. Foundation Center deserves a lot of credit for taking a stand and pushing this forward. I also like the direction of the Charting Impact initiative.

We should all work to raise the bar on what constitutes transparency that matters. I hope we can push for much, much higher standards for the depth and quality of information necessary for a foundation to get credit for its commitment to performance measurement, for example – or, for that matter, for its transparency in general.

Foundations have made progress when it comes to transparency. But there is a long, long way to go.

 

Phil Buchanan is President of the Center for Effective Philanthropy.

 

Tags: , ,
Posted in Assessing Performance | 2 Comments »

What Foundations are Doing to Understand their Effectiveness

Wednesday, September 7th, 2011

One of the first research projects undertaken by the Center for Effective Philanthropy (CEP) after it was founded 10 years ago was an effort to understand what foundations were doing to assess how effective they are.

What we learned in 2001-2002 was that foundation leaders were frustrated by how limited their ability was to determine how well the foundations they lead were doing. The data they collected was most likely to speak to the financial performance of the foundation or to the success of individual grants. But it was hard to determine a meaningful answer to the bigger question: how a particular foundation is doing in work designed to make a difference in the world.

CEP’s 10th anniversary created an opportunity to take a fresh look at the current state of performance assessment at large foundations in the United States. What our new research determined is that, in brief, progress is real, but much remains to be done.

As my colleague Phil Buchanan said at our 2011 conference, “… even though we can point to many compelling examples of foundations making an impact, there is far too much ineffective philanthropy–foundations funding programs that don’t work or having the opposite of the intended effect. That’s why assessments are so important.”

Today, CEP is releasing new research about foundation CEOs’ attitudes toward assessment and what the foundations they lead are doing to measure performance. Our  new report, The State of Foundation Performance: A Survey of Foundation CEOs, draws on the responses of 173 CEOs who head foundations making $5 million or more in grants annually. Responses to the survey provide a snapshot of a movement in philanthropy that remains controversial and complex.

The majority of CEOs we surveyed believe that foundations have made great strides in being able to assess their effectiveness. A majority also believes that too few foundations understand their overall performance. Clearly, the issue carries weight. Fully 72 percent of CEOs say assessing their foundation’s performance is a high priority for them.

The search for useful metrics has expanded between our first and second reports. Our data indicate that foundations are using a broader range of information to understand their effectiveness than they were a decade ago.

  • With regard to foundation operations and finances, almost every foundation looks at investment performance and administrative costs. But the kind of information that foundations are least likely to look at is data about people working at the foundation: diversity of staff, staff retention rates and what staff say it is like to work at the foundation.
  • With regard to programmatic work, almost every foundation uses anecdotal feedback, grantee reports, site visits and evaluations. But the information that foundations are least likely to turn to for understanding their program work is feedback from their ultimate beneficiaries – those who are receiving the services or programs that foundations are funding.
  • More than 90 percent of foundation CEOs report they conduct formal evaluations of their work, and almost 50 percent of CEOs combine information across multiple functions to generate a foundation-wide assessment.

These findings seem to support CEOs’ perceptions that there is still a ways to go when it comes to understanding their foundations’ performance, and offer foundations potential paths to follow to get closer to that understanding.

Perhaps one of the most sobering findings from this survey is CEOs’ continued desire for more involvement by foundation boards of directors in the important work of performance assessment – a key aspect of any board’s oversight responsibilities. Over the years this finding has repeatedly arisen in CEP’s research. In this new report, we gleaned more insight into what is getting in the way. The most frequent reason CEOs cited is that the board does not have a deep enough understating of the issue areas in which the foundation funds – a reason that can certainly be addressed by any foundation.

As with all of CEP’s research, we hope this report will spur conversations in the staff meetings and board rooms of those who read it.

 

Ellie Buteau is Vice-President—Research at the Center for Effective Philanthropy.

Tags: , , , , ,
Posted in Assessing Performance, Uncategorized | 1 Comment »

The State of Performance Assessment, 10 Years Later

Tuesday, September 6th, 2011

A decade ago, I started my job at CEP and we launched a study on performance assessment at large foundations. It’s not as if we were the first to think the question of whether a foundation was achieving its desired results was important. The earliest major American philanthropists cared deeply about results and some foundations, like Robert Wood Johnson Foundation, had been putting major resources into evaluation for decades.

But what was lacking, at that time, was a look across the large foundations to understand both practices and attitudes when it came to performance assessment.  So we undertook that study with limited resources — $345,000 in grants from the Atlantic Philanthropies, Packard, and Surdna –  and a small staff (three of us worked on the project, and that was CEP’s sole effort in its first year).  We conducted a broad-based survey of CEOs and a series of qualitative interviews.

What we found was clear – and sobering.  CEOs didn’t feel particularly satisfied with the information they could tap to understand how their foundations were doing. They weren’t utilizing a broad set of indicators, tending instead to rely on evaluations that told them more about a particular grant or set of grants than about the overall effectiveness of the foundation they led. And many of them told us they wanted more timely indicators to help guide and inform their work.

A lot has changed in the past decade. The Bill & Melinda Gates Foundation and other new foundations have burst onto the scene. A whole set of infrastructure organizations, from CEP to GEO to Bridgespan to Guidestar, have either been founded or grown substantially. And the media has paid more attention to philanthropy, raising questions about how it operates and what it has achieved.

So we thought it made sense to take stock, again, of the state of performance assessment at large foundations. This time, we surveyed CEOs of foundations with at least $5 million in annual grantmaking.  We’ll release results of that study tomorrow at noon, EST.  Check this blog and our Web site to find out how performance assessment has changed and what the attitudes and practices are of foundation CEOs today.

Tags: , , , , , ,
Posted in Assessing Performance | 1 Comment »

Assessing Foundation Performance

Thursday, April 15th, 2010

At last year’s CEP conference in Los Angeles, I presented on the Irvine Foundation’s approach to assessing foundation performance, joined by David Colby from the Robert Wood Johnson Foundation. I think it’s fair to say that David and I were presenting approaches used by our respective foundations that remain works in progress – Irvine’s even more so than RWJF’s given its focus on this area for many more years.

In view of the interest in this topic at the conference, and my own desire to share what we are doing at Irvine in an effort to improve upon it, I appreciate the opportunity offered by CEP to write a series of blog posts on the subject of assessing foundation performance.

 I plan to do this in four parts, addressing the following topics:

  1. Why we developed an approach to foundation performance at Irvine
  2. What we have found particularly challenging about assessing foundation performance
  3. How our board has engaged with us on this subject
  4. Why assessing foundation performance is both important and necessary

In reflecting upon Irvine’s experiences, I hope to stimulate readers’ contributions to deepening our collective understanding of this important subject and to improving our efforts to measure and understand our performance as foundations.

The focus of these blog posts is Irvine’s Annual Performance Report, which informs Irvine’s board about impact in our three program areas and our overall institutional effectiveness. I include links throughout this post and hope readers might take time to scan those related documents. 

We developed our current performance assessment framework after a strategic planning process in 2002-2003 that led to our current focus on three grantmaking programs (Arts, California Democracy, and Youth). As we embarked on this new focus, we wanted a plan to assess the foundation’s performance, both within the grantmaking programs, and across the foundation as a whole. 

A group of board and staff members worked together for several months and, as part of that, explored best practices in foundation-wide assessment. Two examples that stood out at that time were assessments done by the RWJF and the Rockefeller Brothers Fund. 

From those models and our discussions about what the board wanted to learn about the foundation’s performance, we created a framework that balances an assessment of grantmaking impact with ways to track overall institutional effectiveness, which are the two broad areas that organize Irvine’s approach. 

We also felt it was critical to look at our performance through different prisms, so within these two broad areas, we examine specific grantmaking statistics and progress, evaluation results, broader institutional effectiveness beyond the grantmaking programs, and a range of other categories. This framework is the basis for our Annual Performance Report. 

The primary audience to date for these Annual Performance Reports has been our board of directors. This focus has enabled us to be clear about the report’s purpose, ensuring that we are providing the information that is most relevant for that audience. 

Once we created the first report for the board in 2007, we then published it on our website, as a manifestation of our commitment to transparency and as another model for others to draw upon as they explore their own approaches. Throughout, I have also been interested in the critiques and questions that our approach has generated so that we might improve upon it going forward. 

This description of our process for creating Irvine’s Annual Performance Report sounds very straightforward. The reality, of course, is a bit messier, and my next blog post will focus on some of the challenges we’ve experienced and how we have refined our approach based on what we’ve learned. 

I’ll conclude with a final, personal observation: We have invested a great deal of time on this subject because I am persuaded that my obligations as CEO include developing an approach to rigorous foundation performance assessment. Given the autonomy that private foundations enjoy, and the occupational risks of complacency and insularity, creating a framework for foundation assessment and then reporting to our board in some regular interval has kept us focused on how we apply our finite resources for greatest impact. That goal has motivated us to date and it remains why we are keenly interested in improving upon and refining our approach to foundation performance assessment. 

There are undoubtedly other perspectives, however, including those that might disagree with this premise and/or our approach. I hope you’ll take a few minutes to share your thoughts, and I would especially value any comments or questions that will push us as we continue to grapple with this important work. 

Jim Canales is President of the James Irvine Foundation 

*****

Disclaimers and Disclosures: The views expressed in the CEP blog by guest bloggers are entirely their own and do not necessarily reflect the opinions of the Center for Effective Philanthropy.

Tags: , , , , , ,
Posted in Assessing Performance, Assessment Tools | No Comments »

Has Anything Really Changed in Philanthropy?

Monday, April 12th, 2010

Sean Stannard-Stockton, CEO of Tactical Philanthropy Advisors, and founder of the terrific blog Tactical Philanthropy, invited me to join a team of bloggers sharing thoughts on the GEO conference this week in Pittsburgh.  What follows are some musings on what’s changed since the first GEO conference I attended in 2002.  To link to this and the other guest blogs on the Tactical Philanthropy site, go to http://tacticalphilanthropy.com/2010/04/has-anything-really-changed-in-philanthropy.

The first Grantmakers for Effective Organizations (GEO) conference I attended was in Washington, D.C., in 2002.  It was tough to get in: GEO wouldn’t let me register because I was the executive director of a tiny (four staff) organization that no one had ever heard of  – the Center for Effective Philanthropy (CEP). I said, “Come on, we share a middle name.  We’re practically related!”  But they weren’t having it. 

So, with a little help from Vince Stehle, then program officer at the Surdna Foundation (one of CEP’s initial funders), I registered as if I worked for Surdna and showed up.  Vince hosted a breakfast roundtable to discuss CEP’s first report, on overall foundation performance assessment, and I was thrilled when we had to pull two tables together because 17 people showed up.

Fast-forward to the eve of the GEO conference in 2010 and allow me to switch – as my 9-year old daughter often does when she writes – to the second person and say this: you are expecting me now to discuss “how much has changed” when it comes to funder effectiveness and assessment since that conference almost a decade ago.  And, I will, but it’s not quite so simple.

Fact is, I think we tend, in philanthropy (and maybe in life), to overlook historical examples and proclaim that something is “new” when it is really only new to us. 

The media, and even those within philanthropy, often speak as if the focus on impact, outcomes, measurable results – whatever we call it – is brand new when, of course, it is not.  As Ed Skloot, now of Duke University, has noted, “Our earliest American philanthropic ancestors—John D. Rockefeller, Margaret Olivia Sage . . . , Andrew Carnegie — bequeathed a kind of courage and determination to us, a confidence we can take on large-scale problems with deeply rooted causes.”  Bill Schambra, of the Hudson Institute, also argues – although for him, unlike Skloot, it is a lament –  that the “mania to measure” goes back to the “first days” of the Rockefeller Foundation.

So, let’s not pretend that, just because our friends at McKinsey have launched a Web site focused on impact assessment, the concept was invented yesterday.  But, at the same time, it’s clear there has been dramatic change over the last decade in philanthropy.  New donors have brought new ideas, new and creative approaches, and new energy to the table. Numerous organizations that were either fledgling or non-existent a decade ago are now providing crucial data on nonprofits to funders:  Guidestar, Charity Navigator, Philanthropedia, Great Nonprofits.  And the list goes on.  And then there are the organizations focused on funder effectiveness:  GEO, GrantCraft, CEP.

But are funders operating more effectively?  Are they achieving more impact?  We at CEP see real signs of hope  – funders making clear their goals, strategies, and performance indicators; funders getting – and acting on – feedback from crucial stakeholders, from grantees to intended beneficiaries; boards really holding CEOs accountable for performance. 

I can point to some inspiring examples of present-day foundation effectiveness and impact, from the Stuart Foundation’s Child Welfare Program to the Wilburforce Foundation’s work protecting wildlife habitat to the Gill Foundation’s work on gay rights to the groundbreaking and much-discussed work of the Edna McConnell Clark Foundation.  And we at CEP have some new data we’ve been analyzing that will suggest some movement in the right direction. But, quite frankly, the movement is less dramatic than many of us would hope.

The reality is that there are still far too many staffed foundations doing significant (in dollars) grantmaking that aren’t even clear on what, exactly, they’re trying to do, much less how they’ll do it or how they’ll know if they’ve been successful.  So, my question is, how much really has changed since that conference in 2002? 

In my less optimistic moments, I wonder: is it possible that there has always been a subset of foundations that really care about effectiveness and impact and that their proportion of the overall number of foundations hasn’t changed much since the early days of Carnegie, Rockefeller, and Sage – even as the resources available to help them have grown and improved?  (A related question: how common is it that the very same foundation goes through periods of being highly effective – with clear goals, coherent strategies, and good performance indicators – and then other periods of being unfocused and adrift? I have seen many examples of this, suggesting that it’s not about “new” philanthropy or “old” philanthropy but rather “effective” philanthropy.)

My fundamental question is, how much change, really, has there been? Today, we can’t really answer that question as definitively as I would hope.  I’d be curious to know what others think.

(Disclosure: The foundations I mention above are among the more than 200 we have either provided assessment tools to or received funding from – and the former CEO of Edna McConnell Clark sits on our Board of Directors.)

Tags: , , , , , ,
Posted in Assessing Performance, Developing Strategy, Funder/Grantee Relationships, Uncategorized | No Comments »

« Older Entries