Posts Tagged ‘performance measurement’

Data Point: What Information are Foundations Using to Assess Their Programmatic Work?

Friday, December 9th, 2011

For our recent State of Foundation Performance Assessment report, we gathered data from 173 CEOs of U.S. foundations with annual grantmaking of at least $5 million on what types of information they use to assess the effectiveness of their foundations’ work – operations, finance, and programmatic. Compared to a similar survey we completed almost a decade ago, it appears that foundation CEOs today are drawing on an increasingly broad array of performance indicators.

In this survey, we listed a range of types of information CEOs may be using to assess the effectiveness of their foundations’ work, and CEOs could select as many as applicable.

On average, CEOs report using seven types of information to understand their foundations’ programmatic effectiveness. Almost all foundations are using anecdotal feedback, written reports from grantees, site visits, and evaluations – either of individual grants, clusters of grants, or program areas.

A minority, however, seek information from their ultimate beneficiaries – the people whose lives foundations are ultimately trying to affect – either through surveys, focus groups, or convenings. When we compare responses of CEOs whose foundations do and do not seek the voices of their ultimate beneficiaries, two important differences emerge: CEOs who report that their foundation does collect beneficiary feedback rate themselves as having 1) a better understanding of the progress their foundation is making against its strategies, and 2) a more accurate understanding of the impact the foundation is having on the communities and fields in which it works.

 * * * * * * * *

To read about current foundation CEOs’ attitudes toward assessment and what foundations are doing to understand their performance, see the report, The State of Foundation Performance Assessment: A Survey of Foundation CEOs written by Ellie Buteau, Ph.D. and Phil Buchanan and published by the Center for Effective Philanthropy.

Ellie Buteau is Vice President – Research at the Center for Effective Philanthropy.

 

Evaluation Roundtable Study Highlights the Role of the CEO in Evaluation

Wednesday, November 16th, 2011

report written by Elizabeth Heid Thompson and Patricia Patrizi (and currently available on the CEP website) explores the extent to which foundations evaluate the results of their work. An examination of 31 foundations that have demonstrated a commitment to evaluation over time highlights several key facts.

According to the report, published for the Evaluation Roundtable, funding to support evaluative activities has decreased despite an increase in the demand for the information those activities produce. The number of foundation staff devoted to these activities has also declined in recent years.

Much of the current investment in evaluation is focused on performance metrics, often administrative metrics, rather than on the measurement of the strategy behind the work or on the implementation process. In fact, many evaluation leaders raised the following concerns about the metrics used:

  • That the metrics they were tracking did not adequately align with their strategies;
  • That their investments did not make a difference in moving the needle; and
  • That metrics chosen often reflect goals too distant to inform the way a strategy is implemented.

The good news discovered in this research, which took place in 2009, was about the role of foundation CEOs in the evaluation process. They report that when the evaluation unit reports to the CEO, more financial resources will be provided, the evaluations will be more widely distributed and more attention will be paid to the findings.

 

Unexpected Examples of the Very Real Power of Data and Analysis

Wednesday, October 26th, 2011

To my surprise, baseball and Shakespeare have both recently underscored for me the importance of the mission and values of the Center for Effective Philanthropy, putting in to focus why I made the decision to join this organization. Not two elements I would expect to find sharing a Venn diagram, but the common theme has to do with the importance of data and the power of analysis.

By baseball, I mean the hit movie Moneyball (based on the book by Michael Lewis), which examines the innovative strategy put into play by general manager Billy Beane. He depended on close analysis of the proven performance by a more or less motley crew of players to build a team within tight budget constraints. What these players held in common was an ability to play at a level that exceeded their value as judged by conventional metrics. Beane stands for the power of data over hunch, and for a rejection of the accumulated experience of anecdotal evidence, a significant departure in the culture of baseball.

And by Shakespeare I mean a recent essay by Stephen Marche, a columnist most frequently read in Esquire, the Wall Street Journal and the New York Times. In a Sunday Times Magazine piece, Marche presents a compelling and timely critique of the durable legend that Shakespeare’s plays were written by someone else—in this case by Edward de Vere, earl of Oxford, most recently aired in the current film Anonymous. Marche dismisses the substance of this implausible narrative with a handful of compelling examples. (For instance: De Vere died before the events that inspired the plays MacBeth and The Tempest—let alone before the plays were written.)

Marche’s more important point is about the erosion of a culture of rigor and the threat that presents for all of us, outside of the somewhat restricted fields of theater and sport. The growing sense that it is somehow unnecessary, or even an affront, to speak from a position of informed expertise has profound implications on a wide swath of issues that matter a great deal to many of us and that will most likely matter more in the future, from climate change to the economic consequences of U.S. government default.

Moneyball offers compelling testimony to the real-world impact of data, making it possible to come to a deeper understanding of the processes that make things happen. It is easy to see the parallels to many thorny problems we confront—think education, or health care—that are themselves often understood through the power of anecdote. And Marche’s essay, titled ‘Wouldn’t it be Cool if Shakespeare Wasn’t Shakespeare?, speaks to the importance of beginning with the data and building understanding from there, rather than starting with a hypothesis and retro-fitting the evidence to accommodate it.

I am willing to concede that if someone were to make a movie out of CEP’s recent report, The State of Foundation Performance Assessment: A Survey of Foundation CEOs, it probably wouldn’t serve as a good vehicle for Brad Pitt. But the arguments pressed by these two examples drawn from the culture of our times take me back to the mission of CEP: To provide data and insight so philanthropic funders can better define, assess and improve their effectiveness. And to the vision: We seek a world in which pressing social needs are more effectively addressed.

In an age when the challenges we face are so large and complex, I just don’t see how we get to that world without a culture of rigor and analysis. We must maintain the ability to step away from conventional understanding and from the pleasures of conspiratorial thinking when the data rules them out.

Since leaving the Boston Foundation this summer to join CEP, colleagues have asked me why I made the jump from a well-established community foundation with a long record of visible accomplishment to join a young nonprofit with a mission that can seem abstract at times.

This is why. Because I want to spread the message that data matters, that rigor matters, if we are to make the changes we wish to see in our society.

 

David Trueblood is vice president — Communications & Programming at CEP.

Data Point: Developing Shared Measurement Systems

Friday, October 14th, 2011

The use and management of data stands at the core of the work undertaken by the Center for Effective Philanthropy. The set of survey tools CEP has developed as well as field-wide research builds comparative data drawn from key constituent groups—grantees, donors, staff members and others—providing insights that enable funders to better define, assess and improve their effectiveness.

 

This data point is drawn from a survey of the CEOs of 173 U.S. foundations with annual grantmaking of at least $5 million and focuses on the current status of performance assessment among larger foundations. The survey was conducted in January and February 2011.

Our survey focused on assessing individual foundation performance. However, because foundations are typically working as one of many actors seeking to achieve shared goals, there has been significant interest in the development of shared measures.

The majority of CEOs report their foundations are already using, or have considered using, shared measurement systems:

» 26 percent said they are using coordinated measurement systems with other funders.

» 23 percent said they are considering using such measurement systems.

» 10 percent said they considered such systems but decided not to use them.

In addition, 36 percent of CEOs cited the tracking of data collected by other organizations as a source of information for assessing programmatic performance.

Readers of this blog post are invited to respond. What has your experience been with shared measurement systems?

 

* * * * * * * *

To read about current foundation CEOs’ attitudes toward assessment and what foundations are doing to understand their performance, see the report, The State of Foundation Performance Assessment: A Survey of Foundation CEOs written by Ellie Buteau, Ph.D. and Phil Buchanan and published by the Center for Effective Philanthropy.

Ellie Buteau is Vice President – Research at the Center for Effective Philanthropy.

 

 

The Power of Donor Feedback

Wednesday, September 28th, 2011

The conversation at the Center for Effective Philanthropy (CEP) tends to move rapidly toward the subject of feedback. The assessment tools we provide to foundations make it possible to collect responses in areas relevant to foundation organization and practice from a wide range of constituent groups — grantees, local stakeholders and donors among them. The goal in every case is credible, comparative feedback that gives those constituents a voice and places them in context.

Donors and the Donor Perception Report (DPR) were the particular focus of a recent seminar conducted at the Council on Foundations’ fall conference by Kevin Bolduc, CEP’s vice president – assessment tools. Video from that session makes a strong statement about the utility of feedback in general but also about the power of comparative data.

What we see here is a look at what foundations get from the DPR. Above all, that is the contrast between what foundation staff members surmise donors think of them, and what the data compiled through CEP’s cumulative surveys actually reveal. During the seminar, Kevin asked attendees to suggest one word that donors might be expected to use to describe their community foundation. Participants offered up a list: trustworthy, responsible, connector, leader, engaged, pretty much in that order.

From research, he then listed what donors actually said to describe the community foundation they work with. These were, in order: effective, professional, helpful and efficient.

The point is that data replaces assumptions with facts. And even a seasoned hunch can benefit from that sort of informational update.

Taking part in the rest of the session with Kevin Bolduc were Amy Cheney, vice president for Giving Strategies at the Greater Cincinnati Foundation; Terence Mulligan, president of Napa Valley Community Foundation; and Sarah Nelson, chief philanthropy officer at the Communities Foundation of Texas. These foundations differ in terms of geography, mission and history but all share an appreciation of how measurable feedback can improve their work processes as well as the specific findings they drew from constituent feedback. A full video of this seminar will be available soon at www.effectivephilanthropy.org.

To find out more about the DPR and the power of feedback, friends and colleagues are invited to join CEP for a webinar from 2 to 3 p.m., EST, on Friday, October 7. Amy Cheney will share her experience with the DPR, and participants will have the chance to pose questions to her, as well as to CEP President Phil Buchanan and Kevin Bolduc. We have had a good show of interest in the webinar already, and expect a lively conversation.

Click here to register.

David Trueblood is vice president – communications and programming at CEP.