Posts Tagged ‘assessing performance’

Data Point: Learning From Failure

Friday, October 21st, 2011

The use and management of data stands at the core of the work undertaken by the Center for Effective Philanthropy. The set of survey tools CEP has developed as well as field-wide research builds comparative data drawn from key constituent groups—grantees, donors, staff members and others—providing insights that enable funders to better define, assess and improve their effectiveness.

Here is another finding from our recent survey exploring the current status of performance assessment among larger foundations. The survey was conducted in January and February of this year, and we received responses from CEOs of 173 U.S. foundations with annual grantmaking of at least $5 million.

For the above item, CEOs taking this survey were asked to indicate, on a scale of 1 (Strongly disagree) to 7 (Strongly agree), the extent to which they agreed or disagreed with the statement:

“Foundations would benefit from being able to hear more stories about foundation failures”

Just over three-quarters of CEOs agreed with this statement and another 14 percent were neutral. Meanwhile, fewer than one in 10 CEOs disagreed and not a single one responded stating they “strongly disagree” that stories about foundation failures would benefit their work.

*   *   *   *   *   *   *   *

To read about current foundation CEOs’ attitudes toward assessment and what foundations are doing to understand their performance, see the report, The State of Foundation Performance Assessment: A Survey of Foundation CEOs written by Ellie Buteau, Ph.D. and Phil Buchanan and published by the Center for Effective Philanthropy.

 

Ellie Buteau is Vice President – Research at the Center for Effective Philanthropy.

 

 

 

 

 

Data Point: Developing Shared Measurement Systems

Friday, October 14th, 2011

The use and management of data stands at the core of the work undertaken by the Center for Effective Philanthropy. The set of survey tools CEP has developed as well as field-wide research builds comparative data drawn from key constituent groups—grantees, donors, staff members and others—providing insights that enable funders to better define, assess and improve their effectiveness.

 

This data point is drawn from a survey of the CEOs of 173 U.S. foundations with annual grantmaking of at least $5 million and focuses on the current status of performance assessment among larger foundations. The survey was conducted in January and February 2011.

Our survey focused on assessing individual foundation performance. However, because foundations are typically working as one of many actors seeking to achieve shared goals, there has been significant interest in the development of shared measures.

The majority of CEOs report their foundations are already using, or have considered using, shared measurement systems:

» 26 percent said they are using coordinated measurement systems with other funders.

» 23 percent said they are considering using such measurement systems.

» 10 percent said they considered such systems but decided not to use them.

In addition, 36 percent of CEOs cited the tracking of data collected by other organizations as a source of information for assessing programmatic performance.

Readers of this blog post are invited to respond. What has your experience been with shared measurement systems?

 

* * * * * * * *

To read about current foundation CEOs’ attitudes toward assessment and what foundations are doing to understand their performance, see the report, The State of Foundation Performance Assessment: A Survey of Foundation CEOs written by Ellie Buteau, Ph.D. and Phil Buchanan and published by the Center for Effective Philanthropy.

Ellie Buteau is Vice President – Research at the Center for Effective Philanthropy.

 

 

Guest Post: How the Robert Wood Johnson Foundation Seeks To Improve

Monday, October 3rd, 2011
At the Center for Effective Philanthropy, we believe that improved performance of philanthropic funders can have a positive impact on nonprofit organizations and the people and communities they serve. As part of our work, we aim to highlight stories from funders who share that vision and who value the role of data and assessment in efforts to increase their impact.

 

PROCESS PROGRESS: HOW RWJF STRIVES TO IMPROVE
by Robin Mockenhaupt, Ph.D., M.P.H., Chief of Staff,
Dee Colello, Senior Manager, Program Operations, and
David Adler, M.P.A., Communications Officer
of the Robert Wood Johnson Foundation

 

They must often change, who would be constant in happiness or wisdom. ~Confucius

Change and continual improvement is a valued part of the Robert Wood Johnson Foundation’s culture. It’s in our guiding principles, which state, “We must commit ourselves to lifelong learning and continual improvement.” In addition to change and improvement, RWJF is also committed to transparency and peer learning, and it is in that vein that we are sharing our progress on quality improvement since 2004.

As part of our 2004 annual organizational assessment, we commissioned the Center for Effective Philanthropy’s Grantee Perception Report (GPR). We learned that while our grantees rated us comparatively well in several areas, we were using up a lot of grantee time meeting our administrative demands and we weren’t moving as fast as we needed to in processing grants. We also learned we needed better clarity in our communications of goals and strategies.

A 2011 CEP case study, Frequent Checkups Make for Healthier Funding Relationships, illustrated that we changed. We wanted to share one of the ways we took that advice to heart.

With the findings from the GPR, input from an all-staff retreat, and a focus group of grantees, we developed our first Foundation-wide Quality Improvement (QI) initiative. Our process for implementing this QI project can be broken down to five steps.

  • We set the right tone. The all-staff retreat and the announcement of the QI process by our president and CEO, Risa Lavizzo-Mourey, M.D., M.B.A., helped create widespread buy-in among staff. Any organization considering quality improvement projects should recognize that having the buy-in and public support of the CEO is a crucial first step for setting the appropriate environment for change.
  • We started with something manageable. The initial focus of our quality improvement work was on a category of grants that accounts for about 20 percent of our grantmaking. We wanted to start with something manageable to see how it worked before rolling out broader efforts.
  • We gathered the troops. After the announcement, staff interested in quality improvement convened to map out current grantmaking processes; shortly after that, a smaller core staff team was chartered, supported by a group of project sponsors from senior management.
  • We jumped in. The team designed a pilot. After testing and implementation, the project moved into control (maintenance) phase. Our first QI project resulted in sequencing and prioritizing steps in our grantmaking process, as well as launching the Foundation’s Program Information Management System (PIMS).  After our first QI project, two other projects were designed and implemented, using a similar structure and process. In addition, three smaller projects were led by staff trained in the QI process.
  • We are monitoring ongoing progress. We needed a way to monitor how we were doing and for identifying new ideas for improvement. We organized a standing staff group called the Process Improvement Group to help track metrics for our grantmaking and to initiate new quality improvement initiatives. Additionally, other units within RWJF have taken up their own quality improvement initiatives.

What did we learn?

  • Communications is a key component to implementing quality initiatives and staff behavior change.
  • Staff like being involved in cross-functional improvement projects
    when they see the need for change and can be a part of the solution.
    It’s also an opportunity to involve staff at every level of the
    organization.
  • Staff need dedicated time for QI work, as opposed to trying to “fit it in” around other responsibilities.
  • Over time, managers learned better how to scope and implement QI projects.
  • The automation of our grantmaking process (which initially was
    paper) allowed for project milestones and timelines to be standardized
    and to become transparent to all staff.

 

We’re pleased that more recent CEP reports have concluded our grantee perceptions have gotten better over time, and we believe our quality improvement efforts were a factor in this change. Our responsiveness measures are now higher as well as our quality of interactions. With that said, all our quality improvement efforts were not successful and we are receptive to revising any processes that may have missed the mark. For example, we are still working toward reducing the amount of time in both selection process and improving our ongoing monitoring and reporting.

We’re happy to share additional information about our QI process and are eager to hear how our colleagues are approaching this as well.

Data Point: Talking About Staff Performance

Friday, September 23rd, 2011

The use and management of data stands at the core of the work undertaken by the Center for Effective Philanthropy. The set of survey tools CEP has developed as well as field-wide research builds comparative data drawn from key constituent groups—grantees, donors, staff members and others—providing insights that enable funders to better define, assess and improve their effectiveness.  We are posting this series to share our data more broadly and to highlight specific data points.

In this case the source is the Staff Perception Report (SPR), which explores foundation staff members’ perceptions of foundation effectiveness and job satisfaction on a comparative basis. The SPR is based on a survey specific to foundations that includes questions related to staff members’ impressions of their role in philanthropy, satisfaction with their jobs, their foundation’s impact, and opportunities for foundation improvement.

 

 

The data point shown above comes from responses to the following question:

Was your performance formally reviewed during the last 12 months? Those who answered ‘Yes,’ were then invited to agree or disagree with the following statement: Yes, I had a conversation about my recent performance with my supervisor.

Over 700 foundation staff members responded to this question. Of those, 551 individuals or 76 percent indicated that they had such a conversation, and 170 individuals or 24 percent said they had had no conversation with their supervisor about their performance.

 

Kevin Bolduc is vice president — assessment tools at CEP.

New Report on Strategy at Community Foundations Sparks Debate

Thursday, September 22nd, 2011

The topic of strategy has long been central to research undertaken by the Center for Effective Philanthropy, because we are convinced that strategy is essential if foundations are to maximize their impact on people, communities and issues.

For this reason, we have spent considerable energy exploring how to define strategy in philanthropy and how to determine the degree to which foundations do – or don’t – use strategy in their day-to-day work. Our most recent research report (released last week) looks at strategy at community foundations and concludes that, just as is the case at private foundations, there is a gulf between rhetoric and reality. CEOs say strategy is crucial, but few are really using strategy – at least as we define the term.

The majority of the feedback we have received from community foundation leaders in response to this report has been positive so far. Our findings resonate with them and their experience. But we have also received a few challenging comments from community foundation CEOs. I want to lay out the three critiques we have heard, and our response to each.

Critique 1: The fatal flaw is…the application of the definition of strategy developed for private foundations as the lens through which you evaluated strategic behavior at community foundations.

The definition of strategy used in our research was the following:

A framework for decision-making that is 1) focused on the external context in which the foundation works, and 2) includes a hypothesized causal connection between use of foundation resources and goal achievement.

Before designing this research, we sought feedback from a variety of community foundation leaders about whether or not we should be approaching this research with this definition. Overwhelmingly, community foundation leaders supported the use of this definition in our research.

Then, while analyzing the data collected through our interviews, we considered whether or not the definition of strategy should be changed or expanded. After several months of data analysis and conversation, our conclusion was that nothing in CEOs responses indicated a different definition was applicable to community foundations.

Community foundations generally met the first condition of our definition: external orientation. It was the second condition – includes a hypothesized causal connection between use of foundation resources and goal achievement – where community foundations did not come through.

Might there have been a better definition for strategy in community foundations than the one we used in our research? Perhaps. But it’s hard for me to imagine any definition of strategy that doesn’t include the very basic requirement of logical connections between use of resources and goal achievement. As a result, our finding that few community foundations are strategic would still hold.

Critique 2: Being strategic is not black or white; there’s a huge spectrum along which all of us lie.

We at CEP agree with this statement. In the methodology for this report, we explained that in order for a CEO to be categorized as strategic, half of the decision-making processes CEOs shared with us with regard to how they work to achieve the foundation’s goals had to meet the definition of strategy. This criterion allowed us to be very inclusive about who was categorized as strategic.

In addition, we did identify a group of CEOs as partly strategic, as noted in the report. The bar for being categorized in this way was even lower – requiring only one decision-making process to fit our definition of strategy.

We also tried to capture some of the nuance and spectrum by categorizing CEOs separately with regard to their donor and programmatic work. That means, for example, a CEO could be categorized as strategic in donor work but not in programmatic work.

We did receive a suggestion during the process of collecting external feedback for this report, prior to its release, that we should frame our findings more gently by refraining from using the term “not strategic.” But rather than make our decision about how to word these findings on the basis of what people wanted to hear, we made our decision according to what the data were indicating.

Critique 3: But wait – I can name many community foundations I think are strategic.

One of the main reasons that CEP conducts research is to steer conversations away from anecdotes and towards more systematic data.

Based on our data and analysis for this research, we did identify, among our respondents, some strategic community foundation CEOs – and we highlighted the work of three of them in the report: California Community Foundation; Orange County Community Foundation; and Community Foundation of the Eastern Shore. What we found, however, is that strategic foundations are more the exception than the rule.

Some community foundation leaders clearly disagree with our findings and how we arrived at them. Others, however, told us they did not find our results surprising at all. They said the fact that the majority of those we interviewed were not classified as strategic hardly seemed like news.

In all of CEP’s research, we recognize the critical role foundations play in our society. Our research is conducted to advance knowledge and contribute to conversations about important and challenging issues that foundations face. As with all of our reports, we invite you to share your comments.

 Ellie Buteau, Ph.D., is vice president – research at CEP.