Posts Tagged ‘qualitative data’

A Dose of Honesty: The Tyranny of the Anecdote

Wednesday, February 10th, 2010

“We always felt we had a ‘Hannibal-the-cannibal’ relationship: Stay where you are and mind yourself and you’re okay. Take one step forward and your arm will be bitten off.”

“The Foundation has had a negative impact on our organization in spite of funding. Runaround by foundation staff and [the Foundation’s] nontransparent priorities and processes have wasted staff time and effort…. It is disheartening to think that the viability of an important program is subject to an unqualified person’s review, analysis, and presentation of our project to the full decision group.”

In my last “Dose of Honesty,” I highlighted the positive end of the spectrum of commentary we receive from grantees. Here, I figured I’d talk a bit about the other end, using these comments from surveys of two different funders.  Just like positive remarks, the terrible need equally careful consideration.

It’s a rare foundation that solicits candid, grantee feedback through CEP’s Grantee Perception Report, that doesn’t end up with a zinger in there. Invariably, comments like these cause serious freak-outs when staff members read them in the full list of redacted comments we provide.

Yet, as these one-off comments absorb attention and focus, they can distract from the important insights hidden in less snappy language in the rest of the qualitative feedback. So, staff should be upset – but mostly if those comments are part of a broader theme and not, as the zingers often are, just an indication of one very broken funding relationship.

One comment is scant evidence of failure – or, for that matter – of success. The glowing comment that gets tacked up on a bulletin board can wipe away dozens of comments that reveal utter mediocrity, or worse, a serious problem. As intoxicating as they may seem, like the zingers, one or two compliments – in a survey or at a dinner party – are just anecdotes.

Don’t get me wrong – qualitative anecdotes have their place. Terrible comments like those at the beginning of this post make for important warnings to foundations about what to try to avoid. They provide a visceral punch in the gut. But our focus at CEP is to raise up themes worthy of significant attention – and so we try to focus on trends in negative comments that highlight areas in which grantees say funders most need to improve.

****

Kevin Bolduc is Vice President - Assessment Tools at CEP


A Dose of Honesty: Greatness

Monday, February 8th, 2010

“Honestly, [this foundation] is the ideal funder. I find the whole grantee/grantor relationship distasteful in most cases, but with [this foundation] I really feel like we are equal partners working towards the same goals.”

In a guest post on this blog, Bob Hughes of RWJF highlighted one of the frequent apprehensions about assessment in philanthropy and, too often, about CEP’s work: that assessment is only focused on the numbers. Quantify this. Count that.

While I proudly admit to spending quite a bit of time thinking about the insights that we can pull from our quantitative assessment data, my colleagues and I spend a ton of time analyzing the comments we carefully solicit as part of our surveys. In fact, on many days, our teams spend a lot more time analyzing qualitative data about a foundation than they do buried in numbers.

It’s a rare assessment tool that doesn’t have at least one key finding drawn largely from the qualitative feedback. Doing substantive, frequently comparative analysis of qualitative data, though often difficult and time-consuming, is fundamental to our work. And, we believe, ultimately critical to surfacing new insight and to supporting themes from quantitative analysis.

This post begins a series that will highlight some of the candid, open-ended feedback we receive about funders. I hope the comments I choose to reflect upon will resonate, sometimes make you laugh – or gasp – be food for thought, or just plain illustrate the profound difference that many funders are making through their work.

Take, for instance, the comment at the beginning of this post. You might think that grantees throw around words like “ideal” and “best” pretty frequently in reference to organizations that have given them substantial sums of money. If you thought that, though, you’d be dead wrong.

In our experience it’s fairly unusual. However, for this funder, which I’ll identify when they make their Grantee Perception Report public, that comment was only the tip of the iceberg. About a quarter of its grantees provided open-ended feedback that used a superlative. Here are a few more.

  • “Best ever out of all the funders we work with.”
  • “Best foundation staff on earth.”
  • “The most thoughtful and helpful foundation in my field.  It adds value but respects its grantees.”

The litany continued with words like “favorite,” “most outstanding,” “stellar,” “fabulous,” “awesome,” “A++,” “impeccable,” that drove the point home to the board and staff – I think just as much as the exceptionally positive quantitative ratings grantee provided – that grantees believed this foundation was unusually effective.

In his next post, Bolduc will examine grantee quotes that give negative feedback in “The Tyranny of the Anecdote.”

****

Kevin Bolduc is Vice President - Assessment Tools at CEP