Posts Tagged ‘quantitative performance meassurement’

Esther Duflo Explains Why She Believes Randomized Controlled Trials Are So Vital

Thursday, June 23rd, 2011

Even before Esther Duflo rose to speak at the Center for Effective Philanthropy’s (CEP) 2011 conference, she had already stirred controversy. Her well-known belief in the ability of randomized controlled trails to understand the effectiveness of policy interventions in developing countries brings up strong feelings among some in the philanthropy world.

As Ford Foundation president Luis Ubiñas said in his introduction of Duflo, “Feelings runs so deep about Esther that Phil [Buchanan] actually received some tough emails about her coming to speak here to us.”

But Duflo seemed to quickly win the crowd over.

“I’m sure I’m not very dangerous,” said Duflo, professor of Poverty Alleviation and Development Economics at MIT. “I speak with a funny French accent. I don’t bite.”

In explaining her approach, Duflo said that discussions about foreign aid often devolve into a debate about whether the aid works or not. It is impossible to resolve that debate by staying on that level, said Duflo who is founder and director of the Abdul Latif Jameel Poverty Action Lab.  For one, the question of whether aid works is not particularly well defined, she said. Second, aid is a small part of developing countries’ overall budgets. For example, aid comprises about 5.7% of the budgets of African countries that receive it.

“A big part of my work is to try and shift the conversation from whether aid is good or bad to think about policy or programs instead,” Duflo said. “Another objective of my work is to think about not just the five percent [of aid], but the 100 percent.  [That is], what role this five percent can play in improving the quality of programs. I want to think about the efforts of most private donors…as not being an end in itself, but as being venture capitalism and finding the good ideas in development. In that case, think of each dollar you are spending as being multiplied many, many fold. If these programs help us identify what really works then that can be taken up as a policy on a very large scale. That is the reason for my work and the reason for placing so much emphasis on the evaluation of specific programs.”

Duflo cited several examples from the book she co-authored Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty.

“I thought I would identify surprising facts that I didn’t know and maybe you didn’t know either and that would have been difficult to know without employing the kinds of methods we work with, particularly randomized controlled trials,” she said.  “The first one is that sometimes you save money by paying people.”

Childhood vaccines are generally available for free and yet each year millions of children go without basic immunization, Duflo said. She described a randomized controlled trial of three immunization camps in rural India.  She found that by giving parents an incentive to go to the camps to immunize their children—in the form of one kilo of lentils for each child immunized—the rate of immunization increased from 6% to 38%. (For more on this topic, see Duflo’s TED Talk.)

She said it the trial showed that it actually saves money to give the lentils. The cost for a child to be fully immunized in a camp without the incentives was $50 while it was $27 in a camp with incentives. That’s because the camps must pay for a nurses’ time regardless of the number of children they immunize. At the camps with incentives, nurses immunized more children and the costs per immunization decreased.

“Because of the randomization, seeing the effect is very transparent and easy,” she said.

Duflo said that findings like these challenge what she calls the three “I”s of development policy: ideology, ignorance and inertia.

“Programs are often borne in ideology,” she said, “[An ideology that] the poor are entrepreneurs, or they are starving or they are slothful. [Programs] are conceived in ignorance of the reality of the field, and then they persist because once they exist there is a consistency for them to just continue. I think we need to fight against that.”

As an example of the three “I”s Duflo cited a government policy in India to have village education committees on which villagers serve. That policy has not worked as planned.During the question and answer period, audience members asked Duflo to expand on her ideas.

At the end of the session in response to one question, Duflo showed how she can be both soft-spoken and laser sharp in her point of view.

“From our position of being reasonably well off and comfortable, [perhaps] university professors, we tend to be patronizing about the poor in a very specific sense, which is that we tend to think, ‘Why don’t they take more responsibility for their lives?’ And what we are forgetting is that the richer you are the less responsibility you need to take for your own life because everything is taken care for you. And the poorer you are the more you have to be responsible for everything about your life….My lesson is to stop berating people for not being responsible and start to think of ways instead of providing the poor with the luxury that we all have, which is that a lot of decisions are taken for us. If we do nothing, we are on the right track. For most of the poor, if they do nothing, they are on the wrong track.”

More highlights from Duflo’s talk at CEP’s conference will be available on CEP’s YouTube channel soon.

Susan Parker is owner of Clear Thinking Communications

Understanding the Complexities of Assessment

Friday, January 15th, 2010

In yet another excellent and thought-provoking post, Bob Hughes raises important concerns about a focus on quantitative metrics to the exclusion of qualitative assessment.  All of us pushing for greater assessment must guard against the risks Bob outlines, and I wholeheartedly agree that “the heart of measurement is comparative assessment, and this doesn’t require quantitative analyses.” 

In our discussions of assessment, we must always keep in mind the question Mario Morino raises in a provocative and important new column on the VPP Web site: “To What End?” His thinking echoes Bob’s – and vice versa – in important ways.  I applaud Mario for his candor and his willingness to shine the light on what he has come to see as flaws in his past approach.  

Mario writes:

Here’s an example of how I looked too narrowly at outcomes—and, as a result, risked knocking nonprofits off mission.

I regret not being more open in my thinking back then. Instead of pushing back on what we were hearing, we should have done more to understand “soft” achievements that may in fact be every bit as real and important as “harder” outcomes. I aspire to do a better job of making them part and parcel of future efforts to assess outcomes and performance—even if that means using qualitative and/or anecdotal indicators.

In the early years of Venture Philanthropy Partners, we got a lot of resistance to my push for “clearly defined outcomes” from leaders whose organizations placed a premium on being holistic with their services and functioning as “community builders.” Although I agreed with them in concept, I felt that a focus on “community building” was too soft to be a legitimate outcome. Outcomes related to “community building” are, after all, radically ambiguous compared to outcomes like reduction in teenage pregnancy and substance abuse.

I now see better that serving the entire family (holistic services) and building community are some of the very things that create the environment—a web of support and community—that helps youth avoid high-risk behavior, get an education, and prepare for college or a job. But talking about “community building” was too intangible, and not readily measureable to us at the time—and, candidly, difficult to sell to our own stakeholders and the emerging field of nonprofit performance at large.

This column should be read in its entirety by every funder.

What Are the Limits of Quantitative Performance Measurement?

Wednesday, January 13th, 2010

CEP’s Essentials of Foundation Strategy concludes, “Assessment of results against strategies remains a significant challenge for foundations: staff struggle to determine the right data to collect and how to collect it.”

This is an important finding, and explanations are well worth exploring.  The report suggests several possible reasons – technical challenges, inadequate resources and support, and lack of grantee capacity and skill.  I’d like to explore another possibility: The undue attraction to quantitative analysis.

The recipe for foundations is simple and well known:  assess the environment, choose a goal, align resources, implement programs, measure performance, adjust. Repeat as needed until goal is achieved.

Indeed, every foundation’s recipe will have a different mix of ingredients, but maybe we need to think about the limits of quantitative measurement. As much as I value quantitative data to assess results, the proliferation of such work and its unintended consequences suggest we consider (to extend the recipe analogy) if the basic philanthropic ingredients contain a dash too much fascination with numbers.

It might seem impolite to raise this question on CEP’s blog, given that the heading on this new Web site begins with “better data . . .”!  But I know CEP is committed to being a learning organization, which includes fostering a supportive environment for raising different points of view.  So – what better place to ask about the seemingly unending efforts to quantify performance at every level to assess impact than on the Web site of an organization devoted to empirical analysis?

What are the possible downsides of quantitative performance measurement?  One is that doing this work has its own costs, so spending resources on this ingredient has to be done as effectively as possible, and weighed against spending for other ingredients in the philanthropic recipe.  Another is the mistaken belief that quantitative measurement is the only type of measurement.

Even when qualitative measurement is considered, it is often viewed as inferior to quantitative analysis.  Yet the heart of measurement is comparative assessment, and this doesn’t require quantitative analyses.  Think of the meaning of “taking the measure of a person” or “to speak in measured terms.” These phrases denote the sense of measurement as judgment or comparison.

A related risk is the lack of fit between the measure of performance and what is being measured.  What is the nature of philanthropic practice that we want to measure?  Are foundations measuring their own performance (i.e., strategy, selection of goals) or their grantees’ performance (project results, program impact)?  One conception of philanthropy is that it is craft.  How amenable to quantitative measurement is the craft of philanthropic practice?  At what levels?

Thinking about the limits of quantitative performance measurement suggests several things that might help foundations improve assessing their impact.  One is to take into account the costs and benefits to everyone involved when undertaking this work, and what is most important to learn to accomplish strategic goals.

For example, it is probably more valuable to put performance assessment resources to broader levels of performance (e.g., is an overall strategy working?) than to discrete grantee projects.  Focusing on individual projects inhibits an understanding of how projects fit together over time and relate to the environment in which they operate.  In turn, this reinforces a focus on a project’s internal risk – implementation – and away from strategic and design risks.

Note that the failure of implementation puts the spotlight on grantees; failure of strategy and, to some extent, design, puts the spotlight on foundations.  An excellent example of a higher-level, strategic assessment is the qualitative assessment that Patti Patrizi and colleagues did on RWJF’s end-of-life grant-making from 1996-2005.

In preparing for performance assessment, thinking carefully about design, the limitations of the data to be produced, and how the data produced will be used before committing resources may also minimize an overreliance on quantitative assessment.

In undertaking performance assessment, it is useful to think of the kinds of comparisons (quantitative or qualitative) that fit with the goals.  Case studies, like the one CEP published on the Stuart Foundation, are excellent ways of using a comparative framework – in this case to a model of bringing about social change – to assess and learn about performance.

Assessing performance rigorously is a critical ingredient in strategic philanthropy.   Questioning the limits of quantitative performance assessment should not be used as a reason not to do performance assessment – just the opposite. The field needs more resources devoted to this ingredient, but they need to be used as wisely as possible.

The challenge is getting the right mixture of ingredients, including types of measurement, for each foundation.  And as long as we keep in mind that all data are not numeric, CEP’s banner that begins with “better data . . .” is the start of a good recipe for philanthropic chefs.

Bob Hughes is an independent consultant on strategy and organizational learning in health and philanthropy.

*****

Disclaimers and Disclosures: The views expressed in the CEP blog by guest bloggers are entirely their own and do not necessarily reflect the opinions of the Center for Effective Philanthropy.