While the Cleveland foundation received good marks in general, nonprofit officials criticized its Civic Affairs grant program, which supported gun-control projects, racial-equity efforts, and projects designed to improve urban life. The fund had ambiguous goals and confusing guidelines, grantees said. The result: Gund eliminated the program and funneled the money to other efforts.

The survey that drove Gund's decision to abolish the $1-million program was conducted by the Center for Effective Philanthropy, a small nonprofit research group here that in a relatively short time has shaken up the foundation world.

The center's surveys, known as Grantee Perception Reports, for the first time allow a foundation to compare how charities rank its efforts with how other grant makers of similar size are judged. The reports have resulted in changes in foundation operations and have fostered a frank dialogue between grant makers and charities, which historically have been wary of speaking out against their supporters for fear of losing money.

19,000 Opinions

Driven in part by the scrutiny of nonprofit groups by the U.S. Senate and state regulators, many of the largest U.S. foundations have commissioned surveys of their grant recipients. Since the center's founding four years ago, 97 foundations — including seven of the top 10 wealthiest charitable funds in the nation — have asked for the reports, and the center has solicited the opinions of more than 19,000 charities in the process.

The Grantee Perception Report is one of five research tools the center uses to measure the effectiveness of foundations. The tools include surveys that ask for the views of foundation boards, staff members, and even charities that applied for grants but were denied.

Yet it is the Grantee Perception Report that has received the most attention — and caused the most consternation.

"We are certainly not making people comfortable when we are providing them data on what their grantees perceive," says Phil Buchanan, the group's executive director. As for the grant makers' beneficiaries, he says, "they're pleased to finally have the chance to turn the tables."

But some nonprofit experts warn that the data has led to too much foundation navel-gazing and that the Grantee Perception Report is no panacea for the long-standing problem of how to evaluate grant making.

The center examines "customer satisfaction," says Peter Frumkin, professor of public affairs at the RGK Center for Philanthropy and Community Service at the University of Texas at Austin. It does not examine what he says is key to measuring grant making: the overall success of the charities foundations support. "I'm not sure it's an answer to the quagmire of measuring foundation effectiveness."

Foundation Debate

The Center for Effective Philanthropy emerged out of a debate sparked by an article in the Harvard Business Review.

The controversial 1999 article, by Michael E. Porter, a Harvard Business School professor, and Mark R. Kramer, a nonprofit consultant and a regular contributor to The Chronicle's opinion pages, pushed foundations to improve their grant making. "Satisfied with their historic agenda of doing good, too few foundations work strategically to do better," the authors wrote.

Many grant makers resisted the ideas put forth in the article, but Atlantic Philanthropies, the David and Lucile Packard Foundation, and the Surdna Foundation gave the authors $345,000 in 2001 to explore ways to measure the effectiveness of foundations. That effort spawned the center.

Since then, the center's co-founders have left the organization, but its staff size has grown to 13 people and its budget has increased to $1.8-million, with 40 percent raised from fees it charges grant makers for surveys and conferences and the rest from grants.

Operating out of new offices it moved to in January, the organization's researchers crisscross the country, presenting their data to foundations.

The center's employees compare the group to a young Internet company, where the hours are long, but the work is exciting. "It's exhausting and exhilarating," says Sarah Di Troia, the center's associate director who leads its project to examine foundation governance practices. "It still feels like a start-up."

The center has hired a diverse group of people, including former nonprofit employees and business-school graduates. Mr. Buchanan, for example, worked as an assistant to the president of Mount Holyoke College and as a business consultant before joining the center.

The organization, however, has yet to hire someone with foundation experience, a move that some grant makers say would benefit the group's work.

While foundations talk to their grant recipients regularly and often have hired outside consultants to help them improve their work, the Grantee Perception Reports have elicited praise from grant makers for their comparative data and anonymous comments from charity officials.

The reports are part of the small amount of honest feedback foundations receive, says David T. Abbott, executive director of the Gund Foundation.

"It's a danger in this business that all we hear is how good we are and people laughing at all our jokes," he says. "We can't assume just because people tell us we're doing a good job that we're really doing a good job."

The center's efforts also receive high marks from charities. When grant makers commission the perception reports, "it shows to us they're interested in how it is to work with them. That's appreciated," says Erin S. Majernick, director of foundation relations at Pathfinder International, a charity in Watertown, Mass.

Ms. Majernick has filled out surveys for both Packard, in Los Altos, Calif., and the William and Flora Hewlett Foundation, in Menlo Park, Calif.

Using a scale of 1 to 7, the Grantee Perception Report grades foundations in categories such as "community impact," "foundation impact on public policy," and even overall "satisfaction" among grant recipients.

The report is derived from a 12-page survey with 58 questions that the center sends to every nonprofit organization a grant maker supported during one fiscal year. About 63 percent of grantees who receive the survey respond, the center says. The Center for Effective Philanthropy charges a grant maker from $5,000 to $40,000 for a report, depending on the number of grant recipients it has to survey.

Using the Data

Mr. Buchanan says he is pleased that almost 100 foundations have commissioned the reports — with nine of them paying for a second survey — and his target is to work with the 1,000 largest grant makers in the nation. But he says the number of foundations is less important than the center's primary goal.

"We don't care about how many foundations are using the Grantee Perception Report," he says. "We care how many are using the data to change and improve."

A recent study suggests that the majority of foundations that have commissioned a Grantee Perception Report have made changes, in part, as a result of it.

The study, which was conducted by LaFrance Associates, a San Francisco consulting company the center hired to assess its work, says the most common changes foundations have made include establishing better ways to communicate with grantees, easing application procedures, and providing more management training and other nonfinancial assistance.

The Bush Foundation, in St. Paul, for example, rewrote its grant-making guidelines as a result of its Grantee Perception Report, says the fund's president, Anita M. Pampusch.

"There were areas where the response gave us pause and made us think about how clear our guidelines are and whether people really get enough information from us to write a good proposal," she says.

Ms. Pampusch says the group cut out excess verbiage and wrote the guidelines in "more common prose" to help charities.

While Bush and other charitable funds have made adjustments, Mr. Buchanan says that some foundations have neglected to make changes despite unsatisfactory marks from grant beneficiaries. "I can't sit here and tell you that every foundation that has participated in this process is doing everything they should with the results," he says.

The center cannot make its grievances about specific foundations public because it has agreed to keep the survey data confidential.

"We made a decision very early on that we were going to make foundations more effective by working with folks in the field," Mr. Buchanan says. "There are trade-offs implicit in that decision."

Advertising the Results

But while the center keeps the information on the reports secret, 14 foundations have made their Grantee Perception Reports or excerpts from them public, warts and all.

While most of these funds have put the reports on their Web sites — including the Hewlett Foundation, which was the first fund to go public — the Rhode Island Foundation went a step further and advertised its results. Last year the fund summarized its report card in a quarter-page ad in the editorial section of the Providence Journal, a daily newspaper in the foundation's hometown.

The ad helped increase the number of people who viewed the report, with more than 675 people downloading it in the months after the advertisement ran, says Rick Schwartz, the foundation's spokesman. Mr. Schwartz encourages other foundations to follow his group's example, saying that negative comments in the report were not used to unfairly criticize the grant maker. "No one used it as a chance to slam us," he says.

While applauding the Rhode Island Foundation's and other grant makers' decision to go public with the results, some foundations say they will not follow suit. The report "is an internal tool. We didn't see it as a vehicle for public relations," says Lowell Weiss, a spokesman for the Bill & Melinda Gates Foundation, in Seattle, which commissioned a survey of the nonprofit groups that its U.S. education program supports.

Questions of Objectivity

To be sure, the center has its share of critics.

Some of them question whether the center can be independent of foundation influence, especially because it relies on their largess to finance its operations.

The National Committee for Responsive Philanthropy, a foundation watchdog in Washington, says the center furthers the "political agenda" of foundations by helping them justify high administrative expenses with its research, which has said charities are less concerned about the dollar size of their grants than with the noncash assistance foundations can provide.

Mr. Buchanan dismisses the charges, saying the center only presents what charities are telling it and that it is not subject to its supporters' whims.

Another official whose foundation commissioned a Grantee Perception Report, says the center is "cocky" and uses a one-size-fits-all method that gave an uneven snapshot of the fund's work. "I was disappointed," says the official, who asked not to be identified.

Mr. Buchanan acknowledges the limits of the center's research tools and says the center will continue to refine them. "We feel we've made some progress in providing tools that help to answer somewhat those questions" about effectiveness, he says. "There's a lot more we need to do."

To that end, the center is embarking on perhaps its most ambitious — and controversial — project: evaluating foundation "strategy."

As the first step, the group will ask foundation leaders how they define strategy and what measures it should use to create comparable data. The center will ask, "What does it mean to have a strategy? What is it that you use to define what you're really good at?" says Kevin Bolduc, the group's associate director who is working on the strategy-assessment effort.

Nonprofit officials, even those who support the center, are skeptical of this effort.

"The center has bit off an incredibly big challenge in this next phase," says Kathleen P. Enright, executive director of Grantmakers for Effective Organizations, in Washington.

Ms. Enright, who sits on the center's advisory board, questions whether the center can compare the effectiveness of a foundation that, for example, supports education versus one that supports environmental causes.

Mr. Buchanan admits the center's mission is not an easy one. But he says the alternative — never trying to gain an accurate measure of foundation efforts — will hurt grant makers and leave them subject to further questions by lawmakers and others.

"Foundations won't have a persuasive case to make that folks should back off from scrutinizing them until the cases about their own effectiveness and impact can be presented," he says. "Saying you're doing good isn't a compelling statement anymore, because that's the baseline assumption."

Reprinted with the permission of The Chronicle of Philanthropy, http://philanthropy.com.