Archive for August, 2011

Education, Philanthropy and Metrics

Tuesday, August 16th, 2011

In a recent interview with Black Enterprise magazine highlighted on the Foundation’s blog, Bill & Melinda Gates Foundation U.S. Programs President Allan Golston discusses the

“[H]uge inequity in this country … [Y]ou can live in a different state, or different zip code, and that [is] … a determinate of what type of education you’re getting through what you’re expected to know. So, one goal is to level that out so we have high expectations regardless of what state, what zip code, what school district you’re in.

The second goal is to have a great teacher for every student. So, [it] shouldn’t matter if you’re in an urban district or a rural district, or you’re in Massachusetts or Mississippi. [For] every student, particularly the ones that don’t get access to a high quality education, which are typically students that are low income, minority, we’ve got to solve that problem so that there is a great teacher for every kid in every classroom.”

Golston is absolutely right, of course, about the importance of these goals. But how do we ensure that the current efforts of the big foundations and philanthropists to improve educational opportunities and outcomes for poor kids don’t meet the same fate as failed efforts of the past?

One part of that challenge will be getting the right measures in place. Golston notes in the interview, “We do think that while testing can’t be the only measure it has to be a component of it.”

That makes sense. But, the fact is, today, many act as if test scores are the only measure. That can lead to some very negative unintended consequences. This is part of the argument Fay Twersky, Valerie Threlfall and I make in an Atlanta Journal-Constitution op ed, published last week.

“The Atlanta Public Schools cheating scandal illustrates the peril of focusing on a single performance measure to the exclusion of all other data and should open the eyes of school leaders and policymakers to the need for a broader set of indicators of school quality.

What the school system created was a focus on test scores so intense that seating of principals at an annual meeting was based on how their schools fared, with high-scorers getting prime seating and low-scorers forced to stand in the back. The message was clear: Deliver high test scores or be humiliated. So, as Superintendent Beverly Hall collected national accolades and hundreds of thousands of dollars in performance bonuses, principals and teachers engaged in rampant cheating.”

In the op ed, we argue for complementing test scores with a variety of other indicators, including student perceptions gathered through our YouthTruth initiative – which received its start-up funding from the Gates Foundation. Through YouthTruth, students precisely like the ones Golston describes can weigh in – and they have a lot to say. (A majority of the students surveyed through YouthTruth are eligible to receive free or reduced price lunch, and two thirds of YouthTruth respondents are students of color.)

Students, after all, may have the best seats in the house to judge the quality of their education. Teachers and parents have invaluable perspectives, too.

Yet, today, it seems all we hear about is test scores. That’s because of the emphasis placed on them by the 2001 No Child Left Behind legislation. It’s also because some reformers seem to want to pretend that assessing educational quality is as simple as gauging company profits.

But that’s not so, and foundations working in education have an opportunity to counter this by tying their support for school reform efforts to a focus on a more well-rounded set of metrics. Foundations, in other words, can be an important counterweight to the federally mandated over-emphasis on test scores.

Of all institutions, foundations should understand that a single metric is insufficient. Just as their own performance cannot be boiled down to a single “Social Return on Investment” calculation – because gauging foundation performance requires reviewing a range of indicators – neither should schools be judged solely on students’ performance on (often flawed) standardized tests.

 

Phil Buchanan is President of CEP.

The Case for Looking at More Than Just Test Scores

Wednesday, August 10th, 2011

Phil Buchanan, Fay Twersky, and Valerie Threlfall highlight the work of the YouthTruth Initiative and argue the case for the importance of using student perceptions to gauge school performance.

This piece originally appeared as an op ed in the Atlanta Journal-Constitution on August 10. A similar op ed appeared August 9 in thePhiladelphia Inquirer.

The Atlanta Public Schools cheating scandal illustrates the peril of focusing on a single performance measure to the exclusion of all other data and should open the eyes of school leaders and policymakers to the need for a broader set of indicators of school quality.

What the school system created was a focus on test scores so intense that seating of principals at an annual meeting was based on how their schools fared, with high-scorers getting prime seating and low-scorers forced to stand in the back. The message was clear: Deliver high test scores or be humiliated. So, as Superintendent Beverly Hall collected national accolades and hundreds of thousands of dollars in performance bonuses, principals and teachers engaged in rampant cheating.

The lesson here is not that tests don’t matter. They do and they should. It’s that they should not be positioned as the one and only measure of school quality. Today, unfortunately, they often are — as a matter of federal policy.

The intentions of those who have pushed for greater accountability in our education system are good ones. But the very students who were supposed to benefit from reform efforts are now collateral damage in scandals such as Atlanta’s — with teachers erasing students’ incorrect answers and putting in the right ones rather than focusing on the much harder task of helping those students learn.

Just as in business, in which a focus on short-term stock appreciation and quarterly profits has resulted time and time again in fraud and accounting scandals, the over-emphasis on test scores — with careers, tenure decisions and compensation made or broken by the results — has led to cheating that has disgraced school systems once held up as exemplars. Sadly, when jobs or money are on the line, people will try to game the system.

One powerful way to reduce the chances of scandal and dishonesty is to create a well-rounded set of indicators of school quality that are reviewed together — to form a more complete understanding of school performance. A robust, multifaceted approach to assessment renders gaming much more difficult.

So what other data should be tapped to complement test scores? Here’s a radical idea for one additional source of data that, today, is largely ignored: students themselves.

Students have a valuable perspective on their educations, yet remarkably little attention is paid to their voices by those in power.

Four years ago, with support from the Bill & Melinda Gates Foundation, we created YouthTruth, a survey program that rigorously captures confidential student perceptions of their high school experience – and then puts their ratings in a comparative context. What we have seen is that when you ask the right questions, when students believe the process is genuine and that administrators will listen to what they have to say, and when you play back the results to the schools and the students, new insights emerge about educational quality.

Through this work, we have seen student survey data fuel dramatic change. One high school revamped its approach to grading and assignments and created new capstone programs for seniors in response to survey data suggesting that students were not being sufficiently challenged. Another school retooled its discipline process, instituting a system for adjudication that involved students themselves and resulted in a better culture for learning. A third school revamped its college advising and mentoring program in response to student feedback.

In addition to providing fodder for important real-time improvements, student survey data could help vigilant leaders and policymakers detect cheating early on. In the Gates Foundation’s Measures of Effective Teaching study, Ron Ferguson of Harvard University has demonstrated empirically that student perceptions of teacher quality and student academic achievement are correlated. So a disconnect between survey results and test scores could be a kind of early warning system, the presence of which in itself also would act as a deterrent to those who would consider cheating.

Students are just one source of potentially powerful data. For example, many districts survey parents, but decision-makers rarely pay attention to the results. Teachers themselves are another key source of data on what’s really happening within the school walls: They, too, should be surveyed regularly — in a way that protects confidentiality to ensure candor — with results making their way to district leaders and policymakers. Some districts do this today, too, but these data are often overlooked in the obsessive, legislated focus on test results.

Perceptual surveys have their limitations, of course, but offering key constituencies — starting with students — the opportunity to share their views on what’s really happening with those in power would be a significant step in the right direction. Ultimately, unless we change policy to ensure that schools and their leaders are not judged by test results alone, we’re destined to read about more scandals like the epic one in Atlanta.

Phil Buchanan is president of the Center for Effective Philanthropy.

Valerie Threlfall is director of YouthTruth, an initiative of the Center for Effective Philanthropy.

Fay Twersky is senior adviser at the Yad Hanadiv Foundation and previously served as director of Impact Planning and Improvement at the Bill & Melinda Gates Foundation.

 

David Trueblood is Vice President — Communications and Programming at CEP.

Gates Annual Report Highlights CEP Role in Increasing Global Impact

Friday, August 5th, 2011

The power of CEP’s Grantee Perception Report (GPR) received a strong endorsement this week from Jeff Raikes, President of the Bill & Melinda Gates Foundation. A widely published AP news story sparked by the giant foundation’s newly released annual report focused on the challenge of getting honest and useful feedback from grantees—and how effective CEP’s tools have been to that end. You can see the article online or in a host of newspapers, or click here.

In an interview with AP reporter Donna Gordon Blankinship, Raikes made the connection between the survey and need for his organization to be more in sync with grantees around the world.

“If we can work more effectively with our grantees, that will increase the impact that we aspire to,” Raikes said, according to Blankinship. 

After talking with Jeff Raikes, Blankinship interviewed Phil Buchanan at CEP, who talked about the ongoing need for foundations to develop credible and objective information about the relationship between grantee and funder.

You can see an overview of the Grantee Perception Report here.

David Trueblood is vice president — communications and programming at the Center for Effective Philanthropy.