An alternate, condensed version of this post can be found on the Markets for Good blog.
We all know it. The look of sheer delight that crosses a child’s face the first time he or she toy-hammers that round peg into the round hole. Happiness is the warm blanket of certitude.
As we know, however, the road of life is littered with square pegs. And it can be messy.
That sense of messiness has become more acute in recent years, thanks to the technologically driven deluge of information in which we’re rapidly drowning. Much of that information is increasingly generated in the form of data—most recently, “big data.” The ability of the latter to cut a swath through the quagmire is seductive, particularly when new technologies or tools pop up. Like shiny toys, there’s an understandable eagerness to try them out, particularly when they promise to distill complex concepts into neatly organized patterns, variables, and factors. That, however, can lead to a rush to the toolkit, with nary a glance at what is being studied and why. Data gets collected, en masse, and then churned through some kind of process to aggregate it in various ways, depending largely on who’s asking for it and for what purpose.
That’s certainly been the case in the social sector, which has been eager to figure out how it can use data more efficiently and effectively. The result has been a torrent of data-based systems, platforms, and tools designed to help social sector organizations assess their performance, evaluate their programs, and operationalize their outcomes.
Some see this trend as progress in a sector in which “doing God’s work” was, until recently, enough to substantiate nonprofits’ value.
Others see it as the end of the universe as we know it.
But there are still others who fall somewhere in the middle—those who appreciate rigorous analysis and evidence-based practice but have legitimate questions about where and how data will be used (and is being used) in a sector where outcomes and impact can’t be easily tied up with a big bright bow.
The biggest question is data for what? To help donors make better philanthropic investment decisions? To help nonprofits benchmark their performance across other organizations or subsectors? To help governments assess which intermediaries are achieving their goals? To provide the public with information about these organizations and what they’re doing? To help inform policy debates?
Each of those questions requires serious thinking about which variables will be used in each circumstance and for each constituency, why, for what purpose, and under which assumptions. They also underscore that data aren’t only a bunch of numbers; they’re merely vehicles through which to analyze, contextualize, and apply those numbers in ways that will help practitioners, policymakers, beneficiaries, and investors make better decisions, improve services, or create more responsive programs or legislation.
Wrestling with all that isn’t for the faint of heart, which is why the usual fallback position is to focus on numbers—and the ones that already are available—as proxies for organizational performance and effectiveness. Most of that data derives from the IRS 990 forms, which have serious limitations, not the least of which is their emphasis on financial factors. Information related to policy or practice issues—arguably the things that nonprofits (and investors, for that matter) might find most useful—is still nonexistent or difficult to extract.
We all know of nonprofits doing extraordinary work, but because it’s complex and multi-layered, it’s difficult to measure. Thus, those groups usually don’t show up on the much-touted lists of “high-performing” organizations—often those with bigger budgets, savvy fundraising staff, and boards with more impressive credentials. In other words, they have the stuff that’s easier to count and translates well on paper.
Distilling the panoply of “messy” relational processes that nonprofits use to do their work into measurable variables is difficult because they’re usually trying to resolve “wicked problems” that can’t be definitively described. But that doesn’t mean we shouldn’t keep trying. After all, social scientists have been trying to measure the squishy stuff for decades. Perhaps it’s time that we in the nonprofit sector figured out how to do likewise, specifically, by finding ways to integrate things like personal values, emotional factors, and cultural ethos into the larger equation—a regression model that might shed more light on to why donors give and to what.
That’s a more nuanced approach, admittedly, but it’s one that would also help nonprofits because it not only appreciates accountability but reflects how results can vary depending on circumstances, geography, context, and other fluid variables. This approach also acknowledges that data goes beyond sheer numbers to include qualitative information about how organizations can improve their practice, rather than “prove” whether they simply succeeded or failed.
Developing and testing this more complex model—which has rarely been attempted in a rigorous way—could and should be more of a priority in the social sector. Perhaps a first step would be asking a wide range of organizations and constituencies for detailed and concrete examples of information that would be useful to them, particularly in ways that it can or will be applied toward achieving impact and across various fields (and even entire sectors). Public officials, for example, may like having more information about the number of people using subsidized child care centers, but that tells them little about whether their services are leading to improvements in people’s quality of life or whether their presence has benefited or enhanced the larger community.
In short, collecting data—of all kinds—is one thing, but analyzing it is another and requires smart people to put it into context. What does it mean? And is it applicable in “real life?” Just because we have stacks of outputs showing that counseling “improves people’s lives” it doesn’t mean that policymakers will rush to fund programs so people have access to that service.
The bottom line? Data isn’t knowledge—nor is it necessarily knowledge that matters.
Some might say that we need a more centralized way to share information of all kinds—not just numbers. But that assumes information sharing to be an unfettered good, which isn’t necessarily true. Yelp might offer helpful information, but it’s hardly the place to get reliable information that has some evidence behind it. The mountain of big data being generated is exciting but it’s also generating misinformation that can be (and is already being) manipulated by people and institutions to convey the findings they want. That leads to more echo chambers or naïve trusting of questionable output.
Rather than databases full of disjointed numbers, perhaps what we really need are more trusted and reliable sources of “what works.” That will require a process to thoughtfully evaluate and analyze data and information through some sort of vetting process. Instead of relying on experts, though, perhaps this process could incorporate some of the open source practices that technology is driving by making information public, but pulling in some experts to weigh in on what’s real and what’s noise. After all, research shows that the best decisions are made when “real people” and experts work together.
Another thorny problem that existing databases have yet to crack is how to assess impact. While there’s a lot of discussion about this, it’s been daunting to put into practice. To help move that process along, perhaps we could all agree that focusing on numbers to assess impact won’t do the trick. That’s because impact goes beyond outputs and outcomes to affect something bigger—whether they’re public policies or cultural practices. Some, like Larry McGill of the Foundation Center, say that even those measures may be inadequate to assessing impact, which should focus on “making change.” Even if an organization did play a role in making policy changes, did those changes make any shred of difference?
Getting to those answers will take time, but one thing we could do now is require organizations (including funding institutions) to give concrete examples of how they envision achieving impact and how they’ll operationalize it—beyond a list of outputs the organization itself drummed up. After all, what better measure of impact is there than investments or activities having traction beyond a list of stipulated outcomes organizations or donors wanted to see?
Yes, these are messy issues for messy times, but if we can commit to doubling down on them—rather than seeing data, by itself, as a magic bullet—it would be a huge step forward in advancing our understanding and appreciation of how data—in all its forms—can be more thoughtfully applied and used by all those who care about the social sector. That may never get us to the satisfaction of pounding a round peg in a round hole, but it will make us more comfortable with the square ones that will always be with us.
Cynthia M. Gibson, Ph.D., is an independent consultant for a wide range of national nonprofits and foundations who serves as a strategist, thought leader, and writer. You can find her on Twitter @Cingib.
All good and important points. I’m struck by one omission – “data from whom?” The “for whom” is implied in your question “for what” but a key to addressing many of the deeper analytic questions you raise (which are, of course, key) is thinking hard about “from whom” the data come. Is there feedback data? customer or user data? beneficiary data? If so, what data are being collected, who decides that they’re being collected, and what is being done with them.
This is a good articulation of the “first order” changes that our access to data storage and collection and analysis make possible. As important – perhaps more so – are the second order changes they present regarding ownership, privacy, transparency. Oversimplified, data are valuable. As such, they are tools of power. We need to be designing data collection, use, access, and sharing policies with that frame in mind – remembering at all times that there are 2 very distinct kinds of data at play here – data on people and data on enterprises. People data is rife with possibility and peril for exacerbating or ameliorating some of the very “power issues,” or “wicked problems” that Cindy identifies
Lucy:
Ironically? That question was actually included in a much longer version of a paper I wrote about all of this. Something had to go, and honestly, that was it. But, I wholly agree. A lot of “talk” about “big data,” but the “data” show that most of big data is used by the Dept. of Defense, huge corporations, and marketing firms. But could be another post! As an aside, in terms of the second-order issues you raise, the Pew Project on Internet and American Life has an outstanding paper about the pros and cons of “big data” that get to some of the issues you appropriately raise. I recommend it highly. http://pewinternet.org/Reports/2012/Future-of-Big-Data.aspx
Ah, I think the word you’re looking for is “tragically”
Much to appreciate here as always, Cindy, but the most important piece (imo) is missing and its absence changes the center and trajectory of everything that follows.
Christine:
Thanks. I understand your point. I assumed that the phrase “data for what” encompassed the important issue(s) you raise, i.e., included data for “whom” and “for what purpose” and “who decides.” Etc. Clearly, there is a need for a LOT more discussion about these issues — more than a short blog post would allow — but I think those of us care about these nuances need to start speaking up more about them in these kinds of forums (rather than allow a “rah rah data” mentality to sway the day). This is a start, I hope. More to follow
Thanks for a great post, Cindy. From the perspective that everything we observe or sense is data, then data — big or otherwise — are unquestionably important and can lead to insight, empowerment, and other social goods. Nobody would deny this. And as you and others point out, data require interpretation; investment in their dissemination and collection requires rationalization; etc. The problem with the current philanthropic obsession du jour, in my view, is that it’s one more example of how the field — fueled by big foundation dollars spent in the name of “innovation” — sidesteps the most central and most difficult questions about social change and the role of foundations in helping to create that change. The data about whether foundations are being effective at ending homelessness, to take one example, have been collected for years, are available to all, and require little by way of special expertise to interpret them. More data will not help us address what is primarily a moral rather than a technical failing.
Albert:
As always, I have little to say except, “indeed.” Why do we continually sidestep deeper questions and important issues?
Albert makes a point that should be made more often. For many of our problems, we assume that we need more data when what we really need is more money. That money has to come from new revenue to the nonprofit/public sector or it has to be moved from existing efforts. And that’s the deeper question, really, and it very quickly becomes political in nature. Nobody wants to talk about publicly shaming the foundations who invest in ineffective programs, and they even more wish to avoid a discussion regarding the distribution of government spending. This is changing, some, as more foundations get into advocacy work, but the fact is that the difficult questions surrounding social change are just that, difficult, and most of us don’t like to spend long periods of time dwelling on difficult questions.
Even if we don’t have hard, concrete data regarding large scale outcomes, we often have decades of academic research and small pilot projects that indicate to us which approaches are most effective (think individual development accounts). Sometimes, though, we really do know- if the goal is to get homeless people off the streets, their are multiple approaches that are known to work. We simply need to invest in them.
The other point that isn’t made often enough is that data in the social sector is extremely easy to fabricate, and if we set up data management and implementation efforts incorrectly people will lie, cheat and steal to get or maintain funding. Daniel Stid made this point very effectively here:
http://www.bridgespan.org/Blogs/Measuring-to-Improve/February-2012/What-The-Wire%E2%80%9D-Has-to-Teach-Us-about-Nonprofit-Per.aspx#.ULYhTuRIhWQ
Systematic cheating on standardized tests, crime stats, social services data, it has all been falsified before and will be again. It is very, very easy for people to justify “massaging” the numbers when they feel their jobs are at risk, or when they feel they are having a real impact that simply isn’t reflected on a spreadsheet. It is absolutely critical that we use data in the right way- and a big part of that is ensuring front line workers feel that data is used to support the mission, rather than used to punish those who are trying incredibly hard to have a positive social impact in a social and political environment that is challenging enough as it is.
Pingback: Data: More Than Numbers (Or The Sum of Its Parts)
A good and welcome piece and useful comments – thank you Cindy. As others have implied, figuring out how different forms of knowledge and information connect to structures of power and interest in society is the key task, so I hope CEP will take on that challenge by organizing one of its conferences around the subject and facilitating an honest, open and equal conversation. That would require a different kind of conference with different participants but it could be done, and I think might mark a step-change in the debate. But the challenge isn’t a new one and it is obviously isn’t limited to philanthropy, so I hope we don’t try to reinvent the wheel. If you haven’t read them I’d recommend books like Nicholas Maxwell’s “From Knowledge to Wisdom” (published in 1984) and “The Construction of Social Reality” by John Searle (which came out in 1995), which helped me enormously when I was struggling with this same set of issues around international development and foreign aid in a previous life. And there are efforts in that community today which we should connect with, like http://www.bigpushforward.net who are organizing an international conference on the “politics of evidence” next year. Perhaps those efforts could be part of what CEP does and vice versa. In fact, the more the better in the struggle to develop a revolutionary social science.
Michael, I appreciate your comment very much. We at CEP are very interested in that conversation and in looking at the issues honestly and from all sides. I would love to discuss this further with you.
I think Cindy raises crucial questions in her post. Although we at CEP hold the view that we need more and better data to inform more effective philanthropy, I also could not agree more with her statement that “collecting data — of all kinds — is one thing, but analyzing it is another and requires smart people to put it into context. What does it mean? And is it applicable in ‘real life?’”
The smart application of data in philanthropy and the nonprofit sector is very complicated. Yet many deny this — reaching for over-simplified and unhelpful analogies to business and the financial markets (as you know all too well, Michael, because you have fought so eloquently agaist it!). (An example of that here: http://www.marketsforgood.org/from-raw-data-to-informed-decisions-what-we-can-learn-from-the-financial-sector/)
I would also note, to Lucy’s point about power, that much of the data we at CEP have focused on collecting has the potential to alter power dynamics in an important way. When funders collect data on grantee experiences through the Grantee Perception Report, and we present the results to a foundation board, it raises the voices of those on the front lines in ways that have led to real change in how foundations operate. Similarly, our YouthTruth effort is bringing the voice of those who should matter most — those whose lives we are trying to improve — front and center. Too many times, the voices of intended beneficiaries are not heard in the discussion of what will help them. Yet who would have a more relevant view?
My point is this: the right data can serve to empower those whose voices are too frequently ignored.
Thanks to Cindy and to all those who have commented here so thoughtfully.
Phil Buchanan
CEP
tks Phil, happy to talk. You make good points, but as always there’s another layer of questions underneath that need to be surfaced about “the right data” – what constitutes ‘legitimate’ knowledge, who decides, and how knowledge, belief, action, agency and politics intersect. But that’s all the more reason to have the conversation. I may be able to help in a small way since I am launching a new web-zine on Transformation next year which will provide a venue for deeper conversations like this on politics, economics and activism. The real-time conversation could be mirrored on the web-site to get to more people. Just a thought.
Also, it is hard to argue with this:
http://www.economist.com/blogs/democracyinamerica/2012/11/crime?fsrc=rss
“Lots of things in life, maybe most things, often the most important things, don’t have explanations that can be packaged as a simple, coherent thesis. Second, given our inability to explain definitively why the crime rate is falling, we may need some scepticism about the recent push to demand scientifically valid evidence for the effectiveness of social betterment programmes.”
Well said. And, yes, it’s interesting that some of the folks investing in things like social impact bonds — many of whom are demanding data/outcome measures from the nonprofits that administer these kinds of “social betterment programs” — are doing so with little or no data behind this concept.
The cacophony of flatulent chatter about the promise of data and information in the social sector has reached the din of a million nails on a giant cosmic chalkboard.
I can’t thank you enough for offering us this tranquil moment of quiet reflection.
Today I can skip the dose of Xanax that I take after reading my Twitter feed.
I loved your opening analogy of the childish satisfaction of hammering the peg into the hole. Humans are wired for pattern matching and interpolation. It is a primal urge that begs to be satisfied even at the risk of miscalculation.
Our data-fetishism and techno-narcissism may be in our DNA.
So rather than offering my usual rant on the subject, I’ll give four examples of why I think we are evolving to become smarter about this stuff:
–More people are skeptical that information that we construct with our latest Big-Data Whatsis and blinking data dashboards is such a powerful driver in making decisions.
–Learning is emerging as more of a central theme in the field of monitoring, evaluation, and outcomes measurement.
–Complexity theorists and allied social science wonks are doing a much better job at nudging us about our chronic Cartesian-Anxiety. They are fantatical about knowledge and information but humble about its limits in helping us understand social phenomenon.
–The social sector is slowly rising up to address the big fat question of power – information for whom, and by whom. Thanks for reminding us, Lucy! Yes, its an ancient debate still relegated to the margins, but I am tickled that community-based research, action research, participatory research or whatever you want to call it is a vibrant and maturing field.
As Michael Edward’s points out, this discussion of the politics of knowledge has a lengthy bibliography well worth skimming before anyone gets too excited about building our next round of data toys.
This is a very good piece that raises all the right questions. My concern these days is the constant drumbeat of interest in metrics without careful validation of the value and manipulation of the metrics for self promotion. Honestly, I’d settle for a good theory of change from most organizations, and follow with better research from the funders. Unfortunately, most funders are either all about measures from their grateful grantees (heaven forbid that they ever ask about the applicants they turned down), or unwilling to invest in the deep research that might confirm the links between interventions and outcomes. Research is so yesterday in the philanthropic world–it’s all go, go around immediate impact. Been there, done that in government forever, and still have nothing to show for it beyond highly politicized measures. That’s what’s happening here, only we don’t call it politicization. It’s impact investing.
For what it’s worth, CEP has surveyed the declined applicants of dozens of foundations through our Applicant Perception Report. So, in fact, there are funders that care about the perspectives of those they decline. Maybe not enough funders, but they are out there.
http://www.effectivephilanthropy.org/index.php?page=apr-subscribers
Cindy – thanks for a very thoughtful blog which stimulated such an important ensuing discussion. What strikes me is that we’re on a spiral (hopefully). With all the appropriate cautions (for whom? by whom? to what purpose? etc.) and without getting into a field of dreams, it is true that many things we will not know unless we try them – i.e. we might not understand what benefit we’ll get from the data until we actually collect it and try to apply it. So – a slow, cautious, conscientious approach, step-by-step and asking hard questions all along the way – I like that. So that we avoid doing this data collection adn information system building in a way that reminds me of the joke about male dogs (“Answer – because they can.”)
Thanks for this blog, and for the stream of comments it has yielded. In my experience, problems in measurement often result from not being clear on the problem that one is trying to solve, not being realistic about what it takes to get there, and not adapting and getting better over time. Ironically, too-rigid theories of change or measurement systems can compound the problem.
In my experience, the most successful measurement systems are created in the context of adaptive strategies–defining success, assessing whether you’re making progress towards that goal, learning from what you and others have and have not been able to accomplish, and then adapting your strategy and approach accordingly based on what you’ve learned. It’s not about measurement in the abstract.
This is not a complex concept, but it does take hard work and persistence. And, things that take hard work and persistence are not as sexy as new fads. So I applaud CEP as usual for breaking it down for us…and keeping it practical!
Thanks, Susan! All credit to Cindy for these thoughts and for guest blogging for us. Open invitation to you to do so also.
Pingback: NYT – Family Foundations Prepare for the Next Generation | Rusty Stahl