[Welcome Scott E. Page, and Host, Cosma Shalizi - bev]

The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies 

scott-page-the-difference.thumbnail.JPG

Scott Page is a professor of political science and economics at the University of Michigan, where he’s also the associate director of the Center for the Study of Complex Systems, and external faculty at the Santa Fe Institute. (Disclosure: I worked for Scott when I was a post-doctoral fellow at the Center, and I’m also external faculty at SFI, so take my enthusiasm with salt to taste.) Scott’s written lots of academic papers, and co-written a textbook, but The Difference is, well, different. It’s a serious, but also playful, look at the power and virtues of diversity when it comes to solving difficult problems. It draws together many insights from many different academic disciplines, without requiring any special knowledge of its readers, just willingness to stretch their minds a little.

All very well, you say, but it’s a lot more abstract than most books which show up here: why should you spend your time reading about heuristics and preference aggregation and so forth? For two reasons: to help us persuade others, and to acquire tools for us to use ourselves.

Most progressives have embraced diversity as a value, but there seem to be competing considerations. A common objection — my guess is it’s even often sincere — is that we shouldn’t care how diverse the people who do X are, but just how good they are at X. If those who have discovered and developed their abilities to do X happen to be mostly privileged, that’s just part of what "privilege" means, and the thing to worry about is unfair privilege, not lack of diversity. This is a general objection, and it calls for an equally general, that means abstract, rebuttal.

Implicitly, the objection assumes that there’s a best way to do X, that there’s One Right Answer, and that the best results, the closest approach to the optimum, will always come from the person who’s best at the job. Some advocates of diversity deny the first premise, the One Right Answer bit. This leads to tedious arguments about relativism and other reminders of the unfun parts of the 1990s. Page doesn’t go there; instead he shows that the other premise is wrong, because it ignores complexity.

The problems we face in the real world are hard. There are always an immense array of variables which might be relevant, and an at least equally huge number of ways we might respond to them, plus overlapping constraints on our actions. Moves which seem to help part of the problem typically screw up others, because everything is interdependent, everything is connected to everything else. Even when problems are (more or less artificially) tidied up into well-defined mathematical tasks of locate-the-optimum, it is, generically, quite intractable to actually find that One Best Answer.

The Difference draws on about half a century of research into how to deal with complex, intractable problems. Faced with an overwhelming array of considerations that could be relevant, people assume perspectives from which some variables are visible and the rest are hidden. Within those perspectives, they apply heuristics, approximate rules for getting from where you are now to someplace that is usually better, though not guaranteed to be the best. A heuristic for a problem will do better or worse on different instances of the problem, and different heuristics will do better or worse on the same instance. One heuristic may do better than another on most instances, which makes the former stronger; a weak heuristic does only a little better, on average, than a random guess.

The key insight is that a single strong heuristic will do worse than a collective of individually weak but diverse heuristics. The problems are too hard for any one heuristic to solve perfectly, but the diverse heuristics can, so to speak, cover each others’ weaknesses and help each other out when they get stuck; a single strong heuristic can’t. A collection of diverse strong heuristics would be even better, but the strong heuristics for a problem tend to be similar to each other, so a group of them lacks diversity. In problem solving and prediction, diversity is exactly as important as individual ability.

This is the answer to "why do we need diverse X-ers instead of just the best X-ers?": if there’s going to be one person who does X and they’ll solve all their problems by their lonesome, sure, go for the one who’s individually best at it, the one with the strongest heuristics. Meanwhile, on Planet Earth, everyone works in teams and groups and networks, and even if all you want is the very best job of X-ing possible, you have no reason to ignore diversity and every reason to seek it out, foster it, and make sure that all voices are heard and aren’t just token presences.

I doubt it’s escaped the reader that the problems facing progressives (how to we get universal health care? how do we keep our country from getting into more stupid wars?) are full of the kind of difficulty and complexity I described. Which means it’s no good hoping for an inspired leader to figure out how to do them, or even a small clique of super-geniuses… There are lots of us, from many backgrounds, and we’re self-selected to agree, mostly, on what we want to happen. So there should ways to use our internal diversity, our range of perspectives and approaches, to better accomplish progressive goals. But before we can do that we need to understand how and when diversity improves problem-solving capacity, and we really need to think about the ways in which we can bring that diversity to bear on our common problems — which is the whole "better societies" part of the book.

OK, with that, let me open up the comments.