Open For Questions – A Critique of Idea Sharing Sites
President-elect Obama’s web site change.gov had a site (“open for questions” ), in the early weeks of his adminstration, where people can submit and vote on questions, and where he promised to answer the top 5 questions in each category. Apparently a journalist encouraged readers to vote for one of the questions (on appointing a special prosecutor to look at Bush’s actions), which resulting in the question going for #6 to #1 in the foreign policy category. Change.gov seemed pretty successful in some ways. It’s a nice example of the Internet allowing a whole new set of players to deeply influence the political debate at a national level. The special prosecutor question was dead in the mainstream media. The questions that rose to the top seem substantive and important and have substantial legitimacy because of how active the voting was (100,000 voters, 70,000 questions, 4 million votes). One can argue that spreading power through society, more broadly than was possible before the Internet, is a great thing. It seems clear that excessive concentration of power in a few individuals and institutions (corporations, executive branch) has been at the root of some pretty serious problems in the States. Idea sharing sites have also been used, with apparent success, in other contexts. The Dell ideastorm site, for example, includes over 11,000 ideas (for Dell products and services) that have received a total of over 680,000 ratings and 84,000 comments. A more recent White House initiative (Open Government Dialogue) has gathered thousands of ideas and tens of thousands of votes. And the google 10 to the 100th project generated over 150,000 submissions on innovations to make the world better.
There are some important problems, however, with such sites. One is redundancy. In all these site, a large proportion of the questions are minor variants of each other. Often, when some kind of political action is in play, such sites can attract many submissions that make the same point (see, for example, Obama, Inviting Ideas Online, Finds a Few on the Fringe – NY Times). The many redundant posts has the effect of “crowding out” the diverse range of ideas the sites hope to gain, making it harder to identify the best ideas. When there are thousands of posts submitted, manually pruning this list to consolidate equivalent posts is a massive undertaking. The 10 to the 100th project, for example, handled that problem by recruiting 3,000 google employees to filter and consolidate the 150,000 ideas generated in a process that but them 9 months behind their intended schedule. Finally, most people are only likely to look at the most highly rated posts. The result is that the vast majority of the submitted ideas are lost to obscurity.
Another issue involves dysfunctional ranking outcomes. The Better World Campaign used an idea sharing system with user to come up with a proposal for Obama’s first actions upon entering office. After a year of debate, over 5,000 ideas submitted, and close to one hundred thousand votes, the result was the proposal that Obama plant an organic garden at the White House. That vast investment of collective effort resulted in selecting a tiny, purely symbolic, gesture. Or look at the ideas list collected at http://www.change.org/ideas/. Is legalizing marijuana and other drugs really the top priority for this apparently serious and highly diverse (judging from the list of sponsoring organizations) set of voters? One possible explanation for these effects is self-reinforcing ratings. When people are asked to rate a very long list of items sorted by their current average rating, one can expect that the system will quickly “lock” into a fairly static, and arbitrary, ranking. People are more likely to vote for ideas that are already higher in the list, since if there are thousands of ideas, people will in all likelihood stop looking after the first few. So the first few winners take all, even if they are inferior to many other ideas in the list. Duncan Watts observed this property in looking at music rankings (Matthew J. Salganik, Peter Sheridan Dodds, Duncan J. Watt (2006). “Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market” Science 311:5762, pps 854-856): when people could see each other’s rankings before rating, much of the ranking results were essentially random. Another possible effect is that a post with focused voting may beat out a more important post that had it’s votes spread over many redundant instantiations.
A final issue is depth vs breadth. These idea-sharing sites tended to come up with many fairly simple ideas. The ideas generated by the google project, for example, (e.g. make government more transparent, help social entrepreneurs, support public transport, create user-generated news services) are pretty generic and unsurprising and are already being actively pursued in many contexts. While just getting people to participate has its own value, no doubt, was it really worth all that massive effort to come up with ideas that a few experts could have brainstormed in a few minutes? Surely that amount of effort could have been used to compose a smaller number of more deeply-considered ideas, rather than many shallow ones. Idea-sharing sites, however, provide little or no support (and incentive) for this.
These weaknesses with current idea sharing techniques suggest that there is an important need for better methods and tools. A scheme to consolidate redundant questions should help. The problem is probably less technical than social, since computers aren’t very good at detecting this kind of semantic redundancy. The challenge is to evolve rules that govern human filtering that work as well as the rules that govern editing practices on Wikipedia. A argumentation system, especially if we include a rating/aggregation scheme for arguments, could tend to encourage rankings that are based in reason rather than just being “popularity contests” driven by a few influential bloggers. It should also allow people to collaboratively develop more elaborated ideas, since an idea can be linked to issues that ask how to refine some part of the idea, which can in turn be linked to other ideas. Another possible approach is to make sure people are not be able to see rankings until *after* they vote. One trick, proposed by MIT Prof. David Karger is that people be given a randomly selected set of ideas to rank, so each has an equal opportunity to get attention. This will probably only work, however, if the set of ideas is small enough relative to the number of viewers. Otherwise, most ideas will get too few viewings, and ratings, to represent a good picture of the “wisdom of the crowds”. If we had 4 million votes spread randomly over the 70,000 questions submitted to change.gov, for example, then each question an average of only about 60 views. That hardly seems adequate to get a good sense of what a nation’s priorities are.