Open For Questions – A Critique of Idea Sharing Sites

President-elect Obama’s web site change.gov had a site (“open for questions” ), in the early weeks of his adminstration,  where people can submit and vote on questions, and where he promised to answer the top 5 questions in each category. Apparently a journalist encouraged readers to vote for one of the questions (on appointing a special prosecutor to look at Bush’s actions), which resulting in the question going for #6 to #1 in the foreign policy category. Change.gov seemed pretty successful in some ways. It’s a nice example of the Internet allowing a whole new set of players to deeply influence the political debate at a national level. The special prosecutor question was dead in the mainstream media. The questions that rose to the top seem substantive and important and have substantial legitimacy because of how active the voting was (100,000 voters, 70,000 questions, 4 million votes). One can argue that spreading power through society, more broadly than was possible before the Internet, is a great thing. It seems clear that excessive concentration of power in a few individuals and institutions (corporations, executive branch) has been at the root of some pretty serious problems in the States. Idea sharing sites have also been used, with apparent success, in other contexts. The Dell ideastorm site, for example, includes over 11,000 ideas (for Dell products and services) that have received a total of over 680,000 ratings and 84,000 comments. A more recent White House initiative (Open Government Dialogue) has gathered thousands of ideas and tens of thousands of votes. And the google 10 to the 100th project generated over 150,000 submissions on innovations to make the world better.

There are some important problems, however, with such sites. One is redundancy. In all these site, a large proportion of the questions are minor variants of each other. Often, when some kind of political action is in play, such sites can attract many submissions that make the same point (see, for example, Obama, Inviting Ideas Online, Finds a Few on the Fringe – NY Times). The many redundant posts has the effect of “crowding out” the diverse range of ideas the sites hope to gain, making it harder to identify the best ideas.  When there are thousands of posts submitted, manually pruning this list to consolidate equivalent posts is a massive undertaking. The 10 to the 100th project, for example, handled that problem by recruiting 3,000 google employees to filter and consolidate the 150,000 ideas generated in a process that but them 9 months behind their intended schedule. Finally, most people are only likely to look at the most highly rated posts. The result is that the vast majority of the submitted ideas are lost to obscurity.

Another issue involves dysfunctional ranking outcomes. The Better World Campaign used an idea sharing system with user  to come up with a proposal for Obama’s first actions upon entering office. After a year of debate, over 5,000 ideas submitted, and close to one hundred thousand votes, the result was the proposal that Obama plant an organic garden at the White House. That vast investment of collective effort resulted in selecting a tiny, purely symbolic, gesture. Or look at the ideas list collected at http://www.change.org/ideas/. Is legalizing marijuana and other drugs really the top priority for this apparently serious and highly diverse (judging from the list of sponsoring organizations) set of voters? One possible explanation for these effects is self-reinforcing ratings. When people are asked to rate a very long list of items sorted by their current average rating, one can expect that the system will quickly “lock” into a fairly static, and arbitrary, ranking. People are more likely to vote for ideas that are already higher in the list, since if there are thousands of ideas, people will in all likelihood stop looking after the first few. So the first few winners take all, even if they are inferior to many other ideas in the list. Duncan Watts observed this property in looking at music rankings (Matthew J. Salganik, Peter Sheridan Dodds, Duncan J. Watt (2006). “Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market” Science 311:5762, pps 854-856): when people could see each other’s rankings before rating, much of the ranking results were essentially random. Another possible effect is that a post with focused voting may beat out a more important post that had it’s votes spread over many redundant instantiations.

A final issue is depth vs breadth. These idea-sharing sites tended to come up with many fairly simple ideas. The ideas generated by the google project, for example, (e.g. make government more transparent, help social entrepreneurs, support public transport, create user-generated news services) are pretty generic and unsurprising and are already being actively pursued in many contexts. While  just getting people to participate has its own value, no doubt, was it really worth all that massive effort to come up with ideas that a few experts could have brainstormed in a few minutes? Surely that amount of effort could have been used to compose a smaller number of more deeply-considered ideas, rather than many shallow ones. Idea-sharing sites, however, provide little or no support (and incentive) for this.

These weaknesses with current idea sharing techniques suggest that there is an important need for better methods and tools. A scheme to consolidate redundant questions should help. The problem is probably less technical than social, since computers aren’t very good at detecting this kind of semantic redundancy. The challenge is to evolve rules that govern human filtering that work as well as the rules that govern editing practices on Wikipedia. A argumentation system, especially if we include a rating/aggregation scheme for arguments, could tend to encourage rankings that are based in reason rather than just being “popularity contests” driven by a few influential bloggers. It should also allow people to collaboratively develop more elaborated ideas, since an idea can be linked to issues that ask how to refine some part of the idea, which can in turn be linked to other ideas. Another possible approach is to make sure people are not be able to see rankings until *after* they vote. One trick, proposed by MIT Prof. David Karger is that people be given a randomly selected set of ideas to rank, so each has an equal opportunity to get attention. This will probably only work, however, if the set of ideas is small enough relative to the number of viewers. Otherwise, most ideas will get too few viewings, and ratings, to represent a good picture of the “wisdom of the crowds”. If we had 4 million votes spread randomly over the 70,000 questions submitted to change.gov, for example, then each question an average of only about 60 views. That hardly seems adequate to get a good sense of what a nation’s priorities are.

Advertisements
Explore posts in the same categories: Uncategorized

10 Comments on “Open For Questions – A Critique of Idea Sharing Sites”

  1. Michael Turner Says:

    “A scheme to consolidate redundant questions should help. The problem is probably less technical than social, since computers aren’t very good at detecting this kind of semantic redundancy.”

    Pardon me if this has already been addressed in research, but … how about making people rank similarities of existing questions the “cost of entry” for posing a question at all? Their own question might be used as a source of keywords for searching the question database to come up with questions to compare. In the process you

    (1) filter out people averse to doing a little work
    (2) cause some people to see that their question has already been asked (assuming that the search mechanism I propose above is reasonably effective at turning these up)
    (3) defer the issue of more automated semantic comparison

    I’d filter further and require that people choose a category of subject and pass a brief reading comprehension test for short passages about that subject, before proceeding to question equivalence ranking, then finally submitting their own questions. But I’m a stinking intellectual elitist. 😉

    • klein1960 Says:

      The idea of asking users to verify their post is a novel addition to the the idea base is a good one, I think. I use it in the “large-scale deliberation” system I have been developing (the Deliberatorium – see http://cci.mit.edu/klein/selected.html for details). In that system, authors are asked to “unbundle” their contribution into individual issues, ideas, and arguments, and then add each one to an “argument map” IF it does not already appear. In the current implementation, a moderator corps is asked to verify that new posts are in fact non-redundant, though one could imagine a crowd-sourced approach to achieve the same vetting process.

    • Mary Says:

      Check into the way that the dating website OKCupid allows for users to ask their own questions. Their database of questions is in the thousands. However, before posting your own question, you type in the subject. Then dozens of other questions pop up (from an automated semantic comparison). To continue, you are even asked again to ensure your question (or similar form of the question) hasn’t already been asked…

      • klein1960 Says:

        Thanks for the pointer. Others sites do that too, including quota and stackoverflow.

      • Michael Turner Says:

        Thanks, both of you. I can perhaps be forgiven for not knowing how OKCupid does its thing, since I’m married. But since I’m also an erstwhile/still-wanna-be hacker, my ignorance of stackoverflow’s system is inexcusable!


  2. […] are displayed according to “current consensus” or popularity. This will result in what Mark Klein over at MIT calls dysfunctional ranking outcomes: When people are asked to rate a very long list of […]


  3. I found it to be a great analysis. Do you think above mentioned issues still exist at today’s date ?

    • klein1960 Says:

      Yes, from what I’ve seen, large organizations are still using the same kind of idea sharing systems, and thus encountering the scaling problems I described in the post.

      • Michael Turner Says:

        What’s probably missing, more than anything else, is a way to rapidly deploy the crowdsourcing and get the responsiveness up to something like real-time. I’ve put a little time into Amazon Mechanical Turk, on the user side. I’m now interested in learning the API. There’s some recent research on near-real-time crowd response, although it depends on alerting your potential crowdworkers beforehand (perhaps on the order of minutes.)

  4. ex back fast Says:

    What’s Taking place i am new to this, I stumbled upon this I have discovered It positively useful and it has helped me out loads. I’m
    hoping to give a contribution & assist other users like its aided me.

    Good job.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: