Crowdsourcing Yourself?

Image via Wikipedia

The jellybean jar experiment is a relatively well known study that shows a rather surprising fact. The experiment itself involves asking a room full of people to estimate the number of jelly beans in a jar. Each person comes up with their own heuristic for making an estimate. For example, one person might come up with a guess for how many beans would make up a single layer at the bottom of the jar and then multiply by the number of layers they think would fit. Another person might have a completely different way of arriving at an estimate. Despite the fact that many people are way way off in their guesses, if you take the average of all the guesses you'll get an answer that is shockingly close to the correct one. Interestingly though, if you allow the people in the room to discuss their methods before the estimates, the average of their answers will get worse.

The reason this happens is because each person's original guesses are independent - they may not be good guesses, but the error they introduce into the average is unique. However, when you let them discuss, some will be convinced to use another person's technique. When this happens the errors no longer cancel each other out, they end up reinforcing each other and dragging the average away from the true answer.

This is a pretty interesting result when you think about how financial markets work and the role of security analysts in estimating the value of companies. Do our estimates of the stock prices get worse when a new report comes out? One person's opinion (albeit a much more informed person) is influencing many many people, whos individual trades will determine the actual price of the stock.

Anyway, the reason I bring the study up is because recently some psychologists also thought it was pretty neat. In a study described in Scientific American they wanted to see if they could reproduce the effect in a single persons brain. Two groups made a few estimations of a date in history (like the day the Magna Carta was signed). One group made a series of guesses, one after the other. The second group made several guesses, but had to write down reasons their earlier guesses might have been wrong and to use those reasons to form a new guess. And while their secondary guesses were no more accurate than their first, the average of their guesses turned out to be much better! This is in contrast to the first group, who's averages were not as good.

So it seems that we can harness the power of independent errors to make our own individual estimates more accurate, we just have to act like we have multiple personality disorder. For me this conclusion raised a lot of interesting questions, and I hope the psychologists try to tackle some of them.

• If a person has several facts relevant to a situation, would they make better choices if they try to consider all of the information and try to synthesize it into a single answer (the way we're all familiar with) or would they be better off making a series of estimations based on each individual fact and then averaging them together?
• What other kinds of problems is this method better at answering?
• Should we eliminate debate? Have members of congress vote on things without discussing them?
• The next time I want to count my jelly beans should I just pay a hundred people on the Mechanical Turk service to look at a picture of it and guess?