The article touches on it, but the clear differentiator I see here is whether the "crowd" has expert knowledge of the domain.
If it does, then the noise of experts clashing (resolving conflicts between equally informed ideas) starts outweighing the benefits of them contributing the ideas to begin with. Basically you get a too many chiefs issue.
If it doesn't, you're seeking to average variably-educated wild ass guesses and taking advantage of the fact that WAGs have a tendency to form a bell curve around the actual answer if there's any ability to estimate whatsoever. Since there's otherwise no special knowledge, that ends up being the best you can get but requires a lot of people to get right.
Without having read up on it, my guess is the latter is related to central limit, with each person's WAG forming the first term of the mean of means; a WAG is itself sampling your own mental model and boiling it to a number. Given enough of those you get a normal that you can average.
Another dimension not directly addressed is whether or not the question at hand is technical and/or has a very specific correct solution or response.
A failure I've noted over the past 20 years or so of various collaborative filtering systems, where some function of votes on what is "better" are used, is that many of them degrade essentially to a popularity contest. This can be good if you're trying to optimise for "what will sell the most", but doesn't end up so well if you're looking for "what's the most correct" or "what has the greatest truth value" from among options.
It's commonly noted as the hivemind effect, in a negative sense, particularly where the hivemind (or the effectively expressed result of it) tends to quash correct-but-unpopular views.
RAND corporation once developed a procedure to get a result from a group of experts. They introduced an intermediator, a sufficiently educated person whose job is to prod experts, ask the questions (often redirecting issues raised by other experts) and in the end produce a summarized overview of their positions.
this may be why the prediction markets were so terrible at guessing brexit. Each participant was a kind of expert in said domain - not impassionate. Not random. Moreover, it's possible that the crowd view was self re-inforcing.
Absolutely correct. Please note up front that I no longer vote, and I don't really care who wins the election.
That said, I've been watching elections and presidential politics in the US very closely since Watergate. Trump is very likely to win this election. There are many factors in play that the pundit and pollster class don't understand, and don't seem to care to understand. In this scenario, both the large crowds (polls) and the moderate crowds (expert panels) are likely to miss the mark by wide margins.
This is not because the experts lack expertise, but rather because this election has some unusual features that have not been seen for a couple of generations at least, and their models (mental and statistical) don't seem to be taking account of that. Humans have a nasty habit of switching motives without notice.
as of 10pm in London last night, when one could reasonably expect it to be getting more accurate, it was at 11%. And 25% was in any case a very poor prediction considering these markets are supposed to be more accurate than polls.
>And 25% was in any case a very poor prediction considering these markets are supposed to be more accurate than polls
If the markets predicted 75% A vs 25% B and A always turned up, that would make them a poor prediction. Predicting 25% B may mean they're not making much of a prediction (since they're not so far off from 50/50), but it doesn't mean that B being the outcome is indicative of a the prediction being deeply wrong.
yes I know how sampling works. However as the uncertainty decreased, the prediction markets went in the wrong direction, rapidly on the last day and signficantly in the last week. This strongly suggests that there is an element of "herding" going on by what are supposed to be rational, atomic, individual, actors (the first two of those three assumptions were violated by profit incenctive, and a non-secret ballot, respectively, it seems). Hence I stand by my view that the prediction markets turned out to be terribly poor. You will admit that as final polls all hovered around 50%, if accurate (and they seemingly were), prediction markets should have been wildly swinging between, say 25/75 and 75/25. They were not. They were trending consistenly lower for brexit.
This does not imply polls are more or less accurate than prediction markets.
The way to measure that is to look at 100 cases where prediction markets predict 75% odds. If approximately 75 of them result in a winner, prediction markets are accurate.
Looking at a single data point and declaring a probabilistic predictor to be inaccurate is not even wrong.
but who is looking only at a single data point? We have more information than that, namely, the strongly consistent incorrect direction of the prediction markets, as the uncertainly decreased (40% to 25% in the last week, and 25% to 11% in the last day). It is clear that there was a feedback loop into the predictors. These markets appeared structurally biased towards the larger odds -> this is where the profit motive is possibly distorting.
The primary thing that determines overall group performance (assuming no interpersonal biases like peer pressure or coercion, etc) would be the probability that any individual chooses the "right" answer. If this is greater than 0.5, then according to the Condorcet Jury Theorem, the limit of the group as a whole choosing the right answer is 1 as the group size increases.
Perhaps having a smaller size allows the "noise" in the group to still produce the "right" answer more often than it would in a larger group?
Well, assuming a standard wisdom-of-the-crowds problem like "how much does that cow weigh?", the answer is drawn from a continuous interval and the probability of choosing the right answer is necessarily zero. Yet, the fact that that's a standard example of a wisdom-of-the-crowds problem suggests that the zero probability of being correct isn't really a big issue.
For a slightly different example like "how many jellybeans are there in this jar?", the answer is drawn from a discretized interval, but I feel pretty safe in assuming that the odds of any one crowd member getting it right are well below 50%.
No, that's incorrect in two ways. First, you must also assume that the individual opinions are independent. Second, the 0.5 probability threshold is only correct for binary situations; in real life where they may be a range of opinions, the threshold is much lower.
> Second, the 0.5 probability threshold is only correct for binary situations; in real life where they may be a range of opinions, the threshold is much lower.
The study from the article did focus on binary situations. From the article:
> Where previous research on collective intelligence deals mainly with decisions of ‘how much’ or ‘how many,’ the current study applies to ‘this or that’ decisions under a majority vote.
Crowds are wise only if their errors are at least pseudo-stochastic; if they are skewed in some direction by a pervasive bias (high/low, left/right, etc.), their predictions or choices will be driven by that bias.
This is one reason you need to be careful about disqualifying people from the decision if you're relying on the large-crowd effect--for example, an educational requirement to vote or only letting certain engineers participate in planning poker type estimations.
Unless you successfully boil it down to a small-crowd expert decision you'll spoil the process with skew. The uninformed people have a place in a larger-crowd process.
I personally think the brexit was a mistake, but now that it's over, I hope the best for them, and hope they manage to prove me wrong. I certainly don't think that my opinion is necessarily the correct one.
There are a few elements here, one of which geoelectric addresses in an earlier comment: does the crowd have expert knowledge.
Another is whether or not the question at hand requires expert knowledge. There are times when you're better off handing off the helm or pilot's seat to a qualified pilot than trying to average inputs of a large crowd, or to allow an electrician or plumber to address a problem within their skill scope.
A third aspect though might concern what the negative cost functions of a crowd might be.
The wisdom of crowds concept generally assumes the larger the crowd, the better it will be at arriving at some truth. In reality, various biases, distortions, and manipulations can emerge, to the point that the crowd's view is far worse than other options. Aristotle drinks hemlock. Trump is presumptive nominee. Brexit.
An element of networked systems, including decisionmaking systems, is what their cost functions are, in the sense of imposing negative results on those participating. I've been arguing for a year or two now that there is such a cost function, and that you can estimate that by noting the maximum size an effective network can grow to. Conversely, you can increase (or decrease) the effective size of a network by addressing that cost function. Increase it and you'll make large-scale aggregation less viable. Decrease it, and you can increase the size of effective aggregation.
As examples, a village is constrained in total size not only by its ability to secure necessary inputs (especially food and water), but in its ability to dispose of wastes and noxious emissions. London of the late 18th century had a mortality rate above its natural birthrate, and the only way the city could maintain its population was through net in-migration from the countryside (or foreign lands). This wasn't materially addressed until revolutions in water provision and sanitation, including the first modern sewerage system around 1850, addressed such concerns as cholera epidemics which were killing as many as 50,000 people a year.
In programming, Fred Brooks' The Mythical Man Month notes that few programming teams scale well beyond about 6-12 developers. The inter-personal communications costs make larger groups not only inefficient, but less effective, net than smaller ones. To produce larger teams, you've effectively got to split them into smaller ones. That's among the things that a highly modularised development process as is common in Free Software projects achieves -- see Apache, the Linux Kernel, or the Debian Project as examples (Gabriella Coleman, now of McGill University, wrote her dissertation on this topic, it's fascinating reading).
Computer chip design essentially removes the space and resistance costs of crowding high densities of electronic gates in small spaces. Again, the cost function is reduced.
In email and traditional (POTS and mobile) phone service, increasing amounts of spam are increasing cost functions, reducing the appeal and utility of the network to all involved. POTS has been shedding subscribers for some time, my expectation is that mobile phone service itself will be as well, more especially if interconnects, and filtering of VOIP alternatives (including iChat, Google voice chat, Skype, etc.) are further developed. Those networks, as a Long Island friend of mine some time back said, "gotta leahn to tawk to each othah!
If it does, then the noise of experts clashing (resolving conflicts between equally informed ideas) starts outweighing the benefits of them contributing the ideas to begin with. Basically you get a too many chiefs issue.
If it doesn't, you're seeking to average variably-educated wild ass guesses and taking advantage of the fact that WAGs have a tendency to form a bell curve around the actual answer if there's any ability to estimate whatsoever. Since there's otherwise no special knowledge, that ends up being the best you can get but requires a lot of people to get right.
Without having read up on it, my guess is the latter is related to central limit, with each person's WAG forming the first term of the mean of means; a WAG is itself sampling your own mental model and boiling it to a number. Given enough of those you get a normal that you can average.