Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Stats

Crowd Wisdom Better At Predictions Than Top CIA Analysts 136

Posted by timothy from the random-walk-down-global-politics-street dept.
First time accepted submitter tkalfigo (1448133) writes "The Good Judgment Project is an experiment put together by three well-known psychologists and some people inside the intelligence community. What they aim to prove is that average, ordinary people in large groups and access just to Google search can predict far more accurately events of geopolitical importance than smart intelligence analysts with access to actual classified information. In fact there is a clearly identified top 1 percent of the 3000 predictors group, who have been identified as super-forecasters: people whose predictions are reportedly 30 percent better than intelligence officers."
This discussion has been archived. No new comments can be posted.

Crowd Wisdom Better At Predictions Than Top CIA Analysts More

Crowd Wisdom Better At Predictions Than Top CIA Analysts

Comments Filter:

  • Luck resets every time you guess. (Score:4, Interesting)

    by rmdingler ( 1955220 ) on Friday April 11, 2014 @08:25PM (#46730705) Journal
    People ahead in guessing games such as these are probably more likely to regress to the mean than to continue defying probability.

    • Re: (Score:3, Insightful)

      by Arker ( 91948 )
      Except that this has nothing to do with luck. It has to do with independent observers having less pressure on them to, consciously or subconsciously, produce rhetoric ostensibly concerning foreign policy but whose content is determined by domestic political needs.

      • Why are you being modded down? For the font? And yes, they produce the results they are told to produce. That's how they play their game.

      • independent observers ,,, concerning foreign policy but whose content is determined by domestic political needs

        Damn! That's a usage case I hadn't heard of before. In politics it was always someone else's fault when it didn't work out. Now it's going to be EVERYone else's fault.

        Cloud-blaming! (tm)

      • Re: (Score:2)

        by ceoyoyo ( 59147 )

        Yes, that's what the article says. In a scientific paper that would be in the discussion, possibly in the conclusions. Experienced scientists know that the discussion, and depressingly frequently the conclusions, are BS the authors made up that's not really supported by the data, one way or the other.

      • Re: (Score:3)

        by Sique ( 173459 )
        Not only that, but people with some distance to the situation and only superficial knowledge are not blinded by facts and details. We have the same phenomenon with predicting sport results. There, people who are not absolute fans or professionals in the sport usually fare better at predicting results as they don't give too much weight on some details, or their own preferences, which in the long run prove to have much less influence than expected. Instead they basicly tend to put teams or athletes they often

      • By someone else: http://www.amazon.com/The-Diff... [amazon.com]

        By me on the need for better intelligence tools for the public: http://pcast.ideascale.com/a/d... [ideascale.com]
        http://www.phibetaiota.net/201... [phibetaiota.net]

        By me on the security clearance process reduces cognitive diversity in three letter agencies: http://www.phibetaiota.net/201... [phibetaiota.net]
        "This essay discusses how the USA's security clearance process (mainly related to ensuring secrecy) may [ironically] have a counter-productive negative effect on the USA's national security by reducing "co

    • People speak of the wisdom of crowds but the madness of crowds create things such as financial crashes or stupid wars.
      Shall we discuss and research this?

  • CIA uses Dirty Tricks Project (Score:2, Interesting)

    by Anonymous Coward

    CIA cannot believe a wisdom based output, they have to believe that their actions will change the outcome.

  • There was a project affiliated with Google that aimed to predict disease outbreaks using the search engine, but that didn't turn out that well. In fact, it barely succeeded in any of its predictions.

  • Well yeah (Score:5, Insightful)

    by MichaelSmith ( 789609 ) on Friday April 11, 2014 @08:29PM (#46730739) Homepage Journal

    This is why its better to have elections than let the CIA select the government. AFAIK, anyway.

  • Seems fishy (Score:5, Insightful)

    by ceoyoyo ( 59147 ) on Friday April 11, 2014 @08:32PM (#46730767)

    I wonder if they properly controlled for luck. Take three thousand people and get them to make predictions and some of them are going to appear unusually accurate than others even if all of them are just making completely random guesses. You'd be surprised how many people don't correctly account for that. Every paper proposing clinical diagnostic criteria I've ever read, for example.

    • Re:Seems fishy (Score:4, Interesting)

      by mikael ( 484 ) on Friday April 11, 2014 @08:56PM (#46730909)

      Why do you think it is purely luck? When you have these wild discussion parties - things like "is a bright blob of pixels on a Mars Rover image a cosmic ray, a high-voltage dust-devil, light contamination of a camera box, a gas geyser", you will have an incredible combination of experts - everyone from geologists, ranchers, hill-hikers, photographers, astronomers. Geologists will tell you want can and can't come from the ground, ranchers and hill-hikers will tell you things they have seen and never seen, photographers will tell you what visual artifacts can appear on a camera, and astronomers tell you what can fall from the sky and can't, and what those falling things look like.

      It's like solving a giant logic problem where everyone can cross off or tick what what they know. Eventually the set of possible answers reduces down to one or two.

      • Re:Seems fishy (Score:4, Interesting)

        by hibiki_r ( 649814 ) on Friday April 11, 2014 @11:23PM (#46731597)

        Except this isn't how it works at all.

        The wisdom of crowds works doesn't have anything to do with having experts. After all, the experts have no way of influencing the crowd. It is a well defined phenomenon that works when people's biases are pretty random, so mistakes cancel each other out. It's a lower quality estimation mechanism than a market, where people that are sure of their answer can be 'louder' than those that don't know said answer, and it lacks the feedback mechanisms of a market, but still, it is helpful to predict things based on widely available information. Ask the crowd information few of them have any idea about, and their result will suck.

        So what does the average beating CIA personnel? That the CIA's biases are large enough to need quite a bit of quality control.

        Now, having a 1% of the respondents be far better than the CIA experts probably means nothing. If I invite 3000 people over to guess how 10 coin flips will turn out, chances are one or two of them will guess all of them correctly, but that would not make them seers capable of seeing the future. how many people were worse than 30% worse than those same CIA experts?

        • Re: (Score:2)

          by ceoyoyo ( 59147 )

          In statistics it's called an unbiased estimator. Most people know it as an average. It doesn't have any particular link to crowds and the behaviour is very well defined. It does, however, require that the individual estimates be wrong in a random way.

          You managed to pick exactly the same example I did.

        • The wisdom of crowds works doesn't have anything to do with having experts.

          You are right that the wisdom of crowds does not come from having experts. The wisdom of crowds comes from having a lot of people who all have a little bit of knowledge relevant to the subject. Some of that knowledge might be something that you would not necessarily think was relevant, but when applied as a filter on the other knowledge present produces a result much more accurate than an expert on the subject would ever produce.

          The results of this study are not new. Back in the lat 70s, early 80s, there

          • Posting AC isn't *the* answer, only a part. Here on /. we frequently see ACs post with an appeal to authority, claiming national publication, having designed some new, cool and super-advanced kernel, etc. Those would need to be filtered out as well. I myself have referenced some personal project. Those too. Even the language used would have to be examined to remove certain subtext.
            • Actually, the key is to eliminate anything which allows me to connect the majority of the comments by one individual as all being by that individual. All of those other things are only relevant if they combine with multiple comments to build a picture of one person who has more wisdom on the subject than everyone else. When I read comments by an Anonymous Coward on slashdot which claims the things which you reference, I always read the rest of their comment more critically. If I find their argument to use s

      • Re: (Score:2)

        by ceoyoyo ( 59147 )

        Because that's not what's going on here. The example in the article is a pharmacist who somehow manages to be better than everyone else at predicting geopolitical events. Not a party with a bunch of experts in various fields hashing things out, just a pharmacist in her kitchen in her spare time.

        Flip a coin ten times and there's only a tenth of one percent of a chance of it coming up heads every time. Flip a thousand coins ten times and there's only a small chance one won't come up heads ten times in a ro

        • Actually, if you read the entire article, it appears that what that pharmacist is doing, inside her own head, is averaging what all of the various sources she reads have to say on any given subject. The mistake would be to say, "Oh, over the last three years, her predictions have been 30% better than the experts. Let's make her one of the experts!" If at some point she were to start to believe that she was an expert at this forecasting, her accuracy would fall off because she would stop adjusting her under

          • Re: (Score:2)

            by ceoyoyo ( 59147 )

            Expert forecasters also average their perceived probabilities from lots of different sources. There's nothing magic about a small town pharmacist doing it. The summary and a lot of Slashdotters seem to like to play up the anti-expert angle, but it's certainly not relevant to my OP, and I doubt the project has produced any evidence for your conclusion.

            • Except that the "expert" forecasters are influenced by the charisma* of certain individuals to produce the type of results those individuals desire. There have been studies that show that one of the most important aspects of "crowd wisdom" is eliminating the ability of individuals to use their force of personality to influence the decision reached by the crowd.


              *I am using "charisma" here to sum up all of the aspects of force of personality and authority over those evaluating the data.

              • Re: (Score:2)

                by ceoyoyo ( 59147 )

                You're still making conclusions that aren't based on the data. There's nothing in their project that identifies why individuals, or the group, might be better than the experts, and my question is about whether the individuals they've identified ARE even better than the experts or if they've simply discovered the right side of a Bell curve (a la Niven).

                There have been studies that have shown that crowds, under certain circumstances, can be somewhat resistant to bias. In other circumstances this is obviousl

    • I wonder if they properly controlled for luck.

      You obviously didn't RTFA.

      But she signed up, got a little training in how to estimate probabilities from the people running the program, and then was given access to a website that listed dozens of carefully worded questions on events of interest to the intelligence community, along with a place for her to enter her numerical estimate of their likelihood.

      "Usually I just do a Google search," she said.

      In fact, she's so good she's been put on a special team with other superforecasters whose predictions are reportedly 30 percent better than intelligence officers with access to actual classified information.

      It's not luck they've selected for, it's the ability to make educated guesses.

      • Re: (Score:2)

        by ceoyoyo ( 59147 )

        I did read the article. Nothing in what you quoted is at all relevant. Perhaps you didn't understand my post?

    • That was my thought too. For example, if you grab n people off the street and ask them to make a total guess at 10 coin flips, just by pure chance alone 1.1% of them are going to get 8, 9, or all 10 correct (80% or better "correct" rate). A cumulative binomial distribution of 8 correct of 10 trials with 50% success rate is 98.9%. Likewise the bottom 1.1% will guess correctly 20% or worse. This is the the usual "cause" of some research investigating psychic phenomenon. If the researchers aren't very wel

      • Re: (Score:2)

        by ceoyoyo ( 59147 )

        Or does it mean the experts were correct 55% of the time and the top guessers, I mean, super-predictors, were right 56.5% of the time? That would be the normal meaning of a percent difference.

  • Reminds me of the Policy Analysis Market (Score:5, Interesting)

    by khallow ( 566160 ) on Friday April 11, 2014 @08:36PM (#46730793)
    Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress (see my opinion [kuro5hin.org] then). It's too bad that we have to go to a somewhat contrived surveying/polling system rather than use something that we know works.

    For example, I think a PAM system would have given us (and I mean everyone not just US policy makers) insight into how the events of the Arab Spring revolutions would evolve even if it couldn't have predicted the original flash point.

    • Re:Reminds me of the Policy Analysis Market (Score:5, Interesting)

      by TapeCutter ( 624760 ) on Friday April 11, 2014 @10:36PM (#46731401) Journal

      even if it couldn't have predicted the original flash point.

      Funny you should say that, the diplomatic cable leaks showed [wikileaks.org] that high level western diplomats in Syria were concerned about a civil war erupting due to the severe "fertile crescent" drought fuelling internal migration from rural areas to the cities (10% of Syria's total population simply abandoned their farms due to lack of water). The drought caused food prices to rise sharply and food riots became a regular occurrence in cities across the middle east and North Africa.

      "flash point" - Have a look at why that protester set fire to himself in the public square and why it resonated so strongly across the Arab world, it wasn't because they all logged on to FB and suddenly realised their governments were tyrannical. Predicting this sort of social unrest is like predicting an earthquake in LA, you can be pretty confident that your prediction will come to pass but have no idea when.

    • Re: (Score:3)

      by nut ( 19435 )

      Back in 2003, there was a similar system called the Policy Analysis Market (PAM) that was close to being implemented. It got deep-sixed by some world-class idiots from Congress ...

      Maybe they weren't idiots. Maybe the were protecting a lucrative after-Congress job [techdirt.com] market...

    • Re: (Score:3)

      by gringer ( 252588 )

      New Zealand has iPredict, where you can make money off correct guesses about the future:

      https://www.ipredict.co.nz/ [ipredict.co.nz]

  • Question of scale (Score:4, Insightful)

    by Livius ( 318358 ) on Friday April 11, 2014 @08:42PM (#46730827)

    With enough people, there will be someone with insightful information, and probably a balance of opinions. Searching for bugs in open source works a little like that.

    But in theory if a professional intelligence service had hard evidence that, for example, a politician is bluffing about something, then a policy can be adopted even if it goes against some conventional wisdom.

    For example, the information that Saddam Hussein's WMD programme was a hoax prevented a rash invasion...., um, never mind.

    • Re:Question of scale (Score:5, Funny)

      by mikael ( 484 ) on Friday April 11, 2014 @08:57PM (#46730927)

      Both the British and Americans used the same government contact for their information, but they didn't tell each other who that contact was. In fact, they had different codenames for the person. When they cross-referenced each others information, they got two confirmations.

      • Re: (Score:2)

        by Livius ( 318358 )

        Professional intelligent agents were not fooled. People who only heard what they wanted to hear do not count as professionals.

        • Professional intelligent agents were not fooled. People who only heard what they wanted to hear do not count as professionals.

          In my experience, professional people fool themselves all the time, and refuse to listen to anyone who contradicts them, so why would professional intelligence agents be an exception?

          CIA is "the Company"; like all others, it's run by pointy-haired bosses.

  • Old news (Score:4, Informative)

    by Crashmarik ( 635988 ) on Friday April 11, 2014 @08:52PM (#46730883)

    http://en.wikipedia.org/wiki/D... [wikipedia.org]

    This has been known since the 60s. Only reason it keeps cropping up is the ego of the people involved in analysis, and the organizational inertia of the agencies involved.

    • Re: (Score:1)

      by Jmc23 ( 2353706 )
      Did you read your link? Delphi method is basically what they are comparing against, a structured group of experts.

      • Not sure if you are trolling or not, but the articles make no mention of the comparison.

        • Re: (Score:1)

          by Jmc23 ( 2353706 )
          If you read, and understood, your link and the article or website, then you would understand I'm not trolling. Of course, they then take their unstructured group and structure it to only contain 'experts'.
  • My drunken pet vole makes better predictions than those idiots at the CIA. News at 11, Captain Obvious.
  • I doubt there is any field where one percent of laymen aren't vastly superior to the majority of professionals. The same applies to art, engineering, science, and any other field of study. This is statistically normal.

    • I doubt there is any field where one percent of laymen aren't vastly superior to the majority of professionals. ... This is statistically normal.

      Fine, but just like the quatrains of Nostradamus [nostradamu...ctions.org]: can you identify them correctly beforehand? Counting the perfect hits after the fact isn't fair. (But then again I guess it worked for Miss Cleo [weht.net] for a while [consumeraffairs.com])

      BTW: 16th century Mr. N. is an idiot. But he's better than the current sales-people paying attention to him with 5 centuries more experience. Oh, and multiple Blood Moons [latimes.com] are soon arriving -- buy your Tarot cards and ticket to safety [imdb.com] now, before it's too late!

    • Re:Bell Curve (Score:4, Insightful)

      by TapeCutter ( 624760 ) on Friday April 11, 2014 @10:43PM (#46731425) Journal
      Interesting, so why do "laymen" in the US keep electing zealots, crackpots, and "entrepreneurs" who are clearly lying to their face for fun and profit? Do you guys enjoy being treated with contempt?

      • Re: (Score:2)

        by idji ( 984038 )
        Laymen elect zealots, crackpots and "entrepreneurs" who are clearly lying to their fact for the Bread and Circuses [wikipedia.org]
      • That is because elections happen in a way that eliminates the most important aspect of crowd wisdom. In order for crowd wisdom to be reliable it must be insulated from being influenced by the charisma of individuals. There is no way to set up elections to do this. Actually, I wonder if the secret ballot may in a way actually exacerbate this problem.

        This thought just came to me now, so I do not think I can explain the reasoning as to why that might be so. I will try any way. It seems possible that the nece

    • That doesn't make any sense for things where training is needed though. 1% of laymen being better than civil engineers at building extremely large bridges? 1% of laymen being better at fixing cars than a mechanic? How about 1% of laymen being better at basketball than NBA players? It makes absolutely no sense, because we are talking about things where the training time is extremely valuable, and guessing at random will not help you, because there are too many possible answers.

      Even in yes/no questions, if 1%

  • Two words: Delphi Pool. [wikipedia.org] This is an idea from 1975's The Shockwave Rider.
  • EYYYEEEEE ENNN TEEEE Ps?

    That's where it's at the...

    EYYYEEEEE ENNN TEEEE Pee... eeeeeees

  • I don't exactly get it. Is it the group as a whole that predicts accurately or its "best predictors"? Because clearly the first hypothesis favors direct democracy as a decision-making process. My intuitive guess is that when you pick a large enough group, some people within that group are clearly going to do better than specialists, because, in a certain way, they are themselves specialists.

  • I am really curious as to who makes up the "super forecasters" of these geopolitical problems. I suspect that they are merely the Libertarian contingent. 30 people out of 3000? Sounds about right.

    Or, alternatively, are they spiritual people? People who partaken in psychedelic experiences? What defines this group?
  • This story actually really interested me - On its face, the idea of a website that does these things: Poses user-submitted predictive questions, with user profiles so you can track the most successful predictors, and probably some sort of range voting system for the actual voting process, seems like a really swell idea.

    Unfortunately, I've not nearly the technical skills or capability to jump into making a website that aggregates questions, votes, user statistics, graphs, profiles and so on. I went ahead
  • Ask three thousand people to predict what sequence of heads and tails will come up when you flip a coin 30 times. A few of them will appear to have the ability to predict it correctly, at least in that sample.
  • The US Government came out with an idea to do this for terrorist attacks after 9/11 where people with knowledge in the areas of the middle east, various groups and other specialties would be invited to a program they could make forecasts. Those that were correct would have money donated to their university or another organization of their choice.
    Liberals came out and called it the idea of a bunch of crazies.

Slashdot Top Deals

Two can Live as Cheaply as One for Half as Long. -- Howard Kandel

Close