Story

Intelligence agencies turn to crowdsourcing

The idea of crowdsourcing geopolitical forecasting is increasing in popularity, and not just for spies.  Wikistrat, a private company touted as “the world’s first massively multiplayer online consultancy,” was founded in 2002, and is using crowdsourcing to generate scenarios about future geo-political events. It recently released a report based on a crowdsourced simulation looking at China’s future naval powers.

Warnaar says that Wikistrat’s approach appears to rely on developing “what-if scenarios,” rather than attaching a probability to a specific event happening, which is the goal of the Iarpa project.

Of course, the ultimate question is: how good are the crowd’s predictions? Warnaar compares this science to weather forecasting, which albeit imperfect, still provides useful and reasonably accurate information on future events. Part of what helps weather forecasters improve their prediction is constant feedback: if they predict rain, and they get it wrong (or right), they instantly learn.  “This constant feedback makes them well-calibrated,” says Warnaar.

In fact, this sort of “self-calibration” is how one of the crowdsourcing models works: if the “crowd,” predicts that an event is going to happen with an 80% probability, but in reality this should have been 60%  (crowds tend to be overconfident), then the model is able to aggregate all of the information to churn out a more accurate prediction.

The system is also designed to ensure that any efforts to sabotage forecasts are minimized. “Everyone can make forecasts but not all of those forecasts are included in our models and each forecast may have a different weight,” says Warnaar. “You would therefore have to be a consistently good forecaster to be able to influence the aggregate forecast with a rogue prediction, but even then your forecast must be consistent with your previous pattern.”

To catch any potential rogue elements, the system also flags up any unusual activity for further scrutiny. “So far we’ve not found any evidence that a single forecaster or group of forecasters was able to purposely skew the results,” he says.

The project is already yielding results: in the first year, Warnaar says, they were able to show that the crowdsourced forecasts were 25% more accurate than forecasts produced by a control group, which involved simply averaging the forecasts made by a number of individuals.  The plan is double that accuracy over the next year to 50%.

In addition to improving intelligence forecasts, the research may also yield other benefits, such as understanding what type of person is better at predicting future events. “There is very little research that points to what makes a good forecaster,” says Warnaar.

Those working on the current project are careful to note that the current project is about research, not spying. The names and personal information of the users are not provided to Iarpa, only the results of the forecast. Users entering the site must provide an email, but not a real name, and only answer two questions: whether they are over the age of 18 and if they are American citizens.

But for those who dream of being James Bond, or conversely, worry that their predictions could be used by spies, the website has a simple disclaimer: “Forecast topics are not related to actual intelligence operations.”