Contrasting traditional intelligence analysis with game theory and "superforecasting" to predict future events

birdman12345 · January 22, 2018, 5:05pm

Note: Pardon if this is poorly written, I went over a couple times and made edits but its kinda shoddily put together.

I’ve been looking for prominent techniques, theories and methods of analysis recently and came across two which I think may be particularly useful for predicting future events. I thought some of the people here may find these interesting as well.

The first thing for me to note is that while I dont know much at all about intelligence analysis, I did hear multiple times that analysts are judged is by whether they followed a process for analysis, and that they care more about whether or not they stuck to the process, rather than if they came to the correct understanding or had the correct result. This was intuitively appealing to me, because of how often people underperform the market and underperform when trying to make deviations from algorithms. Apparently the analysts also give a lot of thought to cognitive and emotional biases and their effects on their processes. That reminded me a lot of the cfa program. The cia even published a book about the subject. They go in to a bunch of the biases and psychological phenomenon, including hindsight bias, how to structure problems, analytical techniques, the limits of remembering. I thought that was really cool and intend to look more into that later.

https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/PsychofIntelNew.pdf

I started thinking along these lines when was listening to a podcast on “superforecasting,” which is what the top 2% of predictors who participate in these geopolitical prediction events that are put on by this dude, Philip tetlock, are described as being. The basis of it was contrasted a bit by describing how previous intelligence analysis had been done. Apparently the “superforecasters” were on average something like 25% more accurate in their predictions than the intelligence analysts. Apparently they were consistently correct year after year as well, so it wasnt just a one off thing. At one point in this freakonomics interview, they start shitting on intelligence analysts for being vague, which I thought was interesting.

_TETLOCK: Well, in the context of the Bay of Pigs, the Kennedy administration had just come into power and they were considering whether to support an effort, by Cuban exiles and CIA operatives and others, to launch an invasion to depose Castro in April ’61. And the Kennedy administration asked the Joint Chiefs of Staff to do an independent review of the plan and offer an assessment of how likely this plan was to succeed. And I believe the vague-verbiage phrase that the Joint Chiefs analysts used was they thought there was a “fair chance of success.” And it was later discovered that by “fair chance of success” they meant about one in three. But the Kennedy administration did not interpret “fair chance” as being one in three. They thought it was considerably higher. So, it’s an interesting question of whether they would have been willing to support that invasion if they thought the probability were as low as one in three.

_

They also contrasted more contemporarily against president obama and how he viewed the chance of OBL being in the house in pakistan.

_TETLOCK: I think there’s vast variation among politicians and how numerate they are and how open they are to thinking of their beliefs as gradations along an uncertainty continuum rather than expressions of tribal loyalties. We have the story in the book about President Obama making the decision about going after Osama Bin Laden and the probability estimates he got about Osama’s location, and how he dealt with those probabilities. T he probabilities ranged from about, I don’t know, maybe from about .4 to about .95 with a center gravity around .75. And the President’s reaction was to shrug and say, “Well I don’t know what to do with this. It feels like a 50/50 thing, like a coin toss.” Now, that’s an understandable reaction from a president who is about to make an important decision and feels he’s getting somewhat conflicting advice and feels like he doesn’t have closure on a problem. It’s a common way to use the language. But it’s not how the President would have used the language if he’d been sitting in a TV room in the White House with buddies watching March Madness and Duke University is playing and someone says, “What’s the likelihood of Duke winning this game?” and his friends offer probabilities ranging from 0.5 to about 0.95 with a center of gravity 0.75 once again. He wouldn’t say, “Sounds like 50/50.” He’d say, “That sounds like three to one.” Now, how much better decisions would politicians make if they achieved that improvement in granularity, accuracy, calibration? We don’t know. I think that if the intelligence community had been more diffident about its ability to assign probability estimates, the term “slam dunk” probably wouldn’t have materialized in the discourse about weapons of mass destruction in Iraq. I think the actual documents themselves would have been written in a more circumspect fashion. I think there were good reasons for thinking Saddam Hussein was doing something suspicious. I’m not saying that the probability would have been less than 50 percent. The probability might have been as high as 85 percent or 80 percent but it wouldn’t have been 100 percent.

_

Apparently the “superforecasters” compete using an interesting set of tools thats been put together for them that includes weather prediction models. They make base rates based on historical events, then they adjust from that to fit the new circumstances and how significantly they diverge. They also calibrate their findings by saying shit like “we think there’s a 60% chance of this happening in this time period.”

_TETLOCK: One of the more distinctive differences between how superforecasters approach a problem and how regular forecasters approach it is that superforecasters are much more likely to use what Danny Kahneman calls the outside view, rather than the inside view. So, if I asked you a question about whether a particular sub-Saharan dictator is likely to survive in power for another year, a regular forecaster might get to the job by looking up facts about that particular dictator in that particular country, whereas the superforecasters might be more likely to sit back and say, “Hmm, well, how likely are sub-Saharan dictators who’ve been in power x years likely to survive another year?” And the answer for that particular question tends to be very high. It’s in the area of 85, 95 percent, depending on the exact numbers at stake. And that means their initial judgment will be based on the base rate of similar occurrences in the world. They’ll start off with that and then they will gradually adjust in response to idiosyncratic inside-view circumstances. So, knowing nothing about the African dictator or the country even, let’s say I’ve never heard of this dictator, I’ve never heard of this country, and I just look at the base rate and I say, “hmm, looks like about 87 percent.” That would be my initial hunch estimate. Then the question is, “What do I do?” Well, then I start to learn something about the country and the dictator. And if I learn that the dictator in question is 91 years old and has advanced prostate cancer, I should adjust my probability. And if I learn that there are riots in the capital city and there are hints of military coups in the offing, I should again adjust my probability. But starting with the base-rate probability is a good way to at least ensure that you’re going to be in the plausibility ballpark initially.

_

This next part really interested me because I’m aware that what really makes a good investor is being right 55-60% of the time.

_

TETLOCK: He defines the difference between a great poker player, a world-class poker player and a talented amateur as: the world-class player knows the difference between a 60/40 proposition or a 40/60 proposition. Then he pauses and says, “no more like 55/45, 45/55.” And of course you can get even more granular than that in principle. Now, when you make that claim in the context of poker, most people nod and say, “Sure, that sounds right,” because in poker you’re sampling from a well-defined universe. You have repeated play. You have clear feedback. It’s a textbook case where the probability theory we learned in basic statistics seems to apply. But if you ask people, “What’s the likelihood of a violent Sino-Japanese clash in the East China Sea in the next 12 months?” Or another outbreak of bird flu somewhere? Or Putin was up to more mischief in the Ukraine, or Greece might begin flirting with the idea of exiting the Eurozone? If you ask those types of questions, most people say, “How could you possibly assign probabilities to what seem to be unique historical events?” There just doesn’t seem to be any way to do that. The best we can really do is, use vague verbiage, make vague-verbiage forecasts. We can say things like, “Well, this might happen. This could happen. This may happen.” And to say something could happen isn’t to say a lot. We could be struck by an asteroid in the next 24 hours and vaporized. 0.0000.1 percent. Or the sun could rise tomorrow. 0.99999 percent. So “could” doesn’t tell us a lot. And it’s impossible to learn to make better probability judgments if you conceal those probability judgments under the cloak of vague verbiage.

_

Apparently this method also has some similarities and differences from other prominent dudes making predictions.

Theres more to it but I thought you guys may be interested to listen to it. Now what apparently the academics like kahnemann think is most interesting about it is how their predictions apply to different time periods and how the probability of events occuring changes over time. Apparently these guys are particularly good with that while normal people predicting would have a more constant view of the probability of something happening over time.

http://freakonomics.com/podcast/how-to-be-less-terrible-at-predicting-the-future-a-new-freakonomics-radio-podcast/

Now another dude who I came across is this guy, Bruce bueno de mesquita. Apparently hes one of the predominant game theory dudes in the country at the moment. Apparently he uses his models to make strategies for the NIC. He’s apparently pretty good at it and apparently his models turned out to work out correctly 90% of the time, according to the cia, which was more accurate than their own analysts. He apparently did shit like coming up with the strategy that led to the creation of hamas, and he came up with an alternate conclusion about irans intentions re: building a nuke, which he presented to the NIC. He said that undergrads in his course were able to come up with the same conclusion using his model, and figure out the sort of shit that the intelligence folks figured just by going through the internet and using publicly available data. Apparently he presented the idea to the NIC and then got into an argument with one of the guys, and basically told him that they had the same data and that they were coming to divergent conclusions (the NIC said that they were predicting iran would have a bomb within 9 months). Apparently he asked the dude to think about why they were coming up with such divergent conclusions, and bruce thought it was because the dude had already come up with his belief and was using the available information to support it.

He also plotted out who has power over time in the countries. He’d do things like project who would have more power in the future and what their interests are and what those mean.

He goes through some interesting stories in this video.

[video:https://www.youtube.com/watch?v=jUPEUZy3UX8]

I’d love to hear anyone elses thoughts on the subject, whether yall have some guys/guys work you particularly find interesting, techniques, studies etc. Here are some relevant books by the dudes and one other which I read a long time ago and still go through that I really like because it uses graphs and shit instead of just being a bunch of writing.

Relevant books:

https://www.amazon.com/Predictioneers-Game-Brazen-Self-Interest-Future/dp/081297977X

https://www.amazon.com/Superforecasting-Science-Prediction-Philip-Tetlock/dp/0804136718

https://www.amazon.com/Thinkers-Toolkit-Powerful-Techniques-Problem/dp/0812928083

inb4 “if they were so great at predicting, theyd make a bunch of money on the stock markets”

rawraw · January 22, 2018, 5:17pm

Get out of here with your low conviction forecasting. The stock market will likely decline 6o percent!

ohai · January 22, 2018, 8:39pm

This sort of behavioral science is pretty useful to read about, but you also need some kind of practical attitude to implement it in markets. A lot of these professors start themed funds or something to capitalize on their research, but most of them don’t do too well.

rawraw · January 22, 2018, 8:43pm

The cool thing about Tetlock work is that it was developed in a real world competition. Apparently his guys were better than futures markets on macro trades, but I don’t think they ever actually tried to make bets outside the intelligence framework. I really enjoyed the book

GorillaNerd · January 22, 2018, 8:47pm

you miss 100% of the shots you never take, so that’s why im always down for shots.

^that’s game facts right there!

system · January 23, 2018, 2:00am

It would be!

The above is a summary of why the investment industry sucks at what they do. Intelligence outliers can’t explain why they are correct, because of the IQ gap, and intution. By forcing them into a robotic process, you eliminate outperformance from your superior analysts.

birdman12345 · January 23, 2018, 3:29am

That’s not really true at all PA, abiding by rules in investing and not letting emotion get in the way shows discipline. It doesn’t mean the initial thesis are less valid. Someone’s got to come up with the process to begin with and analyze new securities according to it.

system · January 23, 2018, 2:14am

Which is current state, rule-following robots.

Nope. That is corporate process thinking.

birdman12345 · January 23, 2018, 3:33am

You’re being obtuse and missing the point pa. I think you just want to be a nay sayer about everything, so I’m not going to take what youre saying seriously. These guys, bruce bueno de mesquita and philip tetlock, haven’t fully automated their processes, there still have to be humans involved to work with them and apply them and develop them further, so theres no reason to think that full automation would occur in securities analysis either.

system · January 23, 2018, 12:10pm

I don’t think you can state the point?

And by the time they get done with that, the market opportunity will have passed. Inefficiencies happen NOW, you have to analyze them FAST, and have autonomy to make a move. The corporate process-obsessed losers ARE the market.

system · January 23, 2018, 4:11pm

Snowflake, stop sending me mad PMs.

If you see an error in my logic [corporate process-obsession is the market’s game --> thus beating the market has to use what they miss], then you can point it out here, where it will be publicly dissected. You don’t get to cry “abuse” when you are wrong and silence people, I will continue to intrude on your safe space.

Reality is going to kick your ideas harder than me, you better toughen up kid!

birdman12345 · January 23, 2018, 5:25pm

Thats really how you want to do this PA? Damn man you really do have a toxic personality. To thine own self be true though. I asked you politely to not respond in my threads because you’ve got a toxic, negative personality and all you seem to want to do is shit on people. I’m not mad, its just that you don’t contribute anything constructive or positive, you’re not making this place more welcoming for the free and open expression of ideas, and I don’t want that here. And its not just this thread, its all the threads you post to. What you’re showing here isn’t some “reality” that me or others need to get used to, its just showing that you’ve got this toxic personality and that you’re forcing yourself on me against my will after I asked you politely to stop. I just wanted to have a discussion about analytical techniques and game theory and this is the the result. I don’t know why you insist on making this public and making a bigger deal out of it than it needs to be. I’m really disappointed in how you’re handling yourself. And since I’ve got nothing to hide, this is my pm with PA. Note the civility on my side and the “blow me racist,” the childish name calling and general douchey, negative attitude on PA’s side, which is precisely why I want him to stay out of my threads.

https://imgur.com/a/7Geop

system · January 23, 2018, 4:25pm

No, I’m here for analysis.

You’re the liberal “cry bully”. You can’t handle your threads getting “trolled” with facts, yet you troll other’s threads with zero substance culty nonsense. Boo hoo!