This year, I kicked off with Nate Silver’s The Signal and the Noise. It was an uncharacteristic choice because it’s not a novel, and far from being a vegetative experience, the book encourages some self-examination.
It is a richly-layered examination of prediction, probability, correlation and causality. He is all about finding meaningful relationships between data. A very simple, but highly illustrative example, is the observation that in the same months that ice cream sales peak, there is also an increase in the number of bush fires. There is correlation, but no causality. He makes the point that Big Data increases the opportunity of finding data points that randomly correlate, thereby actually working against good predictions.
Silver expounds a Bayesian approach to prediction, in which a base rate or ‘prior probability’ is the beginning point. Having made the first prediction, the ‘prior’ is updated, which offers the opportunity of fine-tuning second- and third-round predictions. It’s all very interesting and thought-provoking.
There is related subject matter in my second read, Daniel Kahneman’s Thinking, Fast and Slow.
Nate Silver was a professional poker player for a while, putting his predictive abilities to the test. There certainly is a fit between gambling and prediction; over time, unsuccessful gamblers are those who demonstrate poorer understanding of probability.
Chance is linked to probability, but it’s different to luck. Kahneman cites Nassim Taleb in introducing “narrative fallacy”, which is our tendency to string together apparently relevant events in an attempt to construct causality, without allowing for the huge role played by luck. If you don’t like the word ‘luck’, substitute good fortune. For example, Google – and its founders – is the subject of numerous case studies. However, in the early years they would have sold the entire company for less than $1 million, but the potential buyer thought the price was too high. The Larry Page and Sergey Brin legend would have read very differently!
Even if we are not forecasting professionals, we are required to assess probability on a daily basis. We may as well learn to do it better. These books are a great place to start.