some image

Having fun, writing about the stuff I like

Forecasting, Chance, Probability and more

Oscar Foulkes January 10, 2014 Books No comments
In years past, I’ve opened my holiday reading with an escapist novel that I don’t put down until it’s finished, surfacing only for meals. I find it’s a great way to disconnect, which is the point of a holiday, after all.

This year, I kicked off with Nate Silver’s The Signal and the Noise. It was an uncharacteristic choice because it’s not a novel, and far from being a vegetative experience, the book encourages some self-examination.

It is a richly-layered examination of prediction, probability, correlation and causality. He is all about finding meaningful relationships between data. A very simple, but highly illustrative example, is the observation that in the same months that ice cream sales peak, there is also an increase in the number of bush fires. There is correlation, but no causality. He makes the point that Big Data increases the opportunity of finding data points that randomly correlate, thereby actually working against good predictions.

Silver expounds a Bayesian approach to prediction, in which a base rate or ‘prior probability’ is the beginning point. Having made the first prediction, the ‘prior’ is updated, which offers the opportunity of fine-tuning second- and third-round predictions. It’s all very interesting and thought-provoking.

There is related subject matter in my second read, Daniel Kahneman’s Thinking, Fast and Slow.

The spontaneous, generally involuntary ‘fast’ thoughts produced by what he calls System 1, can result in a variety of biases and prejudices. Overcoming these is the work of System 2, our ‘slow’, more analytical way of thinking. It’s not as easy a read as Silver’s book, but it is no less interesting. Kahneman was awarded a Nobel Prize for his work, so it is no surprise that his book is a much more dense read.

Nate Silver was a professional poker player for a while, putting his predictive abilities to the test. There certainly is a fit between gambling and prediction; over time, unsuccessful gamblers are those who demonstrate poorer understanding of probability.

Chance is linked to probability, but it’s different to luck. Kahneman cites Nassim Taleb in introducing “narrative fallacy”, which is our tendency to string together apparently relevant events in an attempt to construct causality, without allowing for the huge role played by luck. If you don’t like the word ‘luck’, substitute good fortune. For example, Google – and its founders – is the subject of numerous case studies. However, in the early years they would have sold the entire company for less than $1 million, but the potential buyer thought the price was too high. The Larry Page and Sergey Brin legend would have read very differently!

Even if we are not forecasting professionals, we are required to assess probability on a daily basis. We may as well learn to do it better. These books are a great place to start.



Add your comment