In a sentence the book explains what causes errors in predictions and the fallacy of forecasters in economics, finance and medicine on being overtly confident of their predictions and mistaking the noise for signal in the data.
Here are some insights I got from reading the book;
- Economic (GDP) forecasts and predicting stock market returns is essentially very hard given so many factors affect them (US has 45,000 economic indicators). However despite this most economists and financial forecasters are too confident of their projections and as a result try to predict too accurately.
- Humans have a need for finding patters and abstractions more than other animals; it is a defense mechanism to simplify the world that is diverse and complex.
- However this tendency to find patterns is subject to our biases and one of them is the knowledge we have accumulated; the more information we accumulate the more our biases get solidified and so we can never make perfectly objective predictions. (It is for this reason that crowd-sourced forecasts are more accurate than individual ones from experts often somewhere between 15 and 20 percent since individual biases of each expert forecaster gets smoothed out in a large group).
- A natural tendency in summarizing data is to look at averages but the average, like the family with 1.7 children, is just a statistical abstraction and does not occur in reality.
- Relying on averages also ignores the extreme values in the data and forecasters often resist considering the out-of-sample data in the model i.e. the black swan problem as popularized by Taleb which is mistaking absence of evidence for evidence of absence.
- We often fail to make a distinction between risk and uncertainty; risk is something that can be measured/priced but uncertainty is something that cannot be estimated e.g. the likelihood of Covid-19 occurring or an earthquake happening. In projecting outcomes, we fail to take into account the probabilities of outcomes which is an essential part of forecasting. The virtue in thinking probabilistically is that you will force yourself to stop and ‘smell the data’ or consider the imperfections in your thinking. Bayes’ theorem can be used to account for errors in your own predictions.
- Another approach is to rely on heuristics, a heuristic approach to problem solving consists of employing rules of thumb when a deterministic solution to a problem is beyond our practical capacities. This approach has been popularized by Taleb who believes that it is impossible to predict ‘black swan’ like pandemics, economic meltdown etc events using models.
- However despite these biases in humans, it is essential that model predictions be reviewed by humans who can use their judgement and intuition to make sense of the data and predictions. This point was also made in the book ‘Prediction Machines – the Simple Economics of Artificial Intelligence’ which reasoned that in contrast to machines, humans are extremely good at prediction with little data.
Overfitting: Overfitting is the process of mistaking noise for a signal and it results in models giving an overly specific solution to a general problem and generally to worse predictions. An overfit model is likely to result when the data is limited and noisy and understanding of the fundamental relationships is poor. An overfit model will score better according to most statistical tests but will perform poorly for out of sample tests.
Challenge of Economic Forecasting: The challenge for economic forecasters is to try to determine cause and effect from economic data since the economy is complex with so many measurements; as an example the government produces data on literally 45,000 economic indicators.