Model Risk – view from Actuarial Industry

Models have been used pervasively in all industries for a long time but their governance and the emphasis on robust development and independent validation was lax and not well defined until 2011. This changed with the publication of SR 11-7 by the Fed in 2011 specially for the financial service firms in US; the SR 11-7 publication for the first time laid out regulatory expectations for model development, model validation and model risk governance for all financial institutions. It has been more than a decade since the publication of the guidance and it has had a monumental impact on use of models within financial services in the US. Model risk which was largely thought to just include independent validation prior to SR 11-7 has moved from the fringes of risk management to occupy a predominant part of  risk management and now covers the entire model lifecycle.

The next decade will see increasing use of models in non-traditional areas and use of big data and machine learning and model risk will have to evolve and adapt. Along those lines, I have been thinking and was curious about model risk practices and approaches in industries other than banks and in particular thought of the actuarial industry which is also heavily reliant on use of models to a substantial extent. My search led me to a model risk project that was established by the ‘Institute and Faculty of Actuaries Risk Management Research and Thought Leadership Committee’  in October 2013. This project was set-up to focus on foundational areas of model risk management and culminated in a report published in 2015. This 2015 paper on model risk within actuarial industry generated lot of positive feedback and won the Peter Clark prize in 2015 which is given annually for the best paper written by members of the actuarial profession. While a lot of the views on model risk governance in the paper leverages and in some cases builds on the concepts outlined in SR 11-7, there were some unique perspectives on model risk presented in that paper which I have summarized below;

Model Risk created by the Cultural Attitude towards Model Risk: One of the most distinct points made in this paper and unique from SR 11-7 and all other other international regulatory guidance is to highlight the various psychological and cultural attitudes towards model risk amongst model users/owners/developers and the type of (non) model risk it creates. The paper divides up the population of model owners/developers into (1) Confident Model Users (2) Conscientious Modellers (3) Uncertainty Avoiders and (4) Intuitive Decision Makers based on their confidence on the model outputs (Y-axis in figure below) and their perceived legitimacy of the modeling approach (X-axis in figure below). The paper makes the argument that each of these 4 attitudes is a partial but necessary response to inherent model uncertainties and recommends that model risk management must require feedback and provide responses to each of these attitudes. Some practical suggestions to address these 4 distinct cultural attitudes within a model risk framework recommended in the paper are;

  • Embedding call kinds of challenge within decision making; irrespective of whether decisions are based on model output or on intuition
  • Identifying individuals having each of the four attitudes noted above and ensuring that, to the extent possible, over-representation of one group to the exclusion of others is avoided in model governance
  • Careful composition of governance committees: in such committees, a range of departments and rationalities should be represented, reflecting risk-specialist, technical, operational, and commercial perspectives

As practitioners, we have to acknowledge that different attitudes to model risk exist and they actually foster effective challenge and the paper recommends ways to incorporate these attitudes within model risk governance.

types of model users.png

 Expected and unexpected model error and Non-Symmetricity: Wherever the use of the model results in a financial impact, the paper recommends incorporating a baseline of the expected model error and building it into the cost of doing business. Additionally, calculation of the financial impact of higher than expected model errors should be assessed where possible and should be added to the capital/reserves. As example, the paper illustrates calculating expected model error in case a proxy model is being used (in financial services this will be analogous to the difference in MTM generated using Front Office and risk models) and if there is a benchmark model being used then measuring expected model error as the difference in model output between the primary and benchmark model. The paper also makes an interesting point that model errors are not symmetrically distributed around the baseline and makes a point that model errors are generally biased in the adverse direction. There is no evidence or support included in the paper for this statement but skeptically I am inclined to believe in that view.
Conclusion: Model risk is pervasive and persistent and inclined to grow with more use of big data and algorithms. Practitioners need to look into the next evolution of model risk and prepare for the growth in model usage and it would be sensible to look into different industries and compare notes on best practices.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s