Model Explainability: LIME & SHAP

Photo by Caleb Woods on Unsplash

Model explainability enhances human trust in machine learning. As the complexity level of a model goes up, it becomes increasingly challenging to interpret feature importance at the model level (i.e. global level). Instead, people focus more on prediction-level explainability for complex models (i.e. local level). This post is going to discuss about the two popular model-agnostic explanation models, LIME and SHAP. It consists of the following sections:

• Explanation Model
• LIME
• SHAP
• Hands-on Exercise
• Conclusion

Explanation Model

Far more complex than linear models, many machine learning algorithms are challenging to be human-interpretable. Therefore, besides development of the key model on the question of interest, it is also worthwhile to develop another model specifically to understand that key model. This second model is what we called “explanation model”.

Lundberg and Lee (2017) mathematically described the explanation model below, where:

• g(.) is the explanation model, and its input is x’ (simplified input)
• f(.) is the original model (key model), and its input is x
• x is a single input, x’ is a simplified input (normally binary), and x=h_x(x’). Note: this mapping function depends on x.
• h_x(.) converts a vector of 1’s or 0’s (present or not) into the original input if the simplified input is 1, or background value if the simplified input is zero
• M is the number of simplified input features
• g(.) is normally a linear function, and the coefficients ϕ_i is related to the local contribution by feature i in x

In short, a local explanation model is a linear model (as we know, linear model is easy to interpret) we train to make it generate predictions as close as possible to the original model (key model) around an input x.

LIME: Local Interpretable Model-agnostic Explanations

LIME was first published in 2016 by Ribeiro, Singh and Guestrin. It is an explanation technique that interprets an individual prediction locally.

The idea of LIME is illustrated in Figure 1. Given a classification model, it generates binary predictions (“blue” and “red” background). To interpret the local contribution of features to a specific prediction (“bold red cross”), it generally involves 3 steps:

• Sample around the original input and obtain corresponding predictions (i.e. “red cross” and “blue circle”)
• Calculate the weights which measure the proximity of a sample point to the original input. The closer the larger (i.e. size of red cross and blue circle)
• Train a linear regression which minimizes the weighted distance between the linear model and the sample predictions.

Figure 1. Intuition of LIME (Edited from Ribeiro, Singh and Guestrin, 2016)

SHAP: SHapley Additive exPlanations

SHAP was first published in 2017 by Lundberg and Lee (also from University of Washington, same as LIME). To understand the idea of SHAP, we need to know what is Shapley values first.

What is Shapley value?

Shapley value comes from game theory, which is the average expected marginal contribution of one player in all possible combinations.

The post by Fernando provides a very nice graphic way to illustrate. Say, there are 4 persons who form a team to make a profit. To understand how much “red hat” contributes, we need to calculate the difference in profits when “red hat” is in the team and not in the team under 4 scenarios:

• the team has only 1 person
• the team has 2 persons
• the team has 3 persons
• the team has 4 persons

The contribution of “red hat” is the average difference in profits for all team combinations (i.e. “coalitions”) under these 4 scenarios, i.e. “Shapley value”. However, when the original model is complicated (with many features), it will be quite computationally intensive to calculate Shapley value. That is why SHAP value is introduced.

Figure 2. Graphic Illustration on Shapley Values (From Fernando’s Post)

What is SHAP Value?

The logics of Shapley value is straightforward, but its calculation effort could go exponentially as the number of coalitions grows. To alleviate this issue, instead of looping over all the coalitions, SHAP uses the coalition sample in the similar spirit of LIME:

• Sample within different coalitions
• Calculate the weights for different coalitions
• Train a linear model which minimizes the weighted distance between the linear model and the sample predictions

The final linear coefficients are SHAP values.

Hands-on Exercise

The following shows how to use python to carry out LIME and SHAP explanation. There are two limitations of LIME: it cannot do batch explanation and it cannot do multi-label explanation directly.

Tree-based Classification Model (“xgb_mod”)

The scripts below shows how LIME works and how to get LIME values and visualisation.

Figure 3. How LIME Works in Python

The scripts below shows how SHAP works and how to get SHAP values and visualisation.

Figure 4. How SHAP Works in Python

Multi-Regression Neural Network (“model”)

The neural network will generate 5 outputs which are all continuous. The scripts below shows how LIME works and how to get LIME values.

Figure 5. How LIME Works in Python

The scripts below shows how SHAP works and how to get SHAP values.

Figure 6. How SHAP Works in Python

Conclusion

As a summary, SHAP normally generates explanation more consistent with human interpretation, but its computation cost will be much higher as the number of features goes up. LIME invokes many ideas in AI explainability, and it is relatively less costly than SHAP in computation when the number of features is large.

Reference

SHAP (SHapley Additive exPlanations) And LIME (Local Interpretable Model-agnostic Explanations)…

Explainable AI(XAI) — A guide to 7 packages in Python to explain your models

SHAP explained the way I wish someone explained it to me

Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.

Ribeiro, M. T., Singh, S., & Guestrin, C. (2016, August). “ Why should i trust you?” Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135–1144).

SHAP: Shapley Additive Explanations