Piercing the Black Box: SHAP
Introduction
One common criticism of machine learning and AI is that the reasoning behind its output can be obscure. This is what's known as the “black box” of AI. However, despite this common misconception, AI & machine learning models can be understood through a variety of techniques that examine the importance of the features that go into a model. Methodologies such as SHapley Additive exPlanations (SHAP), Local Interpretable Model-Agnostic Explanations (LIME), and Explain Like I’m Five (ELI5) can be used to pierce the black box. In this blog post, we will discuss how SHAP can be used to provide digestible insights into why AI models make their predictions.
SHAP: A Game-Theory Approach to Model Interpretation
SHAP (SHapley Additive exPlanations) values are inspired by the concept of Shapley values from cooperative game theory, which provides a way to fairly distribute the total gains (or losses) among the players based on their contributions. This is a game-theory approach used to explain the output of machine learning models. In the context of machine learning, SHAP values quantify the contribution of each feature to the prediction made by the model. By doing so, they help to demystify complex models, making it easier to understand how individual features, the types of data that a model uses to make its predictions, influence the model's decisions. This makes SHAP values a powerful tool for improving transparency and trust in AI systems, as they provide clear, mathematically grounded insights into the inner workings of the models.
Beeswarm Plots
A Beeswarm plot uses SHAP to better understand a model’s output:
https://www.aidancooper.co.uk/a-non-technical-guide-to-interpreting-shap-analyses/
In this Beeswarm plot above, we are visualizing a model that predicts housing prices in Boston. The y-axis shows the various features (input variables) that the model uses to make predictions, such as the percentage of the working-class population, the average number of rooms per dwelling, and the crime rate. These features are ordered by importance, with the most influential feature at the top. Each point on the Beeswarm plot represents a single data point that the model was trained on. The color of the point corresponds to the relative value of said point. Red being high and blue being low values. For example, we can observe that when the most important feature, “% working class”, value is high (red), the model is more likely to predict a lower home price and vice versa. This allows us to understand why a model is making its decisions broadly.
Explaining Individual Predictions with Force Plots
In addition to being able to use SHAP values to understand a model generally, we can utilize SHAP values for understanding individual predictions and classifications.
https://www.aidancooper.co.uk/a-non-technical-guide-to-interpreting-shap-analyses/
Here, we can see a SHAP force plot for an individual prediction. The segments on the bar represent the feature importance of each feature in the model. The direction and color of the segments show how each feature ‘pushes’ the model towards a positive or negative prediction.
In using beeswarm and force plots, we can better understand classification and regression machine learning models. We can also use SHAP to better understand models that have text-based features. The following example below shows an example of a sentiment analysis model run on a movie review.
Text-Based Models
https://shap.readthedocs.io/
Here, we can see which section of text contributes towards a positive sentiment classification (the red sections) and which sections contribute towards a more negative sentiment (the blue sections).
Decoding Image Classifications with SHAP
Additionally, SHAP can also be used for explaining the classifications of image models.
https://shap.readthedocs.io/
In this example, we see which sections of the image contribute towards a specific classification by the model.
Conclusion
By utilizing SHAP and other feature analysis and model explanation techniques, we can gain a broader understanding of why the models make their predictions and thus gain more confidence in them.
At Delphi Intelligence, we pride ourselves in not just creating efficient and precise models, but also in providing transparency and explainability.
