Explainable Artificial Intelligence or XAI

In recent years, we are increasingly being influenced by machine learning models, most of the time without even realizing it.

When you get recommendations for movies from Netflix, it is all done by a special class of machine learning models called recommender systems. Netflix actually has a nice page showcasing their research and usage of machine learning models:


When you visit the bank next time and applying for a loan, the decision about outcome may be actually made not by a human but by a ML model.

Another important area of our lives impacted by AI is in medical diagnosis. The latter is increasingly being driven by AI models, which learn on vast data sets with millions of historical diagnosis. The doctor of the future may still be a human but only explaining a suggestion or diagnosis made by an AI model.

As the number of decisions made by AI models or machines rise, we are increasingly confronted with an important question: How do AI models make decisions? Why was this person particular rejected when applying for the bank loan?

A large issue in these questions is that most of the machine learning models are more or less black boxes. Although some simpler ones, like logistic regression, decision trees and linear regression are more easily understood, the more complex ones like gradient boosting machines and deep learning nets are mostly too complex to have an easy interpretation for them.

This has led to the development of special methods that help us interpret even the most complex ML models. As these methods do not depend on the specifics of a particular ML method, they are known as model agnostic methods. The are an important class of method for data science consultants.

Whatever the problem and the ML algorithms we can apply them to know trained ML model better and also compare them.

Some of the most well known Explainable Artificial Intelligence or XAI methods are LIME, SHAP and Permutation Feature Importance.

Permutation Feature Importance works the following way: to find out how important is the feature F for the ML model prediction, one permutates the values of these feature and recomputes the model’s error. If we see a jump in the error, we deem the feature important.

LIME approach works a bit differently. In this case, one trains an approximate linear model around the data instance, the prediction of which we are trying to explain. Linear models are namely more easily interpretable, one just needs to look at coefficients of linear model. One can then use this simplified linear model to explain which feature values most contributed to the outcome.

SHAP is based on Shapley values introduced in the 1950s by Shapley. Shapley is the recipient of the Nobel Prize.

Shapley values of feature values tell us how much did a given feature value contribute the “net” prediction (prediction in average baseline prediction). SHAP method is the one that is theoretically most rigorously defined. SHAP library even has a method specialized for deep learning nets: Deep Explainer. It has connection with DeepLift.

These methods can help us to understand two things about ML predictions:

  • which features are most important in ML model
  • for given instance, which feature values were most important contributors to outcome probability and in what direction (positive, negative)