Introduction to Explainable AI (XAI)
What is AI?
Artificial intelligence means creating machines or algorithms capable of performing tasks similar to humans. These machines are trained on data and use complex ML algorithms to make decisions on their own. AI and ML are used in every area nowadays. Data is used everywhere to solve problems and understand the lying pattern behind it. As data increases, the complexity of models is also increasing, and understanding the logic behind the decision-making process of models takes work. That’s why these models are called black boxes. Machine learning development services is used in healthcare and safety environments. So understanding the reason behind the decision-making of ML models becomes important. With an understanding of the logic used and how a machine learning model concluded, users may trust them.
Introduction to Explainable AI (XAI)
Explainable AI (XAI) refers to techniques and methods that help humans understand and trust the decisions made by machine learning models. With the increasing adoption of AI in critical fields like healthcare, finance, and autonomous systems, the need for explainability has become crucial. XAI aims to bridge the gap between the “black-box” nature of AI models and human interpretability.
OR
What is Explainable AI?
Consider a production line in which workers run heavy, potentially dangerous equipment to manufacture steel tubing. Company executives hire a team of machine learning (ML) practitioners to develop an artificial intelligence (AI) model that can assist the frontline workers in making safe decisions, with the hopes that this model will revolutionize their business by improving worker efficiency and safety. After an expensive development process, manufacturers unveil their complex, high-accuracy model to the production line expecting to see their investment pay off. Instead, they see extremely limited adoption by their workers. What went wrong?
This hypothetical example, adapted from a real-world case study in McKinsey’s The State of AI in 2020, demonstrates the crucial role that explainability plays in the world of AI. While the model in the example may have been safe and accurate, the target users did not trust the AI system because they didn’t know how it made decisions. End-users deserve to understand the underlying decision-making processes of the systems they are expected to employ, especially in high-stakes situations. Perhaps unsurprisingly, McKinsey found that improving the explainability of systems led to increased technology adoption.
Explainable artificial intelligence (XAI) is a powerful tool in answering critical How? and Why? questions about AI systems and can be used to address rising ethical and legal concerns. As a result, AI researchers have identified XAI as a necessary feature of trustworthy AI, and explainability has experienced a recent surge in attention. However, despite the growing interest in XAI research and the demand for explainability across disparate domains, XAI still suffers from a number of limitations. This blog post presents an introduction to the current state of XAI, including the strengths and weaknesses of this practice.
Purpose of XAI
The main objectives of XAI are:
- Transparency: Offering insights into how an AI model processes data and reaches conclusions.
- Trustworthiness: Building confidence in AI by making its decisions understandable.
- Compliance: Ensuring AI decisions meet regulatory standards.
- Debugging and Model Improvement: Providing feedback for model tuning and improvement.
Key Techniques in XAI
- SHAP (SHapley Additive exPlanations)
- LIME (Local Interpretable Model-agnostic Explanations)
1. SHAP (SHapley Additive exPlanations)
SHAP is based on cooperative game theory and assigns a contribution value to each feature by considering all possible combinations. This method quantifies how much each feature contributes to a model’s prediction.
- Mathematical Intuition: SHAP values are calculated as:ϕi=∑S⊆N∖{i}∣S∣!(∣N∣−∣S∣−1)!∣N∣![f(S∪{i})−f(S)]\phi_i = \sum_{S \subseteq N \setminus \{i\}} \frac{|S|!(|N|-|S|-1)!}{|N|!} \left[ f(S \cup \{i\}) – f(S) \right]ϕi=S⊆N∖{i}∑∣N∣!∣S∣!(∣N∣−∣S∣−1)![f(S∪{i})−f(S)]where:
- ϕi\phi_iϕi: SHAP value for feature iiiSSS: Subset of features excluding iiiNNN: Set of all featuresf(S)f(S)f(S): Model output for subset SSS
-
import shap import xgboost as xgb import numpy as np # Visualize SHAP values shap.summary_plot(shap_values, X)# Calculate SHAP values explainer = shap.Explainer(model, X) shap_values = explainer(X)-
# Train a model X, y = shap.datasets.boston() model =xgb.XGBRegressor().fit(X, y)
2. LIME (Local Interpretable Model-agnostic Explanations)
LIME explains a single prediction by fitting a simple, interpretable model (e.g., linear regression) locally around the instance being explained. It provides insights into what features contributed most to a specific prediction.
- Mathematical Intuition: LIME approximates the complex model fff with an interpretable surrogate model ggg in the neighborhood πx(z)\pi_x(z)πx(z) of a given instance xxx. The optimization problem for LIME can be formulated as:argming∑z∈Zπx(z)(f(z)−g(z))2+Ω(g)\text{argmin}_g \sum_{z \in Z} \pi_x(z) \left( f(z) – g(z) \right)^2 + \Omega(g)argmingz∈Z∑πx(z)(f(z)−g(z))2+Ω(g)where:
- πx(z)\pi_x(z)πx(z): Proximity measure between zzz and xxxΩ(g)\Omega(g)Ω(g): Complexity penalty for the interpretable model ggg
- Python Code for LIME Explanation: pythonCopy code
import lime import lime.lime_tabular from sklearn.ensemble import RandomForestClassifier from sklearn.datasets import load_iris # Load data and train a model iris = load_iris() model = RandomForestClassifier(n_estimators=50) model.fit(iris.data, iris.target) # Create a LIME explainer explainer = lime.lime_tabular.LimeTabularExplainer( training_data=iris.data, feature_names=iris.feature_names, class_names=iris.target_names, mode='classification' ) # Explain a single prediction exp = explainer.explain_instance(iris.data[0], model.predict_proba, num_features=2) exp.show_in_notebook()
import shap
import xgboost as xgb
import numpy as np
# Train a model
X, y = shap.datasets.boston()
model = xgb.XGBRegressor().fit(X, y)
# Calculate SHAP values
explainer = shap.Explainer(model, X)
shap_values = explainer(X)
# Visualize SHAP values
shap.summary_plot(shap_values, X)
Key Differences Between SHAP and LIME
| Feature | SHAP | LIME |
|---|---|---|
| Approach | Global and consistent explanations | Local, instance-specific explanations |
| Theory Basis | Game theory (Shapley values) | Model agnostic, local surrogate models |
| Complexity | Higher computational complexity | Faster but approximate |
| Interpretability | Provides feature contributions for each prediction | Provides explanations for specific instances |
Applications of XAI
- Healthcare: Explaining diagnosis or treatment recommendations from AI models.
- Finance: Clarifying why a credit score was approved or declined.
- Autonomous Systems: Ensuring decisions made by self-driving cars are interpretable for safety compliance.
Current Limitations of XAI
One obstacle that XAI research faces is a lack of consensus on the definitions of several key terms. Precise definitions of explainable AI vary across papers and contexts. Some researchers use the terms explainability and interpretability interchangeably to refer to the concept of making models and their outputs understandable. Others draw a variety of distinctions between the terms. For instance, one academic source asserts that explainability refers to a priori explanations, while interpretability refers to a posterio explanations. Definitions within the domain of XAI must be strengthened and clarified to provide a common language for describing and researching XAI topics.
In a similar vein, while papers proposing new XAI techniques are abundant, real-world guidance on how to select, implement, and test these explanations to support project needs is scarce. Explanations have been shown to improve understanding of ML systems for many audiences, but their ability to build trust among non-AI experts has been debated. Research is ongoing on how to best leverage explainability to build trust among non-AI experts; interactive explanations, including question-and-answer based explanations, have shown promise.
Another subject of debate is the value of explainability compared to other methods for providing transparency. Although explainability for opaque models is in high demand, XAI practitioners run the risk of over-simplifying and/or misrepresenting complicated systems. As a result, the argument has been made that opaque models should be replaced altogether with inherently interpretable models, in which transparency is built in. Others argue that, particularly in the medical domain, opaque models should be evaluated through rigorous testing including clinical trials, rather than explainability. Human-centered XAI research contends that XAI needs to expand beyond technical transparency to include social transparency.
Conclusion
XAI is essential for making AI decisions transparent, interpretable, and trustworthy. With tools like SHAP and LIME, practitioners can gain insights into the inner workings of complex models, fostering a higher degree of confidence in their outputs.
By integrating XAI methods, organizations can align their AI solutions with ethical standards and regulatory requirements while enhancing model reliability.