AI Dictionary of Terms

Explainability / XAI

The ability to understand and clearly articulate how an AI model arrived at a specific decision or prediction — often referred to as Explainable AI (XAI) — bridging the gap between complex “black box” models and human interpretability.

The Simple Version

Imagine you go to a doctor, and they tell you, “You need surgery tomorrow.” If you ask why, and they say, “My medical algorithm said so, but I can’t tell you why,” you wouldn’t trust them. But if they say, “Your blood test shows X, your scan shows Y, and based on medical guidelines, this means Z,” you understand and trust the decision.

Explainability (XAI) is the AI equivalent of the doctor explaining their reasoning. Many advanced AI models (like deep neural networks) are “black boxes” — even their creators don’t know exactly why they make a specific prediction. XAI provides tools to look inside the black box and explain which factors drove the decision.

Detailed Explanation

As AI models become more complex (moving from simple decision trees to deep neural networks with billions of parameters), their accuracy increases, but their transparency decreases. XAI aims to solve this interpretability crisis.

Types of Explainability:

  1. Intrinsic Interpretability: Using models that are naturally easy to understand (e.g., Linear Regression, Decision Trees). High transparency, but often lower accuracy on complex tasks.
  2. Post-Hoc Explainability: Applying tools to a “black box” model after it makes a prediction to explain why.
    • SHAP (SHapley Additive exPlanations): Assigns an importance value to each feature for a specific prediction.
    • LIME (Local Interpretable Model-agnostic Explanations): Perturbs the input slightly to see how the output changes, fitting a simple, interpretable model locally.
    • Attention Visualization: In Transformers, showing which words the model “paid attention to” when generating an answer.

Global vs. Local Explanations:

Key Characteristics

Business Context

Explainability is no longer optional for enterprise AI; it is a business and legal necessity.

Why It Matters:

Enterprise Applications:

Real-World Analogy

A credit score. A bank doesn’t just say “Your score is 650.” They provide a breakdown: “Your score is 650. Positive factors: long credit history. Negative factors: high credit utilization, one late payment.” This breakdown is the “explainability” of the scoring model.

Code Example

# Using SHAP to explain a machine learning model's prediction
import shap
import xgboost as xgb
from sklearn.datasets import make_classification

# 1. Train a "black box" model (XGBoost)
X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
model = xgb.XGBClassifier()
model.fit(X, y)

# 2. Initialize SHAP explainer
explainer = shap.TreeExplainer(model)

# 3. Calculate SHAP values for a specific prediction (Local Explainability)
# Let's explain the prediction for the first data point
shap_values = explainer.shap_values(X[0:1])

# 4. Interpret the results
# SHAP values show how much each feature pushed the prediction 
# away from the baseline (average) prediction.
print("Features that drove this specific decision:")
for i, val in enumerate(shap_values[0]):
    if abs(val) > 0.05:  # Only show significant features
        direction = "increased" if val > 0 else "decreased"
        print(f"Feature {i} {direction} the probability of the positive class by {abs(val):.3f}")

Common Misconceptions

Sources & Further Reading