AI Dictionary of Terms

Model Monitoring / Drift Detection

The continuous practice of tracking AI model performance in production to detect degradation caused by changes in input data distributions (data drift) or changes in the relationship between inputs and outputs (concept drift) — ensuring models remain accurate and reliable over time.

The Simple Version

Imagine you trained a weather prediction model using data from 2020-2024. It worked perfectly. But in 2025, climate patterns shifted dramatically due to a major El Niño event. Your model, trained on “normal” years, starts making wildly inaccurate predictions — not because it’s broken, but because the world it was trained on no longer exists.

This is drift — when the real world diverges from the data the model learned from. Model monitoring is the system that constantly checks: “Is the world the model was trained on still the world we’re living in?” When drift is detected, it’s time to retrain or recalibrate the model.

Detailed Explanation

Unlike traditional software, which behaves consistently unless the code changes, ML models can silently degrade as the world changes around them. Monitoring catches this degradation before it causes business harm.

Types of Drift:

1. Data Drift (Feature Drift / Covariate Shift):

2. Concept Drift (Label Drift):

3. Prediction Drift:

4. Upstream Data Drift:

The Monitoring Stack:

1. Metrics Collection:

2. Drift Detection:

3. Alerting & Action:

Popular Monitoring Tools:

Key Characteristics

Business Context

Model monitoring is the insurance policy for production AI:

Why It Matters:

Real-World Drift Scenarios:

E-commerce Pricing Model:

Healthcare Diagnostic Model:

LLM Customer Support Bot:

ROI of Monitoring:

Real-World Analogy

A car’s dashboard. You don’t just drive and hope everything is fine — you monitor the fuel gauge, engine temperature, oil pressure, and warning lights. When a light comes on, you investigate before the car breaks down. Model monitoring is the dashboard for AI systems, giving you early warning of problems before they become failures.

Code Example

# Drift detection using Evidently AI
import pandas as pd
import numpy as np
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
from sklearn.datasets import make_classification

# Generate reference data (training data)
np.random.seed(42)
X_ref, y_ref = make_classification(n_samples=10000, n_features=10, random_state=42)
reference_data = pd.DataFrame(X_ref, columns=[f"feature_{i}" for i in range(10)])
reference_data["target"] = y_ref

# Generate current production data with drift in 2 features
X_cur, y_cur = make_classification(n_samples=10000, n_features=10, random_state=43)
current_data = pd.DataFrame(X_cur, columns=[f"feature_{i}" for i in range(10)])
current_data["target"] = y_cur

# Introduce drift: shift feature_0 and feature_3 distributions
current_data["feature_0"] = current_data["feature_0"] + 2.0  # Mean shift
current_data["feature_3"] = current_data["feature_3"] * 1.5  # Variance change

# Create drift detection report
report = Report(metrics=[
    DataDriftPreset(),
])

# Run the report
report.run(reference_data=reference_data, current_data=current_data)

# Get results
drift_result = report.as_dict()

# Check if drift was detected
dataset_drift = drift_result["metrics"][0]["result"]["dataset_drift"]
print(f"Dataset drift detected: {dataset_drift}")

# Show per-feature drift
for feature_drift in drift_result["metrics"][0]["result"]["drift_by_columns"]:
    feature_name = feature_drift["column_name"]
    drift_detected = feature_drift["drift_detected"]
    drift_score = feature_drift["drift_score"]
    print(f"  {feature_name}: drift={drift_detected}, score={drift_score:.4f}")

# Output will show feature_0 and feature_3 have significant drift
# This triggers an alert to investigate and potentially retrain the model

Common Misconceptions

Sources & Further Reading