Customer Churn Prediction

A production-minded customer-churn workflow for turning behavioural signals into calibrated churn probabilities, PR-AUC model selection, recall-focused threshold tuning, interpretable importances, and batch churn scoring.

Important: This project is a portfolio and research demo, not a production churn or customer-retention decision system.

The models, thresholds, and reports are designed to demonstrate a professional modelling workflow. They should not be used for real retention, pricing, or customer-facing decisions without domain, fairness, security, and privacy review.

---

Table of Contents

• Project Overview
• What This Project Does
• What This Project Does Not Do
• Key Features
• System Workflow
• Project Structure
• Installation
• Quick Start
• Training and Evaluation
• Use Real Data (IBM Telco)
• Prediction
• Data Schema
• Evaluation Metrics
• Visual Reports
• Testing and CI
• Code Quality
• Limitations
• Responsible Use
• Future Improvements
• Tech Stack
• Author
• License

---

Project Overview

Churn prediction is not only a classification problem. In a real retention workflow, a model score is useful only if it can support a defensible action:

• rank customers by churn risk
• tune a decision threshold to the business cost of a missed churner
• report probability quality, not only ranking quality
• explain which signals drive the score
• run the same pipeline on real data, reproducibly

This project demonstrates an end-to-end churn modelling workflow on both a synthetic dataset and the public IBM Telco churn dataset. It includes schema validation, model training and selection, threshold tuning, optional probability calibration, visual diagnostics, and a batch scoring CLI.

The goal is to show how a churn model can be turned into a decision-support tool, not just a single accuracy or AUC score.

---

What This Project Does

This project can:

• Generate a synthetic churn dataset with a realistic behavioural signal
• Fetch and normalise the public IBM Telco churn dataset
• Validate input schema so any compatible CSV works (dataset-agnostic)
• Train and tune Logistic Regression, Random Forest, and Gradient Boosting
• Select the best model by validation PR-AUC (average precision)
• Tune the decision threshold for F2 with a minimum-precision floor
• Optionally isotonic-calibrate the selected model's probabilities
• Evaluate ROC-AUC, PR-AUC, Brier score, and a full classification report
• Produce ROC, precision-recall, confusion-matrix, calibration, and permutation-importance figures
• Save the trained pipeline and metrics artifacts
• Score new customers in batch with a prediction CLI
• Run automated tests and CI across Python 3.10-3.12

---

What This Project Does Not Do

This project does not:

• Make real retention, pricing, or customer-facing decisions
• Provide business, legal, or financial advice
• Guarantee fairness, compliance, or deployability
• Use a production customer dataset by default
• Provide real-time scoring infrastructure or an API
• Include live monitoring, drift detection, or retraining automation
• Perform fairness or subgroup safety analysis
• Certify that the model is safe for high-stakes deployment

A production churn system would need stronger governance, privacy controls, monitoring, fairness review, and expert validation.

---

Key Features

• Dataset-agnostic pipeline that runs on any CSV with a binary churn target
• Synthetic data generator with a reproducible, readable churn signal
• Real-data loader for the public IBM Telco churn dataset
• PR-AUC model selection across Logistic Regression, Random Forest, and Gradient Boosting
• Threshold tuning for F2 (recall-leaning) with a precision floor
• Optional isotonic calibration with a reliability diagram and Brier score
• Permutation feature importance for model-agnostic, comparable importances
• Batch scoring CLI that reuses the model's own feature schema and tuned threshold
• Schema validation that fails fast with a clear message on bad input
• Model card documenting intended use, provenance, and limitations
• Unit tests and GitHub Actions CI with coverage across Python 3.10-3.12

---

System Workflow

Customer dataset (synthetic or real)
        ↓
Schema validation
        ↓
Stratified train / validation / test split
        ↓
Preprocessing + model search (LogReg / RF / GB)
        ↓
Model selection by validation PR-AUC
        ↓
Threshold tuning (F2 with precision floor)
        ↓
Optional probability calibration
        ↓
Evaluation, figures, and saved artifacts
        ↓
Batch scoring via predict.py

---

Project Structure

Customer-Churn-Prediction/
│
├── .github/
│   └── workflows/
│       └── ci.yml
│
├── data/
│   ├── generate_customers.py
│   ├── fetch_telco.py
│   └── customers.csv
│
├── src/
│   ├── train_models.py
│   ├── predict.py
│   └── utils.py
│
├── tests/
│   ├── test_generate.py
│   ├── test_utils.py
│   ├── test_threshold.py
│   ├── test_preprocessor.py
│   ├── test_predict.py
│   ├── test_calibration.py
│   └── test_fetch.py
│
├── outputs/            (generated; git-ignored)
│
├── reports/
│   └── figures/        (versioned snapshots shown in the README)
│
├── .gitignore
├── README.md
├── MODEL_CARD.md
├── pyproject.toml
├── requirements.txt
├── requirements-dev.txt
└── LICENSE

---

Installation

1. Clone the Repository

git clone https://github.com/AmirhosseinHonardoust/Customer-Churn-Prediction.git
cd Customer-Churn-Prediction

2. Create a Virtual Environment

On Windows CMD:

python -m venv .venv
.venv\Scripts\activate

On macOS/Linux:

python -m venv .venv
source .venv/bin/activate

3. Install Requirements

pip install -r requirements.txt

For the development tools (linting, formatting, type checking, tests):

pip install -r requirements-dev.txt

---

Quick Start

No dataset on hand? Generate a synthetic one and run the whole pipeline with zero external data:

python data/generate_customers.py --n 10000 --seed 42 --out data/customers.csv
python src/train_models.py --input data/customers.csv --outdir outputs --seed 42

Prefer real data? Fetch and train on the public IBM Telco churn dataset:

python data/fetch_telco.py --out data/telco_churn.csv
python src/train_models.py --input data/telco_churn.csv --outdir outputs --seed 42

Score new customers with a trained model:

python src/predict.py --model outputs/best_model.joblib --input new_customers.csv --output outputs/predictions.csv

---

Training and Evaluation

The main pipeline splits the data, searches and selects the best model, tunes the decision threshold, evaluates on a held-out test set, and writes figures and artifacts.

python src/train_models.py \
  --input data/customers.csv \
  --outdir outputs \
  --test-size 0.2 \
  --val-size 0.2 \
  --seed 42

Add --calibrate to isotonic-calibrate the selected model on the validation set before evaluation.

Generated evaluation outputs include:

outputs/metrics.json
outputs/classification_report.txt
outputs/roc_curve.png
outputs/pr_curve.png
outputs/confusion_matrix.png
outputs/calibration_curve.png
outputs/feature_importance.png
outputs/best_model.joblib

Files under outputs/ are regenerated by the command above and are git-ignored.

---

Use Real Data (IBM Telco)

The pipeline is dataset-agnostic — any CSV with a binary churn target and at least one feature column works, and an optional customer_id is passed through. fetch_telco.py downloads the IBM sample dataset, maps customerID→customer_id and Churn→churn (0/1), and coerces TotalCharges to numeric:

python data/fetch_telco.py --out data/telco_churn.csv
python src/train_models.py --input data/telco_churn.csv --outdir outputs --seed 42

The downloaded CSV is git-ignored.

---

Prediction

Score a CSV of customers with a trained model. The prediction CLI selects the exact feature columns the model was trained on, so it works on any schema the model was trained with. The decision threshold defaults to the tuned value in outputs/metrics.json (override with --threshold), and churn is not required in the input:

python src/predict.py \
  --model outputs/best_model.joblib \
  --input new_customers.csv \
  --output outputs/predictions.csv

Output columns: customer_id (if present), churn_proba, churn_pred.

new_customers.csv is a placeholder — supply your own file, or create a quick one from existing data:

python -c "import pandas as pd; pd.read_csv('data/customers.csv').head(20).to_csv('new_customers.csv', index=False)"

The input columns must match the schema the model was trained on. A model trained on the synthetic data (age, region, tenure_months, ...) cannot score a Telco file (gender, tenure, MonthlyCharges, ...) and vice versa — retrain on the matching dataset first if the columns differ.

---

Data Schema

The synthetic generator and the reference schema use the following columns:

column	description
customer_id	unique customer ID
age	customer age
region	{North, South, East, West}
tenure_months	months since signup
is_premium	premium plan (0/1)
monthly_spend	average monthly spend
avg_txn_value	average transaction value
txns_last_30d	transactions in last 30 days
days_since_last_purchase	recency (days)
customer_service_calls	support calls in last 90 days
discounts_used_90d	discounts used in last 90 days
complaints_90d	complaint count
churn	target label (0/1)

Real datasets may use a completely different set of columns; only a binary churn target and at least one feature are required.

---

Evaluation Metrics

The evaluation layer reports metrics designed for imbalanced churn decision workflows.

Metric	Why it matters
Accuracy	Overall decision correctness at the tuned threshold
F1	Balance between churn-class precision and recall
ROC-AUC	Ranking quality across thresholds
Average precision / PR-AUC	Positive-class ranking quality under class imbalance
Brier score	Measures probability quality (calibration)
Recall (churn)	Share of real churners the policy catches
Precision (churn)	Correctness among predicted churners

Example results on synthetic data (Logistic Regression, seed 42):

Metric	Example value
Accuracy	0.838
ROC-AUC	0.823
Average precision / PR-AUC	0.562
Recall (churn)	0.50
Precision (churn)	0.52

Example results on real IBM Telco data (Gradient Boosting, seed 42):