Spot the Scam - AI Job Fraud Detection

Spot the Scam is a full-stack, uncertainty-aware fraud detector for job postings. It combines calibrated classical ML, optional transformer fine-tuning, explainable outputs, and a human-in-the-loop review queue so teams can act fast with confidence.

Table of Contents

• Executive Overview
• Design Goals and Principles
• Current Model Snapshot (Repo-Grounded)
• End-to-End System Flow
• Quickstart
• Common Workflows
• Data Sources, Fields, and Footprint
• Feature Engineering and Signals
• Model Stack, Calibration, and Decision Policy
• Artifacts, Reports, and Tracking
• API Surface and Contracts
• Frontend Experience (Next.js)
• AI Chat Assistant Routing
• MLOps, Packaging, and Deployment Surfaces
• Multi-Cloud Deployment Packs
• Reproducibility and Experiment Hygiene
• Repository Map
• Documentation Map
• Limitations and Responsible Use
• FAQ
• License and Citation

Executive Overview

This repository is not just a model. It is a complete fraud-detection system with:

• Offline training produces a clear artifact contract under artifacts/.
• Online serving loads those artifacts through FraudPredictor.
• A FastAPI service exposes predictions, insights, review queues, and chat.
• A Next.js dashboard provides scoring, review, and AI-assisted analysis.
• A review loop logs predictions, captures feedback, and supports retraining with overrides.

Design Goals and Principles

The architecture reflects operational realities of trust and safety workflows:

• Precision-first defaults so alerts are actionable.
• Calibration as a first-class requirement so probabilities are trustworthy.
• A gray-zone policy so ambiguous cases route to review, not guesswork.
• Explainability on every prediction so decisions are auditable.
• Artifact-driven serving to avoid train/serve drift.

Current Model Snapshot (Repo-Grounded)

All values below are derived directly from the checked-in artifact set, especially artifacts/metadata.json and artifacts/test_predictions.csv.

Winner identity

• Model: ensemble_top3
• Model type: classical
• Feature type: tfidf+tabular
• Decision threshold: 0.5802
• Gray-zone width: 0.10
• Test calibration error (ECE): 0.0066

Metrics summary

Split	F1	Precision	Recall	ROC-AUC	PR-AUC	Brier
Validation	0.8561	0.9297	0.7933	0.9890	0.9053	0.0103
Test	0.7721	0.8537	0.7047	0.9863	0.8659	0.0143

Test confusion matrix at the selected threshold

Derived from artifacts/test_predictions.csv with the artifact threshold:

	Pred Legit	Pred Fraud
True Legit	3156	18
True Fraud	44	105

Gray-zone behavior on the test split

Using the current policy and artifacts:

• Test set size: 3,323
• Decisions: 3,200 legit, 118 fraud, 5 review
• Precision on fraud decisions (post gray-zone routing): 105/118 = 0.8898

For deeper results, see RESULTS.md and experiments/report.md.

End-to-End System Flow

The system has two tightly coupled loops connected by a stable artifact contract.

flowchart LR
    A[Raw CSVs in data/] --> B[Ingest + Preprocess]
    B --> C[Split + Persist]
    C --> D[TF-IDF + Tabular Features]
    D --> E[Classical Models + XGBoost Variants]
    D --> F[DistilBERT Fine-tune]
    E --> G[Calibration + Thresholding]
    F --> G
    G --> H[Ensembles + Selection]
    H --> I[Artifacts + Reports + Benchmarks]
    I --> J[FraudPredictor]
    J --> K[FastAPI]
    K --> L[Frontend + API Clients]
    L --> M[Review Feedback]
    M --> B

Quickstart

You have two fast paths depending on whether you prefer containers or local development.

Option A: Docker Compose (fastest end-to-end)

docker compose build
docker compose up -d

• API: http://localhost:8000
• Dashboard: http://localhost:3000

If artifacts are missing, run training first (locally or in the container).

Option B: Local development

Backend setup:

python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Train quickly (classical only):

PYTHONPATH=src python -m spot_scam.pipeline.train --skip-transformer

Serve the API:

PYTHONPATH=src uvicorn spot_scam.api.app:app --host 0.0.0.0 --port 8000 --reload

Run the frontend:

cd frontend
npm install
npm run dev

Common Workflows

The easiest way to navigate the repo is by workflow.

Workflow 1: Train and evaluate

• Command: make train or make train-fast
• Entrypoint: src/spot_scam/pipeline/train.py
• Outputs: artifacts/, experiments/, tracking/runs.csv

Workflow 2: Serve and score

• Command: make serve
• Entrypoint: src/spot_scam/api/app.py
• Runtime: src/spot_scam/inference/predictor.py

Workflow 3: Review and feedback

• Command: make serve-queue
• Review endpoints: /cases and /feedback
• Storage: tracking/predictions/ and tracking/feedback/
• Retrain with overrides: make retrain-with-feedback

Workflow 4: Tune hyperparameters

• Command: PYTHONPATH=src python scripts/tune_with_optuna.py --model-type logistic --n-trials 20
• Tuner: src/spot_scam/tuning/optuna_tuner.py
• Study DB: optuna_study.db

Minimal API smoke test

curl -X POST http://localhost:8000/predict/single \
  -H "Content-Type: application/json" \
  -d '{
        "title": "Remote Data Entry Specialist",
        "description": "We are urgently hiring... purchase laptop...",
        "requirements": "Detail oriented..."
      }'

Data Sources, Fields, and Footprint

Data sources

The pipeline merges two Kaggle-derived datasets included in data/:

• data/fake_job_postings.csv
• data/Fake_Real_Job_Posting.csv

Observed class balance (raw)

From the primary raw CSV (data/fake_job_postings.csv):

• Rows: 17,880
• Fraudulent rows: 866
• Fraud rate: 4.84%

Fields used in modeling

Key text fields (concatenated into text_all):

• title
• company_profile
• description
• requirements
• benefits

Key structured fields used for signals and missingness flags include:

• telecommuting, has_company_logo, has_questions
• employment_type, required_experience, required_education, industry, function

Split persistence for reproducibility

Training persists both indices and full snapshots:

• Indices: data/processed/split_indices.npz
• Snapshots: data/processed/train.parquet, val.parquet, test.parquet

Feature Engineering and Signals

Feature engineering is deliberately simple, auditable, and effective for this domain.

Text features (TF-IDF)

• Vectorizer: sklearn.feature_extraction.text.TfidfVectorizer
• Defaults (from configs/defaults.yaml):
  • N-grams: 1-2
  • min_df: 3
  • max_df: 0.9
  • Sublinear TF: enabled
  • Max vocabulary size: 100,000

Tabular features (engineered signals)

src/spot_scam/features/tabular.py creates signals such as:

• Text length, uppercase ratio, digit count
• Currency, exclamation, question, and URL counts
• Scam-term counters (10 configured terms by default)
• Binary metadata flags
• Missingness flags on key categorical fields

With defaults enabled, the tabular block contains 25 features.

Feature bundle contract

src/spot_scam/features/builders.py returns a FeatureBundle that includes:

• TF-IDF matrices for train/val/test
• Tabular matrices for train/val/test
• A fitted StandardScaler
• A stable feature-name list used for inference-time validation

Model Stack, Calibration, and Decision Policy

The training pipeline evaluates multiple model families under a shared decision framework.

Training pipeline (candidate generation to selection)

flowchart TD
    A[Feature Bundle] --> B[Classical Candidates]
    A --> C[XGBoost Variants]
    A --> D[Transformer Candidate]
    B --> E[Calibration + Thresholding]
    C --> E
    D --> E
    E --> F[Ensemble Candidates]
    F --> G[Best Validation F1]
    G --> H[Artifacts + Reports]

Classical candidates

Implemented in `src/spot_scam/mo