American Sign Language (ASL) Real-Time Detection

Real-time American Sign Language recognition from a webcam feed. MediaPipe tracks 21 hand landmarks per frame, and a small neural network classifies the normalized landmark positions into a letter or common phrase — no image classification, just geometry, so it runs fast on CPU.

Recognizes the full ASL alphabet (A-Z) plus 10 common phrase signs: HELLO, MY, NAME, YOUR, WHAT, HOW, YOU, FINE, THANK-YOU, PLEASE.

Highlights

• 36-class hand-sign classifier trained on ~36,000 labeled hand-landmark samples I collected and captured myself
• 86.1% validation accuracy with a compact dense network (BatchNorm + Dense(128) + Dense(64), Mish activation, dropout + L2 regularization)
• Exported to both Keras (.keras) and quantized TensorFlow Lite (.tflite) for fast CPU inference
• Prediction smoothing via a rolling-mode filter over the last 5 frames, so a single misread frame doesn't flicker the on-screen label
• Includes the full data pipeline: live webcam capture mode and batch capture from an image dataset, both feeding the same landmark-logging CSV used for training

How It Works

1. Detect — MediaPipe Hands locates up to 2 hands per frame and returns 21 (x, y) landmarks each
2. Normalize — landmarks are shifted relative to the wrist point and scaled by the max absolute coordinate, so the model is invariant to hand position/size in frame
3. Classify — the 42-value normalized vector (21 landmarks x, y) is fed through the trained dense network to predict one of 36 classes
4. Smooth — predictions are held in a 5-frame rolling window and the mode is displayed, reducing frame-to-frame jitter

Project Structure

AI-Sign-language/
├── main.py                                   # Live recognition app + dataset/landmark capture modes
├── app.py                                     # Bonus: standalone face-detection/verification demo
├── keypoint_classification.ipynb              # Model training notebook (data load -> train -> evaluate -> export TFLite)
├── utils/
│   └── cvfpscalc.py                           # FPS counter utility
├── model/keypoint_classifier/
│   ├── keypoint.csv                           # ~36k labeled landmark samples (training data)
│   ├── keypoint_classifier.keras               # Trained Keras model
│   ├── keypoint_classifier.tflite               # Quantized TFLite export
│   ├── keypoint_classifier.py                  # TFLite inference wrapper
│   └── keypoint_classifier_label.csv            # Class labels (A-Z + phrases)
├── assets/                                     # Demo/placeholder images
└── requirements.txt