Installation

Requirements: Python 3.10 or 3.11

git clone https://github.com/fredopoku/gesturekey
cd gesturekey

pip install mediapipe opencv-python pyautogui numpy scikit-learn PyQt6

On macOS you may also need:

pip install pyobjc-framework-Quartz

Running

python3 gesturekey_ui.py

The camera opens immediately. If you already have a trained model it activates automatically — the header badge shows ML XX%. Without a trained model it falls back to the built-in rule-based classifier instantly.

Training your personal model

The ML model is trained on your hands, not a generic dataset. This is why accuracy is high.

In the app:

1. Click the Train tab in the right panel
2. Click Rec next to any gesture — a 3-second countdown starts
3. Hold the gesture steady for ~10 seconds (60 samples captured automatically)
4. Repeat for as many gestures as you want
5. Click ▶ Train Model — training runs in ~15 seconds
6. The model activates immediately. The header badge switches to ML XX%

Each time you retrain, all recorded samples are used — old data is never lost. The more gestures and samples you add, the better the model gets.

From the command line (original trainer):

python3 gesture_trainer.py

Gesture reference

Letters (ASL-inspired)

Selection gestures (word prediction)

Word gestures

Architecture

Camera (OpenCV)
    │
    ▼
MediaPipe Hands  →  21 landmarks (x, y, z)
    │
    ▼
Normalise  →  wrist-centred, scale-invariant 63-dim vector
    │
    ├── GestureMLModel (sklearn MLP 256→128→64)   ← your personal model
    │       confidence ≥ 72% → use result
    │
    └── Rule-based classifier                      ← always available fallback
    │
    ▼
GestureMapper  →  gesture name → character / command
    │
    ├── WordPredictor (Trie + bigram scoring)  →  5 candidates
    │
    └── TextOutput (pyautogui)  →  types into active OS window

Model files:

data/
  vocabulary.json          word frequency table
  bigrams.json             context pairs for prediction
  trie.pkl                 cached prefix tree
  gesture_mlp.pkl          your trained MLP model
gesture_training_data.json raw keypoint samples per gesture

Roadmap

• Rule-based gesture classifier
• MediaPipe hand tracking
• PyAutoGUI text injection (all OS apps)
• Word prediction — Trie + bigram engine
• PyQt6 desktop UI — camera, guide, prediction bar, composer
• In-app MLP training pipeline
• Hybrid ML + rule classifier with hot-reload
• Full A–Z trained model (recording in progress)
• TFLite model export — cross-platform inference
• Web version — MediaPipe.js, runs in any browser on any device
• Packaged installers — .app (macOS), .exe (Windows)
• Android app — MediaPipe SDK + Accessibility Service
• iOS app — MediaPipe iOS + system keyboard extension
• Cloud model sync — train once, use everywhere
• Custom gesture vocabulary — define your own mappings
• LLM-powered word prediction

Project structure

gesturekey/
├── gesturekey_ui.py          main application (PyQt6)
├── gesture_mapper.py         hybrid ML + rule classifier
├── gesture_model.py          MLP training and inference engine
├── gesture_trainer.py        CLI gesture recording tool
├── word_predictor.py         Trie + bigram word prediction
├── text_output.py            pyautogui text injection layer
├── gesture_training_data.json captured keypoint samples
└── data/
    ├── vocabulary.json
    ├── bigrams.json
    ├── trie.pkl
    └── gesture_mlp.pkl       trained model (created after first training)

Author

Frederick Opoku-Afriyie
MSc Computer Science, 2025

GestureKey concept, system design, and implementation.

Built with:

• MediaPipe Hands — Google Research (Zhang et al., 2020)
• OpenCV — Intel / OpenCV Foundation
• PyQt6 — Riverbank Computing
• scikit-learn — Pedregosa et al., 2011
• PyAutoGUI — Al Sweigart

GestureKey — a universal gesture input paradigm for accessible digital communication.