Here is a demo of the Sign Language Recognition System in action:

Customization

The Sign Language Recognition System is fully customizable and can be trained to recognize any hand gesture. To customize the system, follow these steps:

1. Run the app.py file to open the application.
2. Press the 1 key to switch to the Data Collection mode. Press the 0 key if you want to switch to the prediction mode.
3. Make the hand gesture you want to add to the dataset.
4. Press the alphabate key to save the gesture to the dataset corresponding to the letter.
5. Add as many gestures as you want to the dataset. More data will result in better accuracy.
6. After adding all the gestures, press the esc key to quit program. Then run the python train.py to train the model on the new dataset.

Enjoy your customized Sign Language Recognition System!

System Overview

In order to build such systems, data (keypoints of hands) must be obtained, then features involved in sign making must be extracted and finally combination of features must be analysed to describe the performed sign.

Data Collection

The first step in building a sign language recognition system is to collect a dataset of sign language gestures. The dataset is used to train the machine learning model to recognize and interpret these gestures.

For this project we use our hand sign data. And the data is collected using the MediaPipe library. The library provides a hand tracking solution that can detect and track 21 hand landmarks in real-time. The hand landmarks are the key points or 2D coordinates that can be used to determine the pose of the hand.

The hand landmarks are used to extract features from the hand gestures. The features are then used to train a machine learning model to recognize and interpret these gestures.

Preprocessing

The collected data is preprocessed to enhance the quality, remove noise, and extract relevant features. The preprocessing steps include:

1. Getting the hand landmarks from the video stream.

2. Converting the hand landmarks relative to the wrist landmark's coordinate (0, 0). This is done by subtracting the wrist landmark's coordinate from all the other landmarks.

3. Flatten the normalized hand landmarks to a 1 Dimensional list.

   #: Convert into a one-dimensional list
   temp_landmark_list = list(itertools.chain.from_iterable(temp_landmark_list))
   

4. Normalizing the hand landmarks from -1 to 1 value.

   # Normalize the hand landmarks to a fixed size
   #: Normalization (-1 to 1)
   max_value = max(list(map(abs, temp_landmark_list)))
   
   def normalize_(n):
       return n / max_value
   
   landmark_list = list(map(normalize_, temp_landmark_list))
   

5. Adding the normalized hand landmarks to the dataset.

   #: Writing dataset
   with open("slr/model/keypoint.csv", 'a', newline="") as f:
       writer = csv.writer(f)
       writer.writerow([index, *landmark_list])
   

Model Training

The preprocessed data is used to train a machine learning model to recognize and interpret sign language gestures. The model is trained using a convolutional neural network (CNN) architecture. The CNN is trained on the preprocessed data to learn the mapping between input gestures and their corresponding meanings.

1. Obtain or create a sign language dataset.

2. Split the dataset into training and validation sets. It is recommended to use a separate testing set for final evaluation.

3. Choose an appropriate machine learning algorithm or architecture, such as CNNs, RNNs, or their combinations, and implement it using a suitable framework like TensorFlow or PyTorch. For this project, we use a CNN architecture implemented using TensorFlow.

4. Train the model using the training dataset and tune hyperparameters to achieve optimal performance.

5. Validate the model using the validation dataset and make necessary adjustments to improve accuracy and generalization.

6. Test the final model using the testing dataset and evaluate its performance using appropriate metrics.

Results

The model was trained on a dataset of 24,000 hand gestures. The dataset was split into training and validation sets with a ratio of 80:20. The model was trained for 100 epochs with a batch size of 180. The training and validation accuracy and loss were recorded for each epoch.

Our Proposed Model achieved an accuracy of 71.12% on the validation set and 90.60% on the testing set. The model was able to recognize and interpret sign language gestures in real-time with an accuracy of 71.12%.

Contributing

Contributions to this project are welcome. If you encounter any issues or have suggestions for improvements, please open an issue or submit a pull request. Let's work together to make the Sign Language Recognition System even better!

We appreciate your contributions, whether big or small, and we look forward to working together to enhance the Sign Language Recognition System. Let's make a positive impact on the lives of individuals with hearing impairments and promote inclusivity in communication.