Loading repository data…
Loading repository data…
ahmtarekragab / repository
Graduation Project - Sign Language To Text Conversion Using Python, Computer Vision and Machine Learning
A transparent discovery signal based on current public GitHub metadata.
This score does not audit code, security, maintainers, documentation quality, or suitability. Verify the repository and its current documentation before adoption.
Sign language is one of the oldest and most natural form of language for communication, but since most people do not know sign language and interpreters are very difficult to come by we have come up with a real time method using neural networks for fingerspelling based american sign language.
In this method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. This method provides 98.00 % accuracy for the 26 letters of the alphabet.
American sign language is a predominant sign language Since the only disability D&M people have is communication related and they cannot use spoken languages hence the only way for them to communicate is through sign language.
Communication is the process of exchange of thoughts and messages in various ways such as speech, signals, behavior and visuals.
Deaf and Mute(Dumb)(D&M) people make use of their hands to express different gestures to express their ideas with other people.
Gestures are the nonverbally exchanged messages and these gestures are understood with vision. This nonverbal communication of deaf and dumb people is called sign language.
Sign language is a visual language and consists of 3 major components
In this project I basically focus on producing a model which can recognize Fingerspelling based hand gestures in order to form a complete word by combining each gesture.
The gestures I trained are as given in the image below.
⦁ To complete this project, we needed the following: A local development environment for Python 3 with at least 1GB of RAM. ⦁ A working webcam to do real-time image detection.
# Importing the Libraries Required
1. Lastest pip -> pip install --upgrade pip
2. numpy -> pip install numpy
3. string -> pip install strings
4. os-sys -> pip install os-sys
5. opencv -> pip install opencv-python
6. tensorFlow -> i) pip install tensorflow
ii) pip3 install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.8.0-cp34-cp34m-linux_x86_64.whl
7. keras -> pip install keras
8. tkinter -> pip install tk
9. PIL -> pip install Pillow
10. enchant -> pip install pyenchant (Python bindings for the Enchant spellchecking system)
11. hunspell -> pip install cyhunspell (A wrapper on hunspell for use in Python)
⦁ We will build a sign language classifier using a neural network. Our goal is to produce a model that accepts a picture of a hand as input and outputs a letter. ⦁ First, we download the database to our current working directory
We built a neural network with layers, define a loss, an optimizer, and finally, optimize the loss function for our neural network predictions. At the end of this step, we will have a working sign language classifier. ⦁ Create a new file called train.py ⦁ Import the necessary utilities
import tensorflow as tf
from keras.preprocessing.image import ImageDataGenerator
import os
With Keras preprocessing layers, you can build and export models that are truly end-to-end: models that accept raw images or raw structured data as input; models that handle feature normalization or feature value indexing on their own. Image preprocessing These layers are for standardizing the inputs of an image model.
classifier = tf.keras.models.Sequential()
classifier.add(tf.keras.layers.Conv2D(filters=32,
kernel_size=3,
padding="same",
activation="relu",
input_shape=[128, 128, 1]))
classifier.add(tf.keras.layers.MaxPool2D(pool_size=2,
strides=2,
padding='valid'))
classifier.add(tf.keras.layers.Conv2D(filters=32,
kernel_size=3,
padding="same",
activation="relu"))
classifier.add(tf.keras.layers.MaxPool2D(pool_size=2,
strides=2,
padding='valid'))
classifier.add(tf.keras.layers.Flatten())
classifier.add(tf.keras.layers.Dense(units=128,
activation='relu'))
classifier.add(tf.keras.layers.Dropout(0.40))
classifier.add(tf.keras.layers.Dense(units=96, activation='relu'))
classifier.add(tf.keras.layers.Dropout(0.40))
classifier.add(tf.keras.layers.Dense(units=64, activation='relu'))
classifier.add(tf.keras.layers.Dense(units=27, activation='softmax')) # softmax for more than 2
classifier.compile(optimizer = 'adam',
loss = 'categorical_crossentropy',
metrics = ['accuracy'])
classifier.summary()
classifier.fit(training_set,
epochs = 5,
validation_data = test_set)
model_json = classifier.to_json()
with open("model_new.json", "w") as json_file:
json_file.write(model_json)
print('Model Saved')
classifier.save_weights('model_new.h5')
print('Weights saved')
import tensorflow as tf
from keras.preprocessing.image import ImageDataGenerator
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
tf.__version__
train_datagen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
test_datagen = ImageDataGenerator(rescale = 1./255)
training_set = train_datagen.flow_from_directory('E:/Graduation project/Sign-Language-To-Text-Conversion-main/dataSet/trainingData',
target_size = (128, 128),
batch_size = 10,
color_mode = 'grayscale',
class_mode = 'categorical')
test_set = test_datagen.flow_from_directory('E:/Graduation project/Sign-Language-To-Text-Conversion-main/dataSet/testingData',
target_size = (128, 128),
batch_size = 10,
color_mode = 'grayscale',
class_mode = 'categorical')
Launching our train.py file to see our neural network trains:
To obtain lower loss, we could increase the number of epochs to 10, or even 20. However, after a certain period of training time, the network loss will cease to decrease with increased training time. To sidestep this issue, as training time increases.
###4. Evaluating the Sign Language Classifier
⦁ Create a new file called Application.py ⦁ Import the necessary utilities
import numpy as np
import cv2
import os, sys
import time
import operator
from string import ascii_uppercase
import tkinter as tk
from PIL import Image, ImageTk
from hunspell import Hunspell
import enchant
from keras.models import model_from_json
⦁ First of all, we defined a class called Application then we defined init method in it ⦁ then we imported all the models we used in our application
class Application:
def __init__(self):
self.hs = Hunspell('en_US')
self.vs = cv2.VideoCapture(0)
self.current_image = None
self.current_image2 = None
self.json_file = open("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model_new.json", "r")
self.model_json = self.json_file.read()
self.json_file.close()
self.loaded_model = model_from_json(self.model_json)
self.loaded_model.load_weights("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model_new.h5")
self.json_file_dru = open("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model-bw_dru.json" , "r")
self.model_json_dru = self.json_file_dru.read()
self.json_file_dru.close()
self.loaded_model_dru = model_from_json(self.model_json_dru)
self.loaded_model_dru.load_weights("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model-bw_dru.h5")
self.json_file_tkdi = open("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model-bw_tkdi.json" , "r")
self.model_json_tkdi = self.json_file_tkdi.read()
self.json_file_tkdi.close()
self.loaded_model_tkdi = model_from_json(self.model_json_tkdi)
self.loaded_model_tkdi.load_weights("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model-bw_tkdi.h5")
self.json_file_smn = open("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model-bw_smn.json" , "r")
self.model_json_smn = self.json_file_smn.read()
self.json_file_smn.close()
self.loaded_model_smn = model_from_json(self.model_json_smn)
self.loaded_model_smn.load_weights("E:/Graduation project/Sign-Language-To-Text-Conversion-main/Models/model-bw_smn.h5")
⦁ Then we made the GUI (We made it simple, easy to use for right-handed people) ⦁ we used tkinter library to build our project interface
self.root = tk.Tk()
self.root.title("Sign Language To Text Conversion")
self.root.protocol('WM_DELETE_WINDOW', self.destructor)
self.root.geometry("900x900")
self.panel = tk.Label(self.root)
self.panel.place(x = 100, y = 10, width = 580, height = 580)
self.panel2 = tk.Label(self.root) # initialize image panel
self.panel2.place(x = 400, y = 65, width = 275, height = 275)
self.T = tk.Label(self.root)
self.T.place(x = 60, y = 5)
self.T.config(text = "Sign Language To Text Conversion", font = ("Courier", 30, "bold"))
self.panel3 = tk.Label(self.root) # Current Symbol
self.panel3.place(x = 500, y = 540)
self.T1 = tk.Label(self.root)
self.T1.place(x = 10, y = 540)
self.T1.config(text = "Character :", font = ("Courier", 30, "bold"))
self.panel4 = tk.Label(self.root) # Word
self.panel4.place(x = 220, y = 595)
self.T2 = tk.Label(self.root)
self.T2.place(x = 10,y = 595)
self.T2.config(text = "Word :", font = ("Courier", 30, "bold"))
self.panel5 = tk.Label(self.root) # Sentence
self.panel5.place(x = 350, y = 645)
self.T3 = tk.Label(self.root)
self.T3.place(x = 10, y = 645)
self.T3.config(text = "Sentence :",font = ("Courier", 30, "bold"))
self.T4 = tk.Label(self.root)
self.T4.place(x = 250, y = 690)
self.T4.config(text = "Suggestions :", fg = "red", font = ("Courier", 30, "bold"))
self.bt1 = tk.Button(self.root,