Sign Language Recognition Using Multiple Kernel Learning: A Case Study of Pakistan Sign Language

Shah, Farman; Shah, Muhammad Saqlain; Akram, Waseem; Manzoor, Awais; Mahmoud, Rasha; AbdElminaam, Diaa Salama

doi:10.1109/access.2021.3077386

Cited by 44 publications

(19 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dardas, e.t.al7 , used the Bag-of-features technique with SIFT and SVM to obtain 96.23% accuracy using 10 signs of static American SL with cluttered background. Another study by Farman Shah, e.t.al23 , used SURF with SVM but obtained 15.41% accuracy and the final reported accuracy using Histogram of Oriented Gradient (HOG) and SVM was 91.98%, which was also the highest classification accuracy reported, to the best of my knowledge, using static PSL alphabets. Shazia Saqib, e.t.al24 , used dynamic PSL words with CNN with Levenshtein distance to obtain 90.79% accuracy.The highest classification accuracy obtained in this study for static PSL signs, was 97.80% as compared to 96.23% by Nasser H. Dardas, e.t.al7 , who used 1,000 static American SL images with non-uniform lighting, background, scale and rotation, and 15.41% by Farman Shah, e.t.al23 , who used SURF directly with SVM, instead of using the BoW technique.…”

mentioning

confidence: 85%

“…Most of these studies extract specific features and then use machine learning algorithms to classify the SL images. Many different SL have been used in these studies, namely American SL [1][2][3][4][5][6][7][8] , Arabic SL [9][10][11][12] , British SL 13 , Chinese SL 14 , German SL 15,16 , Indian SL 17 , Irish SL 18 , Pakistani SL [19][20][21][22][23][24] , Persian SL 25 , and more in combination such as American & German SL 26 , American & Thai SL 27 and American & Indian SL 28 .…”

Section: Literature Reviewmentioning

confidence: 99%

“…Specifically, those systems that used images and videos from a single camera of bare hands, instead of those that used multiple cameras or different object tracking technologies, were used for this study.Many systems used a combination of image and video-based datasets as input and used different classifiers, such as, Neural Networks like Convolutional Neural Network (CNN) and Multilayer Perceptron (MLP), Support Vector Machine (SVM), K Nearest Neighbor (KNN), Hidden Markov Model (HMM), etc.Saqib, e.t.al, used 20 dynamic PSL words, with 8,000 videos (6,480 training /1,520 testing) collected from 15 participants, resized the images to 234×234 and converted them to grayscale, and used CNN with Convolution layers and fully connected layers, along functional layers such as max pooling Layers, Rectified Linear Units layer (ReLU layer) and SoftMax activation function to achieve a 90.79% accuracy24 . Shah, e.t.al, classified 36 PSL alphabets, with 6,633 images (4,643 training /1,990 testing) collected from 6 participants using SVM and using K-means clusteringbased segmentation and converting them to grayscale, obtained classification accuracies of 15.41% using Speeded Up Robust Features (SURF), 87.67% using Edge Orientation Histogram (EOH), 45.71% using Local Binary Patterns (LBP), and 89.52% using Histogram of Oriented Gradient (HOG) and the final reported accuracy of 91.98%23 .…”

mentioning

confidence: 99%

“…The protocol used by the researchers of all the included PSL studies used RGB images and single-handed static signs of PSL alphabets except for Saqib, e.t.al, who used dynamic PSL words24 . The studies used various lighting conditions and studies by Kausar, e.t.al19 , and Shah, e.t.al23 , mentioned that the clothing should be separate from the skin colour of the participant. Khan, e.t.al22 , and Ahmed, e.t.al21 , used complex backgrounds to collect the data while the rest used uniform backgrounds.Khan, e.t.al, collected a total of 500 (426 training / 74 testing) images of 37 PSL alphabets, converted the RGB images to grayscale, segmented based on skin colour, resized the images to 300×400 pixels, applied Discrete Wavelet Transform (DWT) to extract features and achieved 84.6% classification accuracy using MLP22 .…”

mentioning

confidence: 99%

See 3 more Smart Citations

Vision-based Pakistani Sign Language Recognition Using Bag-of-Words and Support Vector Machines

Mirza

Munaf

Ali

et al. 2022

Preprint

View full text Add to dashboard Cite

In order to perform their daily activities, a person is required to communicating with others. This can be a major obstacle for the deaf population of the world, who communicate using sign languages (SL). Pakistani Sign Language (PSL) is used by more than 250,000 deaf Pakistanis. Developing a SL recognition system would greatly facilitate these people. This study aimed to collect data of static and dynamic PSL alphabets and to develop a vision-based system for their recognition using Bag-of-Words (BoW) and Support Vector Machine (SVM) techniques. A total of 5,120 images for 36 static PSL alphabet signs and 353 videos with 45,224 frames for 3 dynamic PSL alphabet signs were collected from 10 native signers of PSL. The developed system used the collected data as input, resized the data to various scales and converted the RGB images into grayscale. The resized grayscale images were segmented using Thresholding technique and features were extracted using Speeded Up Robust Feature (SURF). The obtained SURF descriptors were clustered using K-means clustering. A BoW was obtained by computing the Euclidean distance between the SURF descriptors and the clustered data. The codebooks were divided into training and testing using 5-fold cross validation. The highest overall classification accuracy for static PSL signs was 97.80% at 750×750 image dimensions and 500 Bags. For dynamic PSL signs a 96.53% accuracy was obtained at 480×270 video resolution and 200 Bags.

show abstract

mentioning

confidence: 85%

Section: Literature Reviewmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Vision-based Pakistani Sign Language Recognition Using Bag-of-Words and Support Vector Machines

Mirza

Munaf

Ali

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The text gathered from sign language is then converted into audio using Google Text to Speech App which helps in converting Urdu text into Urdu audio and a complete system of PSL to Urdu Translator is formed as shown in Figure 8. This is a very user friend system [23] which can facilitate both hearing impaired person [24] and the normal person to communicate with each other without facing any challenges [25].…”

Section: Figure 1 Static Sign Language Translationmentioning

confidence: 99%

Pakistan sign language to Urdu translator using Kinect

Ahmed

Shafiq²,

Raheel³

et al. 2022

Comput. Sci. Inf. Technol.

View full text Add to dashboard Cite

The lack of a standardized sign language, and the inability to communicate with the hearing community through sign language, are the two major issues confronting Pakistan's deaf and dumb society. In this research, we have proposed an approach to help eradicate one of the issues. Now, using the proposed framework, the deaf community can communicate with normal people. The purpose of this work is to reduce the struggles of hearing-impaired people in Pakistan. A Kinect-based Pakistan sign language (PSL) to Urdu language translator is being developed to accomplish this. The system’s dynamic sign language segment works in three phases: acquiring key points from the dataset, training a long short-term memory (LSTM) model, and making real-time predictions using sequences through openCV integrated with the Kinect device. The system’s static sign language segment works in three phases: acquiring an image-based dataset, training a model garden, and making real-time predictions using openCV integrated with the Kinect device. It also allows the hearing user to input Urdu audio to the Kinect microphone. The proposed sign language translator can detect and predict the PSL performed in front of the Kinect device and produce translations in Urdu.

show abstract

An improved custom convolutional neural network based hand sign recognition using machine learning algorithm

Moon,

Yenurkar,

Nyangaresi

et al. 2024

Engineering Reports

View full text Add to dashboard Cite

The biggest challenge the deaf and dumb group faces is that individuals around them do not understand sign language, which they use to communicate with one another. Written communication is slower than face‐to‐face contact, despite the fact that it can be used. Many sign languages have been developed around the world because they are more effective in emergency situations than text‐based communication. India in‐spite of having the large deaf population of almost 18 million and having only around 250 trained/untrained; skilled interpreters. The proposed system can utilize a custom convolution neural networks (CCNNs) model to identify hand motions in order to resolve this issue. This system uses a filter to process the hand before sending it through a classifier to identify the type of hand movements. CCNN strategy employs two levels of algorithm to predict and evaluate symbols that are increasingly similar to one another in order to get as close to precisely recognizing the symbol presented as possible. Convolutional neural networks (CNNs) are able to precisely identify a variety of gestures after being trained on large datasets of hand sign photographs. As a result of their frequent usage of many layers of filters and pooling to extract relevant information from the input images, these networks can recognize hand signs with an accuracy rate of 99.95%, which is much greater than previously built models like SIGNGRAPH, SVM, KNN, CNN + Bi‐LSTM, 3D‐CNN and 2D CNN network and 1D CNN skeleton network. The simulation result shows that a suggested CCNN‐based learning approach is useful for hand sign detection and future usage research when compared with existing machine learning models.

show abstract

Sign Language Recognition Using Multiple Kernel Learning: A Case Study of Pakistan Sign Language

Cited by 44 publications

References 16 publications

Vision-based Pakistani Sign Language Recognition Using Bag-of-Words and Support Vector Machines

Vision-based Pakistani Sign Language Recognition Using Bag-of-Words and Support Vector Machines

Pakistan sign language to Urdu translator using Kinect

An improved custom convolutional neural network based hand sign recognition using machine learning algorithm

Contact Info

Product

Resources

About