SonicASL

Abstract

We propose SonicASL, a real-time gesture recognition system that can recognize sign language gestures on the fly, leveraging front-facing microphones and speakers added to commodity earphones worn by someone facing the person making the gestures. In a user study (N=8), we evaluate the recognition performance of various sign language gestures at both the word and sentence levels. Given 42 frequently used individual words and 30 meaningful sentences, SonicASL can achieve an accuracy of 93.8% and 90.6% for word-level and sentence-level recognition, respectively. The proposed system is tested in two real-world scenarios: indoor (apartment, office, and corridor) and outdoor (sidewalk) environments with pedestrians walking nearby. The results show that our system can provide users with an effective gesture recognition tool with high reliability against environmental factors such as ambient noises and nearby pedestrians.

Behavioural Analysis of Deaf and Mute People Using Gesture Detection

“…). The reliability of the system increases as more pictures of each character are saved used in equation (12). Segmentation methods based on pixels and regions can be used.…”

American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture System

“…The accuracy was 94.2%. In addition, SonicASL [ 26 ] leveraged dual speakers and microphones to capture the sonic feedback from hand gestures. Given 42 frequently used ASL words and 30 sentences, the system could achieve an accuracy of 93.8% at the word level, and 90.6% at the sentence level.…”

“…The summary of previous studies is listed in Table 1. EMG & IMU sensors 150 -8 [20] EMG & ACC sensors 121 -1 [24] Customized data glove 26 -3 [25] Smartwatch & Smart ring 100 -10 [26] Speaker & Microphone 42 30 8 In the above-mentioned studies on sign language recognition using wearable sensors, more studies focused on single gestures. Few datasets for sentence recognition were confined to a limited number of sentences and did not consider grammatical inconsistencies between sign language and spoken language.…”

“…The sign language translation vocabulary. 1,12,15,19,2,26,3,30,31, 35,5,6,7, 87, 9, :, AC, August, Canada, Christmas, Easter, English, Friday, Halloween, Hawaii, I, Monday, Saturday, Sunday, Thanksgiving, Thursday, US, Valentine's, a, across, afraid, afternoon, again, age, ahead, alcohol, all, allergic, alright, also, altogether, always, am, an, and, angry, animal, annoyed, apartment, apple, are, area, arrive, as, at, baby, back, bad, bake, baseball, basketball, bathroom, be, beach, because, become, becomes, before, bells, biologist, birthday, birthdays, black, blonde, blood, book, borrow, both, bowl, boxing, breakfast, bring, bringing, broke, brother, brown, brush, busy, but, cakes, calculus, car, care, cash, cat, celebrating, chair, chemistry, chocolate, church, class, cleaning, clinic, close, clothes, clouds, cloudy, coat, coffee, cold, college, color, come, comfortable, coming, commercial, competing, congratulations, cooked, cookies, cooking, cost, costs, costume, country, cousin, credit, cute, dad, dark, daughter, day, deaf, delicious, department, dessert, did, different, dinner, dislike, dislikes, do, does, dog, dogs, dollars, drinking, drugs, dry, due, during, eat, eating, eggs, elephant, engineering, enjoy, enjoys, every, everyday, excited, excuse, experience, fall, family, fancy, favorite, feel, fine, finger, finish, finished, flowers, food, football, for, forget, friend, friends, fries, from, funny, future, game, get, ghost, give, go, going, good, goodbye, graduate, graduation, grandma, growing, had, haha, hail, hair, half, hamburgers, happy, has, hat, hate, have, hearing, hearing-aid, heart, hello, help, here, hey, high, hiking, him, his, history, holidays, home, hope, host, hot, hotdogs, hotel, house, how, huge, hungry, hunting, husband, ice-cream, important, in, introduce, is, it, jingle, join, karate, ketchup, knit, know, lake, language, large, last, latter, learn, leave, let's, light, like, lion, little, live, lives, local, look, lost, lot, love, lunch, machine, major, make, manage, many, marry, may, me, medicine, medium, meet, meeting, merry, milk, mind, mix, mom, money, monkey, more, morning, mother, mother's, mountain, moved, movie, movies, much, museum, mustache, my, name, nauseous, need, needed, nervous, new, next, nice, night, no, not, now, of, office, officer, old, olympics, on, optometrist, or, orange, order, other, others, our, out, packing, pain, pancakes, paper, parade, party, pass, pay, people, physics, picnic, pineapple, pink, plan, play, please, police, potato, prayer, prefer, pregnant, president, pressure, principal, problem, pumpkin, purse, rabbits, rain, rainbow, rains, rainy, read, ready, red, reduction, remember, resolution, restaurant, resting, running, sad, same, sausages, scared, scarf, school, season, seatbelt, see, seizures, senior, shirt, shop, show, shrimps, shy, sign, since, sister, skills, slower, small, snow, soda, sometimes, son, soup, spaghetti, spell, spider, sport, spread, spring, start, starts, state, states, stay, steak, still, strawberry, street, strongly, student, studying, subscribe, summer, sunny, super, support, sweet, swimming, take, talkative, teacher, team, teeth, tends, test, thank, thankful, that, the, there, they, think, this, three, time, to, today, together,...…”