GnoSys app translates sign language into speech in real time using the power of AI

It’s called GnoSys and it promises to offer an affordable and high quality way to translate sign language into text and speech. Even better, it promises to do so in real time.

Developed by a start-up in Netherlands called Evalk, GnoSys is smartphone app powered by artificial intelligence (AI). Also referred to as “Google translator for the deaf” it works by putting a smartphone before the user. It uses neural networks and computer vision to understand the video of the sign language speaker. This is later converted to speech by smart algorithms.

Evalk is working in partnership with India Accelerator and is in talks with the National Deaf Association (NAD) in India to gather sign language data for India.

GnoSys converts sign language to text and speech in real time and gathering corpus of data to drive the app is critical. Currently Evalk is working on doing that with American Sign Language. We are also working out the modalities with NAB to build a corpus of data relating to Indian Sign language. – Mona Singh, Chief Accelerator Officer, India Accelerator

The challenges of gathering data in a country like India will be many, as Singh points out. “The biggest challenge would be that there are so many languages spoken here and so many variations in Indian sign language. A lot depends on how the person is enacting and given the many regions, every person will have his or her version. The corpus needs to be huge enough to be able to take it.”

In a country where quality interpreter services are so hard to come by, GnoSys could be a godsend. An estimated 18 million people are deaf in India and the app will find uses in many settings. It will play a major role in building inclusion by taking away communication barriers faced by deaf people.

The app has been unveiled in Netherlands and can be used in a range of devices, including tablets, and personal computers. All it needs is a camera on the device facing the person who is signing and an Internet connection.

“I think it will be very useful for deaf people in education and day to day living,” says Gyanendra Purohit, head of Anand Service Society, an organization fighting for the rights of deaf people in Madhya Pradesh. “Especially in a poor country like India, where deaf people cannot afford to hire an interpreter and the government not have a provision to pay for interpreter fees.”

The app will help in mainstreaming the deaf community in India, believes Aman Sharma, Co-founder, Training and Education Centre for Hearing Impaired (TEACH), an organization dedicated to making higher education a reality for deaf children in India.

“I think it’s a great tool and technology to have that shall integrate businesses, education and bridge the existing communication barriers in the society. Not just that, our deaf children miss out on the basic communication with their
parents due to less knowledge of sign language and this would help the deaf to be an integral part of the society rather than standing in isolation.”

Evalk says the app will be made available free with a fixed number of phrases per month or week for the user. This is to make sure that deaf people with limited income do not lose out on the opportunity to use the app.