Click to sign: IIIT Hyderabad students develop Indian Sign Language plugin for web users with hearing impairment | Hyderabad News


Click to sign: IIIT Hyderabad students develop Indian Sign Language plugin for web users with hearing impairment

Hyderabad: Two students of International Institute of Information Technology, Hyderabad (IIIT-H), have developed a simple browser plugin that enables people with hearing impairments to access online content in Indian Sign Language (ISL) through a 3D avatar.Developed by Masthan Sharif Shaik and Mohammed Raza, third-year BTech students at Centre for Visual Information Technology (CVIT), the low-cost tool allows users to highlight text on any webpage and convert it into sign language at the click of a button. It can also be integrated into websites without requiring specialised hardware or server-based AI models.“Websites today support captions, screen readers and high-contrast modes, but almost none of them can show content in sign language. Existing signing avatars are either costly motion-capture systems, heavy AI models that need powerful GPUs, or systems built from sign notation. These require linguistic experts and also tend to move robotically,” said Sharif.The plugin, he said, is designed to run in real time within a browser, making it simpler and more accessible to deploy.To develop the tool, the students used videos from the official Indian Sign Language dictionary of Indian Sign Language Research and Training Centre (ISLRTC). Using computer vision, they reconstructed the signers’ 3D body, hand and finger movements.However, automated reconstruction often produces inaccurate finger movements, potentially changing the meaning of a sign. To address this, the team developed an annotation tool called Gizmo, which allows them to correct only the crucial poses in each sign rather than manually editing every frame.An offline pre-processing step converts sentences into ISL word order. For instance, “He will come home” becomes “He home come” in ISL. When users highlight a sentence and select the ‘click to sign’ button, the corresponding stored motions are retrieved and blended to create a fluid signing sequence. Words without corresponding signs are spelled out letter by letter using finger movements.The research, titled “SignPlug – ISL plugin for the web”, has been accepted at the 17th Indian conference on computer vision, graphics, and image processing (ICVGIP 2026), scheduled to be held in Kolkata in Dec.Tests with deaf and hard-of-hearing signers showed that the avatar’s signs were understood in about 89% of the sentences. However, the absence of facial features remains a limitation, as facial expressions are an important part of ISL grammar and some signs vary depending on context.The team plans to incorporate facial expressions and improve contextual signing in the next version.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *