Design of a Wearable EMG–IMU Gesture Recognition System for Real-Time Sign Language Translation in Assistive Communication Applications

Muhammad Nazmi Fazail; Muhammad Hamdi Mohd Najib; Ahmad Anwar Bin Zainuddin; Mohd Izzuddin Mohd Tamrin; Krishnan Subramaniam; Siti Fairuz Nurr Sadikan; Saidatul Izyanie Kamarudin

doi:10.56532/mjsat.v6i3.751

Authors

Muhammad Nazmi Fazail Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Gombak, Malaysia
Muhammad Hamdi Mohd Najib Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Gombak, Malaysia
Ahmad Anwar Bin Zainuddin Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Gombak, Malaysia
Mohd Izzuddin Mohd Tamrin Kulliyyah of Information and Communication Technology, International Islamic University Malaysia, Gombak, Malaysia https://orcid.org/0000-0003-1397-8174
Krishnan Subramaniam INTI International College, Penang, Malaysia
Siti Fairuz Nurr Sadikan Faculty of Plantation and Agrotechnology, University Teknology Mara, Jasin, Melaka, Malaysia
Dr.Saidatul Izyanie Kamarudin Universiti Teknologi MARA, Malaysia

DOI:

https://doi.org/10.56532/mjsat.v6i3.751

Keywords:

Sign language translation, Artificial intelligence, Electromyography sensors, Internet of Things

Abstract

Communication barriers have existed between the speech-and-hearing-impaired people and rest of the world due to the non-mainstream nature of sign language (SL). While there are proposed options to translate, most of them are costly and undeployable. The paper aims to create an affordable system to provide impaired users ability to communicate with normal people easily using text-to-speech (TTS) technology with the power of artificial intelligence (AI) and Internet of Things (IoT), powered by ESP32 to accurately predict SL. This is supported by dual channel electromyography (EMG) sensors and 6 axis gyrometer that is processed through filtering and feature extraction to give out high quality data for prediction. This system allows seamless communication enabled by MQTT and Wi-Fi connection from a wearable glove to collect required data that is used to train a Random Forest (RF) model. On a held-out validation set the system attained 92% accuracy; however, in on-field deployment this fell to about 53% (38–70% across the three gestures), so the system is presented as a feasibility prototype rather than a deployment-ready aid. As a starting point for this approach, future work will expand the dataset with more participants and gestures, complete a subject-independent evaluation, and refine the wearable device.

References

A. Ben Hej Amor, O. El Ghoul and M. Jemni, “A deep learning based approach for Arabic Sign language alphabet recognition using electromyographic signals,” 2021 8th International Conference on ICT & Accessibility (ICTA), Tunis, Tunisia, 2021, pp. 01-04, doi: https://doi.org/10.1109/ICTA54582.2021.9809780.

J. Gong, C. Li, C. Tang, X. Chen and S. Gao, “An EMG Based Wearable System for Chinese Sign Language Recognition,” 2024 IEEE BioSensors Conference (BioSensors), Cambridge, United Kingdom, 2024, pp. 1-4, doi: https://doi.org/10.1109/BioSensors61405.2024.10712721.

S. Yuan et al., “Chinese Sign Language Alphabet Recognition Based on Random Forest Algorithm,” 2020 IEEE International Workshop on Metrology for Industry 4.0 & IoT, Roma, Italy, 2020, pp. 340-344, doi: https://doi.org/10.1109/MetroInd4.0IoT48571.2020.9138285.

A. Ben Haj Amor, O. El Ghoul and M. Jemni, “Deep learning approach for sign language's handshapes recognition from EMG signals,” 2022 IEEE Information Technologies & Smart Industrial Systems (ITSIS), Paris, France, 2022, pp. 1-5, doi: https://doi.org/10.1109/ITSIS56166.2022.10118359.

N. Petranon and S. Umchid, “Design and Development of Hand Movement Detection Device for Sign Language using IMU and sEMG Sensors,” 2023 Research, Invention, and Innovation Congress: Innovative Electricals and Electronics (RI2C), Bangkok, Thailand, 2023, pp. 1-4, doi: https://doi.org/10.1109/RI2C60382.2023.10356033.

M. A. Ozdemir, D. H. Kisa, O. Guren, A. Onan and A. Akan, “EMG based Hand Gesture Recognition using Deep Learning,” 2020 Medical Technologies Congress (TIPTEKNO), Antalya, Turkey, 2020, pp. 1-4, doi: https://doi.org/10.1109/TIPTEKNO50054.2020.9299264.

H. Chen, T. Qin, Y. Zhang and B. Guan, “Recognition of American Sign Language Gestures Based On Electromyogram (EMG) Signal With XGBoost Machine Learning,” 2021 3rd International Conference on Machine Learning, Big Data and Business Intelligence (MLBDBI), Taiyuan, China, 2021, pp. 24-29, doi: https://doi.org/10.1109/MLBDBI54094.2021.00013.

M. Seddiqi, H. Kivrak and H. Kose, “Recognition of Turkish Sign Language (TID) using sEMG sensor,” 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), Istanbul, Turkey, 2020, pp. 1-6, doi: https://doi.org/10.1109/ASYU50717.2020.9259859.

M. U. Khan, F. Amjad, S. Aziz, S. Z. H. Naqvi, M. Shakeel and M. A. Imtiaz, “Surface electromyography based Pakistani sign language interpreter,” 2020 International Conference on Electrical, Communication, and Computer Engineering (ICECCE), Istanbul, Turkey, 2020, pp. 1-5, doi: https://doi.org/10.1109/ICECCE49384.2020.9179254.

S. K. Singh and A. Chaturvedi, “A reliable and efficient machine learning pipeline for American sign language gesture recognition using EMG sensors,” Multimedia Tools and Applications, vol. 82, no. 15, pp. 23833–23871, Nov. 2022, doi: https://doi.org/10.1007/s11042-022-14117-y.

Y. Gu, C. Zheng, M. Todoh and F. Zha, “American sign language translation using wearable inertial and electromyography sensors for tracking hand movements and facial expressions,” Frontiers in Neuroscience, vol. 16, Jul. 2022, doi: https://doi.org/10.3389/fnins.2022.962141.

S. Tateno, H. Liu, and J. Ou, “Development of sign language recognition system for hearing-impaired people using electromyography signal,” Sensors, vol. 20, no. 20: 5807, 2020, doi: https://doi.org/10.3390/s20205807.

İ Umut. and Ü. C. Kumdereli, “Novel wearable system to recognize sing language in real time,” Sensors, vol. 24, no. 14: 4613, 2024, doi: https://doi.org/10.3390/s24144613.

J. J. A. Mendes Junior et al., “Analysis of influence of segmentation, features and classification in sEMG processing: a case study of recognition of Brazilian sign language alphabet,” Sensors, vol. 20, no. 16: 4359, 2020, doi: https://doi.org/10.3390/s20164359.

A. Filipowska et al., “Pattern recognition in the processing of electromyographic signals for selected expressions of Polish sign language,” Sensors, vol. 24, no. 20: 6710, 2024, doi: https://doi.org/10.3390/s24206710.

W. Li, Z. Luo, W. Li and X. Xi, “Chinese sign language recognition based on surface electromyography and motion information,” PLoS ONE, vol. 18, no. 12, p. e0295398, Dec. 2023, doi: https://doi.org/10.1371/journal.pone.0295398.

K. Challa, I. W. AlHmoud, A. K. M. Kamrul Islam and B. Gokaraju, “EMG-Based hand gesture recognition using individual sensors on different muscle groups,” 2023 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), St. Louis, MO, USA, 2023, pp. 1-4, doi: https://doi.org/10.1109/AIPR60534.2023.10440702.

A. B. Habib, F. B. Ashraf and A. Shakil, “Finding efficient machine learning model for hand gesture classification using EMG data,” 2021 5th International Conference on Electrical Engineering and Information Communication Technology (ICEEICT), Dhaka, Bangladesh, 2021, pp. 1-6, doi: https://doi.org/10.1109/ICEEICT53905.2021.9667856.

R. Jia, L. Yang, Y. Li and Z. Xin, “Gestures recognition of sEMG signal based on Random Forest,” 2021 IEEE 16th Conference on Industrial Electronics and Applications (ICIEA), Chengdu, China, 2021, pp. 1673-1678, doi: https://doi.org/10.1109/ICIEA51954.2021.9516350.

A. Shaafi, O. Salem, M. Gheryani and A. Mehaoua, “Wireless body sensor networks for sign language recognition with real-time data analysis,” ICC 2022 - IEEE International Conference on Communications, Seoul, Korea, Republic of, 2022, pp. 5310-5315, doi: https://doi.org/10.1109/ICC45855.2022.9838474.

P. Rani, S. Pancholi, V. Shaw, M. Atzori and S. Kumar, “Enhancing gesture classification using active EMG band and advanced feature extraction technique,” in IEEE Sensors Journal, vol. 24, no. 4, pp. 5246-5255, 15 Feb.15, 2024, doi: https://doi.org/10.1109/JSEN.2023.3344700.

G. Li, B. Wan, K. Su, J. Huo, C. Jiang and F. Wang, “sEMG and IMU data-based hand gesture recognition method using multistream CNN with a fine-tuning transfer framework,” in IEEE Sensors Journal, vol. 23, no. 24, pp. 31414-31424, 15 Dec.15, 2023, doi: https://doi.org/10.1109/JSEN.2023.3327999.

O. Steven Eyobu and Han D.S., “Feature representation and data augementation for human activity classification based on wearable IMU sensor data using a deep LSTM neural network,” Sensors, vol. 18, no. 9: 2892, 2018, doi: https://doi.org/10.3390/s18092892.

J. G. Colli Alfaro and A. L. Trejos, “User-independent hand gesture recognition classification models using sensor fusion,” Sensors, vol. 22, no. 4: 1321, 2022, doi: https://doi.org/10.3390/s22041321.

J.P. Vasconez, L.I. Barona Lopez, A.L. Valdivieso Caraguay and M.E. Benalcazar, “Hand gesture recognition using EMG-IMU signals and deep Q-networks,” Sensors, vol. 22, no. 24: 9613, 2022, doi: https://doi.org/10.3390/s22249613.