Sign Language Recognition and Application Based on Graph Neural Networks: Innovative Integration in TV News Sign Language

Peilai Yu

doi:10.70393/6a6374616d.323636

Authors

Peilai Yu LMU Ludwig Maximilian University of Munich

DOI:

https://doi.org/10.70393/6a6374616d.323636

ARK:

https://n2t.net/ark:/40704/JCTAM.v2n2a02

Disciplines:

Artificial Intelligence and Intelligence

Subjects:

Machine Learning

References:

28

Keywords:

Graph Neural Networks (GNN), Sign Language Recognition, Television News, Real-time Translation, Automation

Abstract

With the rapid development of information technology, sign language recognition plays an extremely important role in the communication among people with hearing impairments. Especially in the context of television news, the real-time and accuracy of sign language translation are very important. Traditional sign language translation technology faces challenges such as low accuracy of gesture recognition and poor real-time performance, which makes it difficult to meet the translation needs of daily complex news content. This paper proposes a sign language recognition method based on graph neural network (GNN). By constructing a graph structure of gesture nodes and joint connections, GNN can capture the relationship between gestures and efficiently transfer learning information. Through comparative experiments with traditional convolutional neural networks (CNN), the advantages of GNN in sign language recognition are proved, especially in the application of news broadcasting, which significantly improves the real-time and accuracy of sign language translation. Future research will focus on optimizing the generalization ability of the model and broadening its applicability to more languages and scenarios.

Author Biography

Peilai Yu, LMU Ludwig Maximilian University of Munich

LMU Ludwig Maximilian University of Munich, Germany.

References

[1] Dreuw, P., Rybach, D., Deselaers, T., Zahedi, M., & Ney, H. (2007). Speech recognition techniques for a sign language recognition system. hand, 60(80).

[2] Ma, Y., Zhou, G., Wang, S., Zhao, H., & Jung, W. (2018). SignFi: Sign language recognition using WiFi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(1), 1-21.

[3] Chai, X., Li, G., Lin, Y., Xu, Z., Tang, Y., Chen, X., & Zhou, M. (2013, April). Sign language recognition and translation with kinect. In IEEE conf. on AFGR (Vol. 655, p. 4).

[4] Lyu, S. (2024). The Application of Generative AI in Virtual Reality and Augmented Reality. Journal of Industrial Engineering and Applied Science, 2(6), 1-9.

[5] Rastgoo, R., Kiani, K., & Escalera, S. (2021). Sign language recognition: A deep survey. Expert Systems with Applications, 164, 113794.

[6] Lyu, S. (2024). The Technology of Face Synthesis and Editing Based on Generative Models. Journal of Computer Technology and Applied Mathematics, 1(4), 21-27.

[7] Zaki, M. M., & Shaheen, S. I. (2011). Sign language recognition using a combination of new vision based features. Pattern Recognition Letters, 32(4), 572-577.

[8] Lyu, S. (2024). Machine Vision-Based Automatic Detection for Electromechanical Equipment. Journal of Computer Technology and Applied Mathematics, 1(4), 12-20.

[9] Rao, G. A., Syamala, K., Kishore, P. V. V., & Sastry, A. S. C. S. (2018, January). Deep convolutional neural networks for sign language recognition. In 2018 conference on signal processing and communication engineering systems (SPACES) (pp. 194-197). IEEE.

[10] Lin, W. (2024). A Review of Multimodal Interaction Technologies in Virtual Meetings. Journal of Computer Technology and Applied Mathematics, 1(4), 60-68.

[11] Pigou, L., Dieleman, S., Kindermans, P. J., & Schrauwen, B. (2015). Sign language recognition using convolutional neural networks. In Computer Vision-ECCV 2014 Workshops: Zurich, Switzerland, September 6-7 and 12, 2014, Proceedings, Part I 13 (pp. 572-578). Springer International Publishing.

[12] Lin, W. (2024). A Systematic Review of Computer Vision-Based Virtual Conference Assistants and Gesture Recognition. Journal of Computer Technology and Applied Mathematics, 1(4), 28-35.

[13] Wadhawan, A., & Kumar, P. (2021). Sign language recognition systems: A decade systematic literature review. Archives of computational methods in engineering, 28, 785-813.

[14] Luo, M., Zhang, W., Song, T., Li, K., Zhu, H., Du, B., & Wen, H. (2021, January). Rebalancing expanding EV sharing systems with deep reinforcement learning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence (pp. 1338-1344).

[15] Sun, Y., & Ortiz, J. (2024). Data Fusion and Optimization Techniques for Enhancing Autonomous Vehicle Performance in Smart Cities. Journal of Artificial Intelligence and Information, 1, 42-50.

[16] Luo, M., Du, B., Zhang, W., Song, T., Li, K., Zhu, H., ... & Wen, H. (2023). Fleet rebalancing for expanding shared e-Mobility systems: A multi-agent deep reinforcement learning approach. IEEE Transactions on Intelligent Transportation Systems, 24(4), 3868-3881.

[17] Zhu, H., Luo, Y., Liu, Q., Fan, H., Song, T., Yu, C. W., & Du, B. (2019). Multistep flow prediction on car-sharing systems: A multi-graph convolutional neural network with attention mechanism. International Journal of Software Engineering and Knowledge Engineering, 29(11n12), 1727–1740.

[18] Li, K., Chen, X., Song, T., Zhang, H., Zhang, W., & Shan, Q. (2024). GPTDrawer: Enhancing Visual Synthesis through ChatGPT. arXiv preprint arXiv:2412.10429.

[19] Kumar, A., Thankachan, K., & Dominic, M. M. (2016, March). Sign language recognition. In 2016 3rd international conference on recent advances in information technology (RAIT) (pp. 422-428). IEEE.

[20] Von Agris, U., Zieren, J., Canzler, U., Bauer, B., & Kraiss, K. F. (2008). Recent developments in visual sign language recognition. Universal Access in the Information Society, 6, 323-362.

[21] Xu, Y., Lin, Y. S., Zhou, X., & Shan, X. (2024). Utilizing emotion recognition technology to enhance user experience in real-time. Computing and Artificial Intelligence, 2(1), 1388-1388.

[22] Cooper, H. M., Ong, E. J., Pugeault, N., & Bowden, R. (2012). Sign language recognition using sub-units. Journal of Machine Learning Research, 13, 2205-2231.

[23] Li, K., Liu, L., Chen, J., Yu, D., Zhou, X., Li, M., ... & Li, Z. (2024, November). Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layout. In 2024 6th International Conference on Artificial Intelligence and Computer Applications (ICAICA) (pp. 296-301). IEEE.

[24] Bragg, D., Koller, O., Bellard, M., Berke, L., Boudreault, P., Braffort, A., ... & Ringel Morris, M. (2019, October). Sign language recognition, generation, and translation: An interdisciplinary perspective. In Proceedings of the 21st international ACM SIGACCESS conference on computers and accessibility (pp. 16-31).

[25] Li, K., Chen, J., Yu, D., Dajun, T., Qiu, X., Lian, J., ... & Han, J. (2024, October). Deep reinforcement learning-based obstacle avoidance for robot movement in warehouse environments. In 2024 IEEE 6th International Conference on Civil Aviation Safety and Information Technology (ICCASIT) (pp. 342-348). IEEE.

[26] Huang, X., Wu, Y., Zhang, D., Hu, J., & Long, Y. (2024, September). Improving Academic Skills Assessment with NLP and Ensemble Learning. In 2024 IEEE 7th International Conference on Information Systems and Computer Aided Education (ICISCAE) (pp. 37-41). IEEE.

[27] Cooper, H., Holt, B., & Bowden, R. (2011). Sign language recognition. In Visual Analysis of Humans: Looking at People (pp. 539-562). London: Springer London.

[28] Wu, S., & Huang, X. (2025). Psychological Health Prediction Based on the Fusion of Structured and Unstructured Data in EHR: a Case Study of Low-Income Populations. Preprints.