A Comprehensive Framework for Multimodal Sensor Fusion in Intelligent Manufacturing: Innovations, Interpretability, and Real-world Applications

Yue Zhu

doi:10.5281/zenodo.13905495

Authors

Yue Zhu Georgia Institute of Technology

DOI:

https://doi.org/10.5281/zenodo.13905495

ARK:

https://n2t.net/ark:/40704/JCTAM.v1n4a05

Disciplines:

Computer Science

Subjects:

Deep Learning

References:

19

Keywords:

Multimodal Sensor Fusion, Smart Manufacturing, Explainable AI (XAI), Industry 4.0, Predictive Maintenance, Fault Detection, Attention Mechanisms, Deep Learning, Hybrid Fusion, Random Forest Classifier, Data Integration, Machine Learning, Feature Extraction, Real-Time Processing, Condition Monitoring

Abstract

This paper presents the novel work of developing an intelligent manufacturing framework based on multimodal sensor integration and computer vision. In this paper, we propose a hybrid fusion method that includes both early and late fusion with attention mechanisms to select the most important sensor data. Our system gathers visual, thermal, acoustic, and vibration data and offers accurate and interpretable predictions for fault identification, process enhancement, and product quality. (Liu et al. 2024) We meet the challenge of the opacity of AI systems by using explainable AI methods to help the user comprehend the results of the model. It shows that the proposed system is accurate, efficient, scalable, and can be applied to various types of data. Examples from the industry present real-life experiences and issues that may be encountered when implementing our system in different manufacturing contexts. It presents a new paradigm shift in smart manufacturing systems through the enhancement of efficiency, reliability, and interpretability for future research and industrial development.

Author Biography

Yue Zhu, Georgia Institute of Technology

Georgia Institute of Technology, California, United States.

References

Liu, Z., Xiao, G., Liu, H., & Wei, H. (2024). Multi-Sensor Measurement and Data Fusion. IEEE Transactions on Instrumentation and Measurement, 73, 234-245. https://ieeexplore.ieee.org/abstract/document/9693406?casa_token=40VH55MYDtEAAAAA:5M5RuueqkCua6ONFCVjNeNl8V0TlZ4-giZ8VQzK0U6CXh_TefMxhQaCBotzrZXsBrmaDEjnq

Diez-Olivan, A., Del Ser, J., Galar, D., & Sierra, B. (2019). Data Fusion and Machine Learning for Industrial Prognosis: Trends and Perspectives Towards Industry 4.0. Journal of Manufacturing Processes, 40, 123-134. https://www.sciencedirect.com/science/article/pii/S1566253518304706?casa_token=Bi4TM627bvIAAAAA:FIhyFRBhR4K3o7_Ly5thLLoHChk7UYVPCZTvMDEK5aPxTYUJQuAPOI_Dn3ZHRIRz1TUbIaQ_

Helwig, Nikolai, Pignanelli, Eliseo, and Schtze, Andreas. (2018). Condition monitoring of hydraulic systems. UCI Machine Learning Repository. https://doi.org/10.24432/C5CW21.

Vergara, Alexander. (2013). Gas Sensor Array Drift at Different Concentrations. UCI Machine Learning Repository. https://doi.org/10.24432/C5MK6M.

Buscema, M., Terzi, S., & Tastle, W. (2010). Steel Plates Faults. UCI Machine Learning Repository. https://doi.org/10.24432/C5J88N

Zhang, Z., Farnsworth, M., Song, B., Tiwari, D., & Tiwari, A. (2024). Deep Transfer Learning With Self-Attention for Industry Sensor Fusion Tasks. IEEE Transactions on Industrial Informatics. https://ieeexplore.ieee.org/abstract/document/9813440

Sensor Data Fusion in Industry 4.0. (2024). Farnell. https://si.farnell.com/sensor-data-fusion-in-industry-40-trc-ar

Goyal, D., & Pabla, B. S. (2015). Condition-based maintenance of machine tools—A review. CIRP Journal of Manufacturing Science and Technology, 10, 24-35. https://doi.org/10.1016/j.cirpj.2015.05.002

Kang, H. S., Lee, J. Y., Choi, S., Kim, H., Park, J. H., Son, J. Y., ... & Do Noh, S. (2016). Smart manufacturing: Past research, present findings, and future directions. International Journal of Precision Engineering and Manufacturing-Green Technology, 3(1), 111-128. https://doi.org/10.1007/s40684-016-0015-5

Kusiak, A. (2019). Fundamentals of smart manufacturing: A multi-thread perspective. Annual Reviews in Control, 47, 214-220. https://doi.org/10.1016/j.arcontrol.2019.02.001

Liu, Y., Li, S., Liu, C., & Wang, H. (2021). Explainable AI for intelligent manufacturing: A survey. AI Open, 2(1), 105-123. https://doi.org/10.1002/aiopen.76

Monostori, L., Kádár, B., Bauernhansl, T., Kondoh, S., Kumara, S., Reinhart, G., ... & Ueda, K. (2016). Cyber-physical systems in manufacturing. CIRP Annals, 65(2), 621-641. https://doi.org/10.1016/j.cirp.2016.06.005

Tao, F., Cheng, J., Qi, Q., Zhang, M., Zhang, H., & Sui, F. (2018). Digital twin-driven product design, manufacturing, and service with big data. The International Journal of Advanced Manufacturing Technology, 94(9-12), 3563-3576. https://doi.org/10.1007/s00170-017-0233-1

Wuest, T., Weimer, D., Irgens, C., & Thoben, K. D. (2016). Machine learning in manufacturing: advantages, challenges, and applications. Production & Manufacturing Research, 4(1), 23-45. https://doi.org/10.1080/21693277.2016.1192517

Yin, S., Ding, S. X., Xie, X., & Luo, H. (2014). A review on basic data-driven approaches for industrial process monitoring. IEEE Transactions on Industrial Electronics, 61(11), 6418-6428. https://doi.org/10.1109/TIE.2014.2301773

Zheng, P., Wang, H., Sang, Z., Zhong, R. Y., Liu, Y., Liu, C., ... & Mubarok, K. (2018). Smart manufacturing systems for Industry 4.0: Conceptual framework, scenarios, and future perspectives. Frontiers of Mechanical Engineering, 13(2), 137-150. https://doi.org/10.1007/s11465-018-0499-5

Zhong, R. Y., Xu, X., Klotz, E., & Newman, S. T. (2017). Intelligent manufacturing in the context of industry 4.0: a review. Engineering, 3(5), 616-630. https://doi.org/10.1016/J.ENG.2017.05.015

Shao, Y., Yuan, X., Zhang, C., Song, Y., & Xu, Q. (2020). A Novel Fault Diagnosis Algorithm for Rolling Bearings Based on One-Dimensional Convolutional Neural Network and INPSO-SVM. Applied Sciences, 10(12), 4303. https://doi.org/10.3390/app10124303

Qiu, S., Cui, X., Ping, Z., Shan, N., Li, Z., Bao, X., & Xu, X. (2023). Deep Learning Techniques in Intelligent Fault Diagnosis and Prognosis for Industrial Systems: A Review. Sensors, 23(3), 1305. https://doi.org/10.3390/s23031305