Multimodal Emotion Recognition Based on Facial Expression and ECG Signal

• 作者: NIU Jian-wei AN Yue-qi NI Jie JIANG Chang-hua
• 来源期刊: 包装工程
• 学科: 工学
• 年，卷（期）: 2022,（4）
• 所属期刊栏目: Special Subject: Human Factor Performance, Intelligent Interaction and Information Visualization in Industrial Systems
• 页码范围: 71-79
• 页数: 9页
• 分类号: TB472
• 语种: 中文
• DOI: 10.19554/j.cnki.1001-3563.2022.04.008

摘要

As a key link in human-computer interaction, emotion recognition can enable robots to correctly perceive user emotions and provide dynamic and adjustable services according to the emotional needs of different users, which is the key to improve the cognitive level of robot service. Emotion recognition based on facial expression and electrocar-diogram has numerous industrial applications. First, three-dimensional convolutional neural network deep learning archi-tecture is utilized to extract the spatial and temporal features from facial expression video data and electrocardiogram (ECG) data, and emotion classification is carried out. Then two modalities are fused in the data level and the decision level, respectively, and the emotion recognition results are then given. Finally, the emotion recognition results of sin-gle-modality and multi-modality are compared and analyzed. Through the comparative analysis of the experimental re-sults of single-modality and multi-modality under the two fusion methods, it is concluded that the accuracy rate of multi-modal emotion recognition is greatly improved compared with that of single-modal emotion recognition, and deci-sion-level fusion is easier to operate and more effective than data-level fusion.