JDA-RSDB：一种用于脑电与眼动信号跨会话情绪识别的多模态域适应方法

摘要(中文译文)

多模态情感识别在情感计算领域日益受到关注，因为结合脑电图（EEG）和眼动（EM）信号能够捕捉复杂的情感过程。然而，EEG和EM信号在不同日期和记录会话之间存在联合分布差异，导致识别性能下降。目前，域自适应已被开发用于解决此类分布差异。遗憾的是，现有的域自适应方案仍然表现出次优的分类结果，因为在分布匹配过程中仍然学习到模糊且不具判别性的决策边界。本文提出了联合分布对齐与精细化可分离决策边界（JDA-RSDB），一种用于EEG和EM信号跨会话情感识别的多模态域自适应方法。我们提出的方法假设在联合分布匹配过程中，必须确保在新会话上获得更具判别性的特征表示。为此，JDA-RSDB在域之间产生相似的边缘分布和条件分布，首先在模态和域级别对齐特征统计量，然后促使来自不同域但产生相同类别预测的融合样本之间保持一致的相似性。同时，通过在目标数据上学习可分离的特征空间，将决策边界置于低密度区域，从而增强这种相似性。更重要的是，通过使主分类器和辅助分类器的目标预测达成一致，对决策边界进行精细化。实验在三个公开数据集SEED-GER、SEED-IV和SEED-V上以跨会话设置进行。所提出的框架在SEED-GER、SEED-IV和SEED-V的三个可用会话上分别达到了83.33%、80.89%和75.17%的平均准确率，优于当前最先进的方案。

Abstract

Multimodal emotion recognition has shown growing interest in affective computing, as combining Electroencephalogram (EEG) and eye movement (EM) signals enables the capture of complex emotional processes. However, EEG and EM signals are exposed to joint distribution differences across different days and recorded sessions, reducing the recognition performance. Currently, domain adaptation has been developed to address such distribution differences. Unfortunately, existing domain adaptation solutions still show suboptimal classification results, since ambiguous and non-discriminative decision boundaries are still learned during distribution matching. This paper presents Joint Distribution Alignment with Refined and Separable Decision Boundaries (JDA-RSDB), a multimodal domain adaptation method for cross-session emotion recognition from EEG and EM signals. Our proposed method assumes that a more discriminative feature representation must be ensured on new sessions during joint distribution matching. For this, JDA-RSDB produces similar marginal and conditional distributions between domains, first aligning feature statistics at modality and domain levels, and then, motivating consistent similarity between fused samples from different domains that produce the same class prediction. Simultaneously, this similarity is enhanced by learning a separable feature space on target data, placing decision boundaries on low-density regions. More importantly, decision boundaries are refined by achieving an agreement between target predictions from a principal classifier and those from an auxiliary classifier. Experiments were conducted on three public datasets, SEED-GER, SEED-IV, and SEED-V, in a cross-session setting. The proposed framework achieves an average accuracy of 83.33%, 80.89%, and 75.17% across the three available sessions on SEED-GER, SEED-IV, and SEED-V, outperforming state-of-the-art solutions.

如何引用

Cognitive neurodynamics 的其他文献

• 迷迭香酸减轻颞叶皮层与年龄相关的EEG复杂性损失：复发量化分析