基于多模态生物信号的情绪识别：基于EEG与GSR的有监督和无监督机器学习方法

摘要(中文译文)

人工智能（AI）与机器学习（ML）正日益应用于心理学领域，尤其是在教育与临床环境中，用于分析复杂的行为与生理数据。然而，关于多模态生理信号在生态效度较高的高等教育环境中表征认知与情绪过程的能力，现有证据仍然有限。本研究探讨了监督式与非监督式机器学习方法在真实教育情境中分析情绪化虚拟形象所诱发的多模态生理反应的潜力。研究采用便利抽样招募了55名参与者，包括大学生和讲师。经数据清洗后，最终样本包含48名参与者。在参与者观看呈现不同情绪状态的虚拟形象时，记录其脑电图（EEG）和皮肤电反应（GSR）信号。从脑电图记录中提取了事件相关电位（ERP）衍生特征。采用随机森林分类器进行监督式学习，以评估生理变量、主观变量（NASA-TLX）和社会人口学变量在职业类别分类中的预测贡献。此外，采用k均值聚类和主成分分析（PCA）进行非监督式分析，以识别生理数据中的潜在模式。社会人口学变量取得了最高的预测性能，其次是主观工作负荷测量指标，而生理特征的判别能力则较为有限。聚类分析显示参与者反应轮廓之间存在大量重叠，主成分分析显示在降维空间中各组之间的分离程度有限。这些发现表明，与情绪加工相关的生理反应更适合表征为连续变异模式，而非清晰分化的类别。研究结果凸显了在生态效度较高的教育环境中将机器学习技术应用于多模态生理数据的机遇与局限。尽管单独的生理测量指标预测能力有限，但其提供了关于认知与情绪过程的补充信息。此外，本研究提出了一个基于仪表盘的框架，用于多模态数据的自动化整合、分析与可视化，为教育心理学、情感计算和精准心理学的未来发展提供支持。总体而言，研究结果强调了需要更大的样本量和更先进的多模态建模策略，以提高生理信号在真实情境中的可解释性和预测价值。

Abstract

Artificial intelligence (AI) and machine learning (ML) are increasingly applied in psychology, particularly in educational and clinical settings, to analyse complex behavioural and physiological data. However, evidence regarding the capacity of multimodal physiological signals to characterise cognitive and emotional processes in ecologically valid higher education environments remains limited. This study explored the potential of supervised and unsupervised ML approaches to analyse multimodal physiological responses elicited by emotional avatars in a real educational context. A convenience sample of 55 participants, including university students and lecturers, was recruited. After data cleaning, the final sample comprised 48 participants. Electroencephalography (EEG) and galvanic skin response (GSR) signals were recorded while participants observed avatars displaying different emotional states. Event-related potential (ERP)-derived features were extracted from EEG recordings. Supervised learning was conducted using a Random Forest classifier to evaluate the predictive contribution of physiological, subjective (NASA-TLX), and sociodemographic variables in classifying professional category. In addition, unsupervised analyses were performed using k-means clustering and principal component analysis (PCA) to identify latent patterns within the physiological data. Sociodemographic variables achieved the highest predictive performance, followed by subjective workload measures, whereas physiological features demonstrated more limited discriminative power. Cluster analysis revealed substantial overlap among participant response profiles, and PCA showed limited separation between groups in the reduced-dimensional space. These findings suggest that physiological responses associated with emotional processing are better represented as continuous patterns of variability rather than as clearly differentiated categories. The findings highlight both the opportunities and limitations of applying ML techniques to multimodal physiological data in ecologically valid educational settings. Although physiological measures alone showed limited predictive capacity, they provided complementary information regarding cognitive and emotional processes. Furthermore, the study proposes a dashboard-based framework for the automated integration, analysis, and visualisation of multimodal data, supporting future developments in educational psychology, affective computing, and precision psychology. Overall, the results underscore the need for larger samples and more advanced multimodal modelling strategies to improve the interpretability and predictive value of physiological signals in real-world contexts.

如何引用

Frontiers in psychology 的其他文献

• 大学生体育锻炼与抑郁：正念和经验回避的系列中介作用
• 算法娱乐的阴暗面：抖音用户的社交在场与身体在场、短视频成瘾与认知疲劳
• 人与动物的互动：了解狗和猫的互动在情绪健康中的作用
• 迈向可持续美育：探索生成式人工智能协作对学生批判性思维倾向的影响
• 论文被指导者遭受的不当对待与心理健康：基于结构方程模型的横断面研究
• 对年长员工的年龄歧视与工作幸福感：一项系统性综述
• 学习投入对公共管理专业学生学习成果的影响：数字素养的中介作用与人工智能工具接受的调节作用
• 大学校园学术内卷氛围与人工智能依赖：学业压力的中介作用与个体学术内卷行为的调节作用
• 从语言帝国主义到算法主导：数字智能时代语言霸权的心理含义
• 追梦心景：考研考生心理变化过程的探究