一种用于多模态情感分析的文本引导跨层次融合和多任务学习框架
A text-guided cross-hierarchical fusion and multi-task learning framework for multimodal sentiment analysis.
摘要(中文译文)
现有多模态情感分析（MSA）方法已取得较强性能，但仍面临若干挑战：（1）文本模态信息利用不足；（2）对层次化多模态特征（包括深层和浅层特征）的联合建模效果有限；（3）对各模态独立特性的探索不充分。这些问题可能导致模型忽略文本中重要的情感模式，并阻碍其有效学习具有显著情感线索的跨层次特征以及模态特有特征。为应对这些挑战，我们提出了一种多任务学习框架，联合建模三个单模态预测任务和一个多模态情感预测任务。多模态分支产生最终预测输出，而单模态分支在损失函数的监督下帮助优化网络参数并促进模态特有特征的学习。该框架包含三个创新模块：（1）原始跨模态信息融合模块（RCMIF），基于图卷积构建，用于学习浅层多模态表示；（2）净化跨模态信息融合模块（CCMIF），通过动态图卷积和注意力机制捕获更深层的多模态信息；（3）双线性注意力深层独立特征挖掘模块（BADIC），利用双线性池化及相关技术探索单模态独立特征。值得注意的是，BADIC和CCMIF均利用文本引导，而RCMIF和CCMIF协同学习跨层次特征。在多个数据集（MOSI、MOSEI、CH-SIMS和CH-SIMS-V2）上的大量实验表明，与现有方法相比，所提框架在二分类准确率和F1分数上取得了约0.5%–1%的提升。
Abstract
Existing multimodal sentiment analysis (MSA) methods have achieved strong performance, but they still face several challenges: (1) insufficient utilization of textual modality information; (2) limited effectiveness in jointly modeling hierarchical multimodal features (including deep and shallow features); and (3) inadequate exploration of the independent characteristics of each modality. These issues may cause models to overlook important emotional patterns in text and hinder effective learning of cross-hierarchical features with salient emotional cues as well as modality-specific characteristics. To address these challenges, we propose a multi-task learning framework that jointly models three unimodal prediction tasks and one multimodal sentiment prediction task. The multimodal branch produces the final prediction output, while the unimodal branches, under the supervision of loss functions, help optimize the network parameters and promote the learning of modality-specific characteristics. The framework contains three innovative modules: (1) the Raw Cross-Modal Information Fusion Module (RCMIF), built upon graph convolution, for learning shallow multimodal representations; (2) the Cleaned Cross-Modal Information Fusion Module (CCMIF), which captures deeper multimodal information via dynamic graph convolution and attention mechanisms; and (3) the Bilinear Attention Deep Independent Characteristics Mining Module (BADIC), which explores unimodal independent characteristics by leveraging bilinear pooling and related techniques. Notably, both BADIC and CCMIF exploit textual guidance, while RCMIF and CCMIF collaboratively learn cross-hierarchical features. Extensive experiments on multiple datasets (MOSI, MOSEI, CH-SIMS, and CH-SIMS-V2) show that the proposed framework achieves improvements of approximately 0.5%-1% in binary classification accuracy and F1 score compared with existing methods.
如何引用
AMA
Minghui Zhu, Yushan Pan, Nan Xiang, Zuhe Li, Di Wu, Zhiyang Zhao, et al. A text-guided cross-hierarchical fusion and multi-task learning framework for multimodal sentiment analysis.. Neural networks : the official journal of the International Neural Network Society. 2026; doi:10.1016/j.neunet.2026.108995.
APA
Minghui Zhu, Yushan Pan, Nan Xiang, Zuhe Li, Di Wu, Zhiyang Zhao, et al (2026). A text-guided cross-hierarchical fusion and multi-task learning framework for multimodal sentiment analysis.. Neural networks : the official journal of the International Neural Network Society. https://doi.org/10.1016/j.neunet.2026.108995
GB/T 7714
Minghui Zhu, Yushan Pan, Nan Xiang, Zuhe Li, Di Wu, Zhiyang Zhao, et al. A text-guided cross-hierarchical fusion and multi-task learning framework for multimodal sentiment analysis.[J]. Neural networks : the official journal of the International Neural Network Society, 2026 doi:10.1016/j.neunet.2026.108995.
在 PubMed增强版 阅读全文
翻译全文 · AI 文献问答 · PDF 原文下载 · 引用导出