Cross-Space Synergy: A Unified Framework for Multimodal Emotion Recognition in Conversation

Multimodal Emotion Recognition in Conversation (MERC) aims to predict speakers’ emotions by integrating textual, acoustic, and visual cues. Existing approaches either struggle to capture complex cross‑modal interactions or experience gradient conflicts and unstable training when using deeper architectures. To address these issues, we propose Cross-Space Synergy (CSS), which couples a representation component with an optimization component. Synergistic Polynomial Fusion (SPF) serves the representation role, leveraging low-rank tensor factorization to efficiently capture high-order cross-modal interactions. Pareto Gradient Modulator (PGM) serves the optimization role, steering updates along Pareto-optimal directions across competing objectives to alleviate gradient conflicts and improve stability. Experiments show that CSS outperforms existing representative methods on IEMOCAP and MELD in both accuracy and training stability, demonstrating its effectiveness in complex multimodal scenarios.

Authors

• Xiaosen Lyu School of Computer Science and Technology, Huaqiao University, Xiamen, China Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Xiamen, China
• Jiayu Xiong School of Computer Science and Technology, Huaqiao University, Xiamen, China Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Xiamen, China
• Yuren Chen School of Computer Science and Technology, Huaqiao University, Xiamen, China Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Xiamen, China
• Wanlong Wang School of Computer Science and Technology, Huaqiao University, Xiamen, China Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Xiamen, China
• Xiaoqing Dai School of Computer Science and Technology, Huaqiao University, Xiamen, China Xiamen Key Laboratory of Computer Vision and Pattern Recognition, Xiamen, China
• Jing Wang

Keywords

No keywords are indexed for this paper.