A Generative Random Modality Dropout Framework for Robust Multimodal Emotion Recognition

Multimodal sentiment analysis faces significant challenges in real-world applications due to the frequent absence of modalities caused by privacy concerns, device limitations, or security policies. This article introduces a random modality dropout based on generative approach (RMDG), designed to enhance the robustness and performance of multimodal models under various modality absence scenarios. The RMDG method employs a generative approach during the training phase, where random modality dropout is applied to simulate missing modalities. By leveraging the remaining modalities to predict and regenerate the key features of the missing ones, the model effectively adapts to dynamic and unpredictable modality absences. This strategy not only eliminates the need for separate training or adjustments for each modality combination but also significantly improves the efficiency and accuracy of sentiment analysis in incomplete multimodal data scenarios. Extensive experiments demonstrate that RMDG outperforms existing methods, achieving superior performance in both complete and missing modality conditions.

Authors

• Yang Zhang
• Hui Chen
• Imad Rida
• Xianxun Zhu Shanghai University Macquarie University

Keywords

• Training
• Sentiment analysis
• Emotion recognition
• Adaptation models
• Privacy
• Predictive models
• Performance gain
• Robustness
• Security
• Testing
• Multimodal Emotion Recognition
• Data Visualization
• General Approach
• Incomplete Data
• Attention Mechanism
• Human-computer Interaction
• Generative Adversarial Networks
• Adaptive Model
• Feature Fusion
• Security Policy
• Performance In Scenarios
• Audio Data
• Multimodal Analysis
• Modality Conditions
• Alignment Loss
• F1 Score
• Fundamental Frequency
• Mean Absolute Error
• Facial Action Units
• Emotional Intensity
• Visual Modality
• Formant
• Text Data
• Multilayer Perceptron
• Word Embedding
• Corrupted Data
• Linear Projection
• Validation Loss