HarMI: Human Activity Recognition Via Multi-Modality Incremental Learning

Nowadays, with the development of various kinds of sensors in smartphones or wearable devices, human activity recognition (HAR) has been widely researched and has numerous applications in healthcare, smart city, etc. Many techniques based on hand-crafted feature engineering or deep neural network have been proposed for sensor based HAR. However, these existing methods usually recognize activities offline, which means the whole data should be collected before training, occupying large-capacity storage space. Moreover, once the offline model training finished, the trained model can’t recognize new activities unless retraining from the start, thus with a high cost of time and space. In this paper, we propose a multi-modality incremental learning model, called HarMI, with continuous learning ability. The proposed HarMI model can start training quickly with little storage space and easily learn new activities without storing previous training data. In detail, we first adopt attention mechanism to align heterogeneous sensor data with different frequencies. In addition, to overcome catastrophic forgetting in incremental learning, HarMI utilizes the elastic weight consolidation and canonical correlation analysis from a multi-modality perspective. Extensive experiments based on two public datasets demonstrate that HarMI can achieve a superior performance compared with several state-of-the-arts.

Authors

• Xiao Zhang Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
• Hongzheng Yu
• Yang Yang
• Jingjing Gu Singapore Management University
• Yujun Li School of Artifcial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, Xi'an 710072, P.R. China
• Fuzhen Zhuang Institute of Artificial In telligence, Beihang University Zhongguancun Laboratory
• Dongxiao Yu Shandong University
• Zhaochun Ren Leiden University

Keywords

• Sensors
• Training
• Data models
• Activity recognition
• Correlation
• Intelligent sensors
• Training data
• Human Activities
• Action Recognition
• Incremental Learning
• Human Activity Recognition
• Neural Network
• Time And Space
• Deep Network
• Deep Neural Network
• Time Cost
• Public Datasets
• Sensor Data
• Attention Mechanism
• Wearable Devices
• Storage Space
• Smart City
• Canonical Correlation Analysis
• Multimodal Learning
• Multimodal Model
• Kinds Of Sensors
• Attention Layer
• Total Time Cost
• Inertial Measurement Unit
• Multimodal Methods
• Storage Cost
• Fisher Information Matrix
• Multiple Sensors
• Training Time Cost
• Previous Tasks
• Training Sequences
• Catastrophic forgetting
• mobile device
• multi-modality
• Humans
• Machine Learning
• Neural Networks, Computer
• Smartphone
• Wearable Electronic Devices