Lyapunov-Stable Deep Equilibrium Models

Deep equilibrium (DEQ) models have emerged as a promising class of implicit layer models, which abandon traditional depth by solving for the fixed points of a single nonlinear layer. Despite their success, the stability of the fixed points for these models remains poorly understood. By considering DEQ models as nonlinear dynamic systems, we propose a robust DEQ model named LyaDEQ with guaranteed provable stability via Lyapunov theory. The crux of our method is ensuring the Lyapunov stability of the DEQ model's fixed points, which enables the proposed model to resist minor initial perturbations. To avoid poor adversarial defense due to Lyapunov-stable fixed points being located near each other, we orthogonalize the layers after the Lyapunov stability module to separate different fixed points. We evaluate LyaDEQ models under well-known adversarial attacks, and experimental results demonstrate significant improvement in robustness. Furthermore, we show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better adversarial robustness.

Authors

• Haoyu Chu Institute of Information Science, Beijing Jiaotong University Graduate School of Information Science and Technology, Osaka University Beijing Key Laboratory of Advanced Information Science and Network Technology
• Shikui Wei Institute of Information Science, Beijing Jiaotong University, China Beijing Key Laboratory of Advanced Information Science and Network Technology
• Ting Liu Tsinghua University
• Yao Zhao Institute of Information Science, Beijing Jiaotong University Beijing Key Laboratory of Advanced Information Science and Network Technology
• Yuto Miyatake Cybermedia Center, Osaka University

Keywords

• CV: Adversarial Attacks & Robustness
• ML: Adversarial Learning & Robustness
• ML: Classification and Regression
• ML: Deep Neural Architectures and Foundation Models