Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization

Distributionally robust optimization (DRO) is a powerful framework for training robust models against data distribution shifts. This paper focuses on constrained DRO, which has an explicit characterization of the robustness level. Existing studies on constrained DRO mostly focus on convex loss function, and exclude the practical and challenging case with non-convex loss function, e.g., neural network. This paper develops a stochastic algorithm and its performance analysis for non-convex constrained DRO. The computational complexity of our stochastic algorithm at each iteration is independent of the overall dataset size, and thus is suitable for large-scale applications. We focus on the general Cressie-Read family divergence defined uncertainty set which includes chi^2-divergences as a special case. We prove that our algorithm finds an epsilon-stationary point with an improved computational complexity than existing methods. Our method also applies to the smoothed conditional value at risk (CVaR) DRO.

Authors

• Qi Zhang School of Computer Science, Fudan University, Shanghai, China Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai, China
• Yi Zhou Southeast University Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China
• Ashley Prater-Bennette Air Force Research Laboratory
• Lixin Shen Syracuse University
• Shaofeng Zou University at Buffalo, the State University of New York

Keywords

• CSO: Constraint Optimization
• SO: Non-convex Optimization