Covariate-Shift Generalization via Random Sample Weighting

Shifts in the marginal distribution of covariates from training to the test phase, named covariate-shifts, often lead to unstable prediction performance across agnostic testing data, especially under model misspecification. Recent literature on invariant learning attempts to learn an invariant predictor from heterogeneous environments. However, the performance of the learned predictor depends heavily on the availability and quality of provided environments. In this paper, we propose a simple and effective non-parametric method for generating heterogeneous environments via Random Sample Weighting (RSW). Given the training dataset from a single source environment, we randomly generate a set of covariate-determining sample weights and use each weighted training distribution to simulate an environment. We theoretically show that under appropriate conditions, such random sample weighting can produce sufficient heterogeneity to be exploited by common invariance constraints to find the invariant variables for stable prediction under covariate shifts. Extensive experiments on both simulated and real-world datasets clearly validate the effectiveness of our method.

Authors

• Yue He Department of Computer Science and Technology, Tsinghua University, Beijing, China
• Xinwei Shen ETH Zurich
• Renzhe Xu Tsinghua University
• Tong Zhang Nanjing University of Science and Technology
• Yong Jiang Tsinghua Shenzhen International Graduate School
• Wenchao Zou Siemens China
• Peng Cui Tsinghua University

Keywords

• ML: Bias and Fairness
• PEAI: Safety, Robustness & Trustworthiness