Invariant Random Forest: Tree-Based Model Solution for OOD Generalization

Out-Of-Distribution (OOD) generalization is an essential topic in machine learning. However, recent research is only focusing on the corresponding methods for neural networks. This paper introduces a novel and effective solution for OOD generalization of decision tree models, named Invariant Decision Tree (IDT). IDT enforces a penalty term with regard to the unstable/varying behavior of a split across different environments during the growth of the tree. Its ensemble version, the Invariant Random Forest (IRF), is constructed. Our proposed method is motivated by a theoretical result under mild conditions, and validated by numerical tests with both synthetic and real datasets. The superior performance compared to non-OOD tree models implies that considering OOD generalization for tree models is absolutely necessary and should be given more attention.

Authors

• Yufan Liao Renmin University of China City University of Hong Kong
• Qi Wu City University of Hong Kong
• Xing Yan Renmin University of China

Keywords

• ML: Ensemble Methods
• ML: Transfer, Domain Adaptation, Multi-Task Learning