The Effect of Diversity in Meta-Learning

Recent studies show that task distribution plays a vital role in the meta-learner's performance. Conventional wisdom is that task diversity should improve the performance of meta-learning. In this work, we find evidence to the contrary; (i) our experiments draw into question the efficacy of our learned models: similar manifolds can be learned with a subset of the data (lower task diversity). This finding questions the advantage of providing more data to the model, and (ii) adding diversity to the task distribution (higher task diversity) sometimes hinders the model and does not lead to a significant improvement in performance as previously believed. To strengthen our findings, we provide both empirical and theoretical evidence.

Authors

• Ramnath Kumar Google Research, India
• Tristan Deleu Mila, Quebec Artificial Intelligence Institute, Université de Montréal
• Yoshua Bengio Mila University of Montreal

Keywords

• ML: Active Learning
• ML: Classification and Regression
• ML: Deep Neural Network Algorithms
• ML: Evaluation and Analysis (Machine Learning)
• ML: Meta Learning
• ML: Multi-instance/Multi-view Learning
• ML: Optimization
• ML: Other Foundations of Machine Learning
• ML: Representation Learning
• ML: Transfer, Domain Adaptation, Multi-Task Learning