HEBO: Pushing The Limits of Sample-Efficient Hyper-parameter Optimisation

In this work we rigorously analyse assumptions inherent to black-box optimisation hyper-parameter tuning tasks. Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers. Based on these findings, we propose a Heteroscedastic and Evolutionary Bayesian Optimisation solver (HEBO). HEBO performs non-linear input and output warping, admits exact marginal log-likelihood optimisation and is robust to the values of learned parameters. We demonstrate HEBO’s empirical efficacy on the NeurIPS 2020 Black-Box Optimisation challenge, where HEBO placed first. Upon further analysis, we observe that HEBO significantly outperforms existing black-box optimisers on 108 machine learning hyperparameter tuning tasks comprising the Bayesmark benchmark. Our findings indicate that the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, multiobjective acquisition ensembles with Pareto front solutions improve queried configurations, and robust acquisition maximisers afford empirical advantages relative to their non-robust counterparts. We hope these findings may serve as guiding principles for practitioners of Bayesian optimisation.

Authors

• Alexander I. Cowen-Rivers
• Wenlong Lyu
• Rasul Tutunov Huawei R&D
• Zhi Wang Tsinghua University
• Antoine Grosnit
• Ryan Rhys Griffiths
• Alexandre Max Maraval
• Hao Jianye
• Jun Wang University College London
• Jan Peters
• Haitham Bou-Ammar

Keywords

• Automated Reasoning
• Bayesian Optimization
• hyperparameter tuning
• Uncertainty