On 16 July, Erik Bodin presented ‘Modulating Surrogates for Bayesian Optimization’ at the International Conference on Machine Learning (ICML); the premier gathering of professionals dedicated to the advancement of the branch of artificial intelligence known as machine learning.

Abstract

Bayesian optimization methods are designed to find globally optimal settings of parameters when standard approaches (such as gradient descent) cannot be applied; for example, the objective function is very expensive to evaluate, we cannot obtain gradients of the objective or the objective has many local minima or stochastic components. Typical examples range from optimising hyper parameters for engineering design where the objective is the result of a long and expensive simulation (e.g. design constraints for a jet engine) to searching for optimal architectures for deep neural networks (e.g. numbers and size of layers) where evaluating the objective involves a complete training cycle for the deep model.Bayesian optimisation methods proceed by performing a sequence of decisions about which parameter settings to try to search for the best settings. Each new query will be expensive to evaluate so it is important to make the best use of all the previous queries in determining where to try next. Our paper builds a new type of “surrogate” model (an approximation of the true, unknown objective) that matches the important properties of the objective function (e.g. where the big picture minima lie) while ignoring local properties that can distract existing search methods.As an example, the figure below shows standard search methods on the left and middle and the result of using the new surrogate on the right – the “acquisition” function indicates that it would be sensible to take an exploratory step in the middle of the function space whereas the standard approach is distracted by local changes and remains stuck in a local minimum.

 

Modulating Surrogates for Bayesian Optimization

Paper: http://cs.bath.ac.uk/~nc537/publications/#icml20_modulating_surrogates

Erik Bodin, Markus Kaiser, Ieva Kazlauskaite, Zhenwen Dai, Neill D. F. Campbell and Carl Henrik Ek, International Conference on Machine Learning (ICML), 2020

This is collaborative work with the University of Bristol, Technische Universität München and Spotify Research.