Optimize Like a Pro with Bayesian Optimization

Bayesian optimization (BayesOpt) is a powerful tool that has found wide application in global optimization tasks, such as hyperparameter tuning, protein engineering, synthetic chemistry, robot learning, and even baking cookies. One of the main reasons BayesOpt is so useful for these problems is because they all involve optimizing black-box functions that are expensive to evaluate.

A black-box function's underlying mapping from inputs (configurations of the thing we want to optimize) to outputs (a measure of performance) is unknown. However, we can attempt to understand its internal workings by evaluating the function for different combinations of inputs. Because each evaluation can be computationally expensive, we need to find the best inputs in as few evaluations as possible.

BayesOpt works by repeatedly constructing a surrogate model of the black-box function and strategically evaluating the function at the most promising or informative input location, given the information observed so far. Gaussian processes are popular surrogate models for BayesOpt because they are easy to use, can be updated with new data, and provide a confidence level about each of their predictions.

The performance of BayesOpt depends on whether the confidence intervals predicted by the surrogate model contain the black-box function. Traditionally, experts use domain knowledge to quantitatively define the mean and kernel parameters to express their expectations about what the black-box function should look like. However, for many real-world applications like hyperparameter tuning, it is very difficult to understand the landscapes of the tuning objectives. Even for experts with relevant experience, it can be challenging to narrow down appropriate model parameters.

Fortunately, a team of researchers at Google AI has proposed Hyper BayesOpt (HyperBO), a highly customizable interface with an algorithm that removes the need for quantifying model parameters for Gaussian processes in BayesOpt. HyperBO pre-trains a Gaussian process model on data from previous tasks that are relevant to the current task they are trying to solve. This allows experts to simply select the previous tasks and HyperBO will automatically define the model parameters before running BayesOpt.

HyperBO enjoys theoretical guarantees on the alignment between the pre-trained model and the ground truth, as well as the quality of its solutions for black-box optimization. The team has shared strong results of HyperBO both on their new tuning benchmarks for near-state-of-the-art deep learning models and classic multi-task black-box optimization benchmarks (HPO-B). They have also demonstrated that HyperBO is robust to the selection of relevant tasks and has low requirements on the amount of data and tasks for pre-training.

If you're looking to optimize like a pro, Bayesian optimization and Hyper BayesOpt are the tools you need to explore. With their help, you can optimize your systems more efficiently, even when working with expensive black-box functions.

References:

https://ai.googleblog.com/2023/04/pre-trained-gaussian-processes-for.html