Bayesian Optimization

This repository contains my work with Bayesian optimization and related topics. The main goal of this repository is to develop and curate a set of resources that myself and others can use to better understand (and utilize) Bayesian optimization.

Note

Special acknowledgement for content adapted and referenced in this repository is owed to my advisor Professor Aryan Deshwal (University of Minnesota) and his special topics course entitled AI for Sequential Decision Making. Acknowledgement is also owed to Roman Garnett, PhD and his brilliant book on Bayesian optimization. For a complete list of supplementary materials, please see the References section below.

Tip

Please raise an issue if you notice any issues, typos, and/or have any suggestions. Thanks!

📁 docs

This directory contains notes/tutorials on Bayesian optimization and related topics.

• Introduction to Bayesian Optimization
• Bayesian Decision Theory
• Gaussian Processes
• Covariance Functions and Kernels
• Model Evaluation and Selection
• Utility Functions
• Acquisition Functions
• GP Regression
• GP Classification

📁 examples

This directory contains notebooks exploring various examples of Bayesian optimization and its applications.

• BoTorch Tutorials
  This notebook contains tutorials from BoTorch's documentation and tutorials.
• Introduction to GPyTorch and GAUCHE
  This notebook synthesizes information from GPyTorch and GAUCHE's documentation regarding Gaussian processes for machine learning and how to apply them to irregular-structured input representations (i.e., molecular, graph, etc.).

📁 src

This directory contains from-scratch implementations of Bayesian optimization methods and the methods of related topics.

Use the commands below to create a new Conda environment with all of the necessary dependencies:

conda env create -f bo_env.yml
conda activate bo-env

Note that the commands above should also be used to create an environment to run the notebooks in the examples directory.

• branin.py
  This Python script compares the performance of analytic and monte-carlo acquisition functions implemented in BoTorch on the Branin function embedded in higher dimensions. The performance of each acquisition function is automatically tracked using Weights & Biases. To run the script, use this command: python3 branin.py (after activating the bo-env environment).

References

Below is a list of reference texts, papers, and other sources on Bayesian optimization and related topics. The BibTeX entries can be found in the assets/bibliography.bib file.

Reference Texts

• Bayesian Optimization by Roman Garnett (2023)
• Bayesian Optimization: Theory and Practice Using Python by Peng Liu (2023)
• Gaussian Processes for Machine Learning by Rasmussen & Williams (2019)

Papers

Other