A memory constrained bayesian optimization via robust online memory estimation

Abstract Bayesian optimization (BO) is a memory-intensive algorithm that requires training and evaluating an expensive objective function. In contrast to previous works that use an offline memory estimation to make BO memory-efficient, we propose a robust and simple online memory estimation method that requires training a model only for the first two iterations of the first epoch. Our memory estimation method is then integrated with a simple, performance-based surrogate model of BO in a seamless (or in sync) mode that enforces memory efficiency even if it does not bypass a preset threshold. The online memory estimation method has been evaluated on two different datasets, showing that it is more accurate than the existing offline method ( $$2.19\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>2.19</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> for MNIST and $$3.51\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3.51</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> for CIFAR datasets). Furthermore, compared to a memory-unaware baseline, the enhanced BO has no loss of accuracy and is $$11.31\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>11.31</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> memory-efficient for a simple CNN-based image classification, and $$5.03\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>5.03</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> memory efficient but $$9.27\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>9.27</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> slower for a more complex LSTM-based text classification (useful for a resource-constrained environment where delay is tolerable but memory is scarce), while it is $$2.6\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>2.6</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> memory efficient but $$1.23\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>1.23</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> slower on a pretrained ResNet50 model.