Botorch speed optimization

[giokanele]

Hello,
I have been creating a BO-torch model infrastructure that optimizes a real-time black box function with approximately 14 parameters. The model has always performed slowly and now I am tackling the issue. This line of code:
candidates, acqf_list = optimize_acqf(acq_function=UCB, bounds=bounds, q=q_points, num_restarts=num_restarts, raw_samples=raw_samples, options= {"maxiter": 500})

takes approximately 2.5s / iteration. My initial thought was that the machine I was running this on was not capable of maintaining a low iteration time. However, when trying to utilize GPUs the iteration time increases significantly. Are there any resources that I could use to integrate botorch and GPUs? Or does someone know a potential reason for Botorch to be taking so long in this specific operation? I suspect I am doing something wrong but I keep missing it....
Thanks! :)

[Balandat]

What are the parameters of your problem (dimension of the search space, number of observations)? 2.5 seconds is not a whole lot of time for optimizing an acquisition function for Bayesian Optimization. How long does a function evaluation take in your setting?

[giokanele]

Hello Balandat, apologies for the delay, and thank you for the help.

The parameter space changes quite significantly depending on where I am running the code. However, in the most extreme cases, the parameter space is composed of ~12 individual dimensions whose value can range individually from 0 - 1000. Due to the nature of my problem every run I have to gather new observation data which is usually 50 - 100 randomly selected points in the parameter space. The function evaluation is ~2s/it

[Balandat]

I see. A ~2s/it runtime is at the very low end of what I would recommend using Bayesian Optimization for. The default parameters in optimize_acqf are selected to be geared for much more costly problems where the seconds (or even minutes) spend on optimizing the acquisition function are negligible relative to the cost / duration of the function evaluation.

You can speed things up by

• reducing num_restarts
• reducing raw_samples
• reducing the maxiter setting or passing timeout_sec to explicitly limit the runtime

This will reduce runtime of optimize_acqf at the cost of the quality of the optimization. This may be ok though in your setting as it will allow you to collect more data points. The flipside of this though is that as you collect more data points, the model fitting and evaluation will become a lot more costly.