Comments from June 15

[mgelbart]

• print out some sort of progress indicators in synthetic-comparison (e.g. tqdm)
• if results directory doesn't exist, create it with os.mkdir (you can also check whether it exists with the os package)
• update README to reflect that some args are now positional
• stop using lists in LIPO code, it makes the code harder to read

work to do:

• keep working on those two data sets
• finish work in adaptive lipo on not recomputing pairwise distances at each iteration
  • you'll need to be careful that our preallocation of zeros and Infs is working as expected

[bradleypick]

This commit addresses both the use of lists in adaptive LIPO and the redundant computation of pairwise distances. It handles this by allocating all arrays up front, populating them with sensible values, and only using slices where necessary.

First, in this notebook is a quick check that the results generated by the use of adaptive_lipo in synthetic_comparison.py are the same before and after the changes.

Second, this seems to have sped up adaptive_lipo (dare I say significantly?).

Using just the holder_table function with a seed=0 on a single simulation (run) of adaptive_lipo I saw the following (measured in seconds using time.time()):

# of Iterations	Old AdaLIPO	New AdaLIPO
250	0.1363232	0.05544877
500	0.63668919	0.10076570
1000	4.84622860	0.40755009
2000	49.97320175	2.9249508

It's not very reproducible but I got the above results doing something similar to the following in two different terminal sessions (one running old version of adaptive_lipo and the other running the new version):

num_iters = [250 * 2**i for i in range(5)]
for n in num_iters:
    s = time.time()
    t = adaptive_lipo(holder_table, bnds, n, seed=0)
    diff = time.time() - s
    print(str(num_iters) + " took " + str(diff))

[mgelbart]

Just for educational purposes, would be fun to plot vs # iters and see quadratic vs. cubic behaviour in the new and old versions. Sent from my mobile phone.

…

On Jun 23, 2018, at 1:59 PM, Bradley Pick ***@***.***> wrote: This commit addresses both the use of lists in adaptive LIPO and the redundant computation of pairwise distances. It handles this by allocating all arrays up front, populating them with sensible values, and only using slices where necessary. First, in this notebook is a quick check that the results generated by the use of adaptive_lipo in synthetic_comparison.py are the same before and after the changes. Second, this seems to have sped up adaptive_lipo (dare I say significantly?). Using just the holder_table function with a seed=0 on a single simulation (run) of adaptive_lipo I saw the following (measured in seconds using time.time()): # of Iterations Old AdaLIPO New AdaLIPO 250 0.1363232 0.05544877 500 0.63668919 0.10076570 1000 4.84622860 0.40755009 2000 49.97320175 2.9249508 It's not very reproducible but I got the above results doing something similar to the following in two different terminal sessions (one running old version of adaptive_lipo and the other running the new version): num_iters = [250 * 2**i for i in range(5)] for n in num_iters: s = time.time() t = adaptive_lipo(holder_table, bnds, n, seed=0) diff = time.time() - s print(str(num_iters) + " took " + str(diff)) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[mgelbart]

Although it doesn't look quite quadratic, interestingly. Going from 500 to 1000, the time quadruples, which is typical quadratic behaviour. But then from 1000 to 2000 it goes up about 7x, strangely. I guess these things are hard to predict. Anyway, at 1000 iters, which is what we want for reproducing the paper, the speedup seems to be about 10x - that is definitely significant. Nice work 🎉

[mgelbart]

BTW, "in two different terminal sessions" - were they done concurrently? It's better not to, I'd think.

[bradleypick]

They were not run concurrently because I figured that could muck up the results. If I find time I would like to plot the above comparison for all the synthetic objective functions to get a better sense of the impact of the dimension of the search space on performance.