IndexError for AutoQKeras with hyperband mode

[apfusco]

I get the following stack trace when calling get_best_model() on the AutoQKeras object using hyperband mode:

Traceback (most recent call last):
  File "exp_nns.py", line 533, in <module>
    auto_qkeras_model = t.auto_mnist_cnn(auto_qkeras=True, mode="hyperband", tune_filter="layer", activation_bits=8)
  File "exp_nns.py", line 387, in auto_mnist_cnn
    aqnn = autoqk.get_best_model()
  File "~/.pyenv/versions/hls4ml_venv/lib/python3.8/site-packages/qkeras/autoqkeras/autoqkeras_internal.py", line 976, in get_best_model
    params = self.tuner.get_best_hyperparameters()[0]
IndexError: list index out of range

It does not run into an error if I use random or bayesian mode. Does anyone know what could be causing this IndexError?

[danielemoro]

It looks like self.tuner.get_best_hyperparameters() is an empty list, and so the first element returns an IndexError.

If you look at the source code of the method here it seems that this is most likely because it did not find any completed trials. For example, perhaps all of the trials errored out before completion.

Please take a look and see if this is what is causing the issue. If you are still encountering this error, please provide a full code example so we can reproduce the error you see. Thanks!

[apfusco]

Thanks for responding quickly to my issue. I am trying to figure out why none of the trials completed. Here is the code I used to run AutoQKeras:

def mnist_dataset(one_hot = True):
    (x_train,y_train),(x_test,y_test) = keras.datasets.mnist.load_data()
    if one_hot:
        y_train = keras.utils.to_categorical(y_train)
        y_test = keras.utils.to_categorical(y_test)
    return (x_train,y_train),(x_test,y_test)


goal = {
        "type": "energy",
        "params": {
                "delta_p": 8.0,
                "delta_n": 8.0,
                "rate": 2.0,
                "stress": 1.0,
                "process": "horowitz",
                "parameters_on_memory": ["sram", "sram"],
                "activations_on_memory": ["sram", "sram"],
                "rd_wr_on_io": [False, False],
                "min_sram_size": [0, 0],
                "source_quantizers": ["int8"],
                "reference_internal": "int8",
                "reference_accumulator": "int32"
        }
}

if __name__ == '__main__':
    # make the dataset
    (x_train,y_train),(x_test,y_test) = mnist_dataset()

    # make the nn
    nn = simple_mnist_cnn()

    run_config = {
            "output_dir": "./autoqkeras_output_dir",
            "goal": goal,
            # "quantization_config": quantization_config,
            "learning_rate_optimizer": False,
            "transfer_weights": False,
            "mode": "hyperband",
            "seed": 42,
            "activation_bits": 8,
            "limit": limit,
            "tune_filters": "layer",
            "tune_filters_exceptions": "",
            "distribution_strategy": tf.distribute.get_strategy(),
            "layer_indexes": range(1, len(nn.layers) - 1),
    }

    run_config["max_trials"] = 20

    # train and test
    autoqk = AutoQKeras(nn, metrics=["acc"], custom_objects={}, **run_config)
    autoqk.fit(x_train, y_train, validation_data=(x_test, y_test), batch_size=1024, epochs=20)
    aqnn = autoqk.get_best_model()