Successive Halving Classifier for Hyperparameter tuning- HoeffdingTreeClassifier

[medha-chippa]

I'm trying to perform hyperparameter tuning on the HoeffdingTreeClassifier using the Successive Halving Classifier. Ive used the formula budget = ( 2kn/eta ) to calculate the budjet, where k = no of parameter combinations, n = no of samples in the dataset, eta = 2.
The following is the code to the parameter grid:
param_grid2 = utils.expand_param_grid(model_ht,{
'Hoeff_Tree': {
'grace_period': [np.arange(25, 525, 50)],
'max_depth': [np.arange(2, 11, 1)],
'split_criterion': [np.array(['gini','ingo_gain','hellinger'])],
'split_confidence': [np.linspace(0.000000001,0.1,20)],
'tie_threshold': [np.linspace(0.001,1.0,20)],
'leaf_prediction': [np.array(['mc','nb','nba'])],
'nb_threshold': [np.arange(25, 525, 50)]
}
})

The no of samples in the dataset are 64
From the above grid, i see that there are 3240000 combinations of hyperparameters (please correct me if i'm wrong). The Budget becomes equal to 207,360,000.

The following is my code to defining the succesive Halving Classifier:

sh = expert.SuccessiveHalvingClassifier(models=param_grid2,
metric=metrics.Accuracy(),
budget=207360000,
eta=2,
verbose=True)

Please help me rectify the error.

Thank you

[MaxHalford]

Hello! Could you please provide a minimal reproducible example that we can run? At a minimum please format your post to make it legible :)

[medha-chippa]

The following is a minimal reproducible example of my code with the iris dataset:

Please help me rectify this.
Thank you

[MaxHalford]

Even better, can you provide as copy pastable code here please? It's much easier if we can copy paste it.

[medha-chippa]

from sklearn.datasets import load_iris
iris= load_iris()
X= iris.data
y= iris.target

X_df=pd.DataFrame(data=X,columns = iris.feature_names)
y_nm=pd.Series(data=y,name='Target')

model_ht=tree.HoeffdingTreeClassifier()

param_grid2 = utils.expand_param_grid(model_ht,{
'Hoeff_Tree': {
'grace_period': [np.arange(25, 525, 50)],
'max_depth': [np.arange(2, 11, 1)],
'split_criterion': [np.array(['gini','ingo_gain','hellinger'])],
'split_confidence': [np.linspace(0.000000001,0.1,20)],
'tie_threshold': [np.linspace(0.001,1.0,20)],
'leaf_prediction': [np.array(['mc','nb','nba'])],
'nb_threshold': [np.arange(25, 525, 50)]
}
})

sh = expert.SuccessiveHalvingClassifier(models=param_grid2,
metric=metrics.Accuracy(),
budget=486000000,
eta=2,
verbose=True)

#No of samples in the dataset = 150
#No of Hyperparameter combinations = 3240000
#budget = (2 x 3240000 x 150)/2

[MaxHalford]

Have you figured this out by yourself? I checked just now and code was wrong in a lot of ways. You generated a grid parameters but you didn't use it to instantiate models so of course it won't work.

[medha-chippa]

yes, i've corrected my code and got it up and running. The hyper-parameters generated however do not result in the best model in terms of accuracy. I've had to manually try different parameter combinations to arrive at the best accuracy. Is there a way of fixing this?
Thank you

[MaxHalford]

It all depends on the grid that you generate. Are you saying that the procedure didn't select the best model from the grid? If so reducing eta and increasing the budget can help.

[medha-chippa]

ok, i'll try that thank you

[medha-chippa]

I've tried reducing the eta and increasing the budget, after which the progressive_val_score() is called with accuracy as the metric. The best accuracy is displayed. But the grid.best_model method returns the default model hyper parameters and not the best hyper parameters. What is the reason for this? Is there a way of preventing this from happening?