Hyperparameter tuning of HoeffdingTreeClassifier with RandomizedSearchCV of sklearn

[TxusLopez]

Hi,

I get an error when I try to tune the parameters of a HoeffdingTreeClassifier using the RandomizedSearchCV of scikit-learn.

This is the code:

clf=tree.HoeffdingTreeClassifier()
new_clf=compat.convert_river_to_sklearn(clf)

#For RGS and Halving
grid = dict(
    grace_period=np.arange(25, 525, 50),
    max_depth=np.arange(2, 11, 1),
    split_criterion=np.array(['gini','ingo_gain','hellinger']),
    split_confidence=np.linspace(0.000000001,0.1,20),
    tie_threshold=np.linspace(0.001,1.0,20),
    leaf_prediction=np.array(['mc','nb','nba']),
    nb_threshold=np.arange(25, 525, 50)
)

rgs=RandomizedSearchCV(estimator=new_clf,param_distributions=grid)
rgs.fit(X_train,y_train)

The error is: "ValueError: Invalid parameter tie_threshold for estimator River2SKLClassifier(HoeffdingTreeClassifier). Check the list of available parameters with estimator.get_params().keys()."

Thanks in advance

[medha-chippa]

Hello, I keep getting the same error when I try to tune the parameters of an extremely fast decision tree classifier. Please let me know if you happen to fix this.

Thank you

[TxusLopez]

This could be an issue related to the compatibility between "river" and "scikit-learn". I guess it was discussed in the "creme" project, but as it has been merged with "scikit-multiflow" (and then emerging "river"), this issue has probably been lost in the current discussions. As soon as I fixed it, I will let you know. Same as you ok? :-)

[medha-chippa]

Sure. I'll get back to you as soon as I'm able to fix it.
Thank you

[HarshaAsh]

From my understanding, cross-validation is difficult (and probably not currently implemented) for streaming machine learning. Hyperparameter tuning can be done by implementing stacking or halving methods: https://riverml.xyz/latest/api/expert/SuccessiveHalvingRegressor/

[TxusLopez]

Hi,

it is true that cross-validation-based hyperparameter tuning approaches take too much time to be considered under streaming conditions. But, in some scenarios, they could be useful: we could alleviate some parameters (e.g. number of folds, number of training instances, etc.) or even carrying out some kind of incremental search. I still find useful the possibility of including them in river library, or at least, make sklearn techniques compatible with river.

[MaxHalford]

Is this something we want to look into @smastelini? I would say no.

[smastelini]

HP tuning is a valid concern to ML tasks and online learning is not an exception. However, it does sound strange to me to tune dynamical/incremental models in a batch fashion.

I strongly believe tuning strategies should be aligned with the constraints and requirements of online learning. For instance, a good set of HPs might become outdated or inadequate after a concept drift. I starting to hear some buzz concerning AutoML and HP tuning for stream mining. I believe we should shift the focus to this kind of technique rather than batch-based ones.

Now, if someone really wants to perform cross-validation and so on, it's totally doable. So, the inability to use sklearn's tools must be related to the compatibility wrapper.

[MaxHalford]

Yes I agree with you on the approach. But I guess being fully compatible with scikit-learn is still a nice to thing to have.