Feature importance in Hoeffding tree

[Piyushbalwani]

Hi Team,

How to calculate feature importance in Hoeffding tree, like sklearn decision tree ?

Thanks,
Piyush

[smastelini]

Hi @Piyushbalwani. Unfortunately, this cannot be done at the moment. I think there's some recent research about this topic in the data stream literature. We might give it a go in the future. Help is always welcome!

[Piyushbalwani]

Thanks @smastelini.
There is a parameter remove_poor_attrs for tree.HoeffdingTreeClassifier. How poor feature or attribute is removing? I didn't find code for this parameter either in github code section. Can you please help me in pointing out, might be classifying good and bad feature may help in calculating importance of feature?

[smastelini]

Hi @Piyushbalwani, the specific portion responsible for attribute removal/filtering is this one (in the classification case):

river/river/tree/hoeffding_tree_classifier.py

Lines 268 to 279 in eda2723

	if self.remove_poor_attrs:
	poor_atts = set()
	# Add any poor attribute to set
	for suggestion in best_split_suggestions:
	if (
	suggestion.feature
	and best_suggestion.merit - suggestion.merit
	> hoeffding_bound
	):
	poor_atts.add(suggestion.feature)
	for poor_att in poor_atts:
	leaf.disable_attribute(poor_att)

This was inherited from MOA and, as far as I am concerned, was proposed by Richard Kirkby in his Ph.D. thesis (section 3.3).

I believe a similar criteria could be used for ranking features (in combination with their split merit).

Note that there is some existing research in the realm of using trees to get feature scores:

• please check this paper
• and this one

Just to name a few. I'm far from being an expert on that matter. My research is mostly focused on decreasing the resource usage of incremental decision trees. I'm far from being an expert on that matter.

I hope these links can give you a starting point for further exploring this important topic!

[Piyushbalwani]

Hi @smastelini , Thanks for all your suggestions. For calculating feature importance, there is need of Gini Index or entropy, but somehow this information is not avaialable in dataframe returned by clf.to_dataframe(). Can you please help me to find a way to calculate gini index of each non-leaf node? Thanks!!!

[smastelini]

Hi @Piyushbalwani, the split merits are only used during the split attempts. They are not stored by the trees, so that's why they are not accessible via the to_dataframe method. You can check the tree splitters in river.tree.splitter to understand how the trees handle split merits.

The split criteria code is under tree.split_criterion. Theoretically, you could use these criteria along with the leaves' statistics to compute split merits anytime.

[smastelini]

Maybe you could develop a method shared by all the trees that would traverse the tree structure and compute the split heuristics using the up-to-date statistics. The split criterion could even be a parameter.