Is there a way to create a custom early stopping criterion?

[erussell92]

Hello all! I am so glad that this package exists and thus far have had a great experience with it.

For context: I am using the HillClimbingOptimizer to test an explanation method for research. In this particular case, I am working with a binary classification problem where every point has a true label of belonging to either the positive class or the negative class. The results of the optimization return parameters defining a set. For the positive class, this set is always non-empty and I have no issues using the stopping criterion of n_iter_no_change.

For the negative class, however, it should be an empty set. Because of the way the parameters are set up. This would be indicated by only one of the parameters needing to be returned as 0. Is there a way for me to create a custom stopping criterion like n_iter_no_change but for only one or two of the parameter results?

Thank you!

[SimonBlanke]

Hello @erussell92,

I am glad you had a great experience with this package. Let's see if I can make it better.

Making the early_stopping more flexible is something I find interesting. For this I have some questions about your use-case.

So the return of the objective-function in GFO is either non-empty (positive) or empty (negative), right? And if the negative case gets returned multiple times in a row the early_stopping gets triggered, because there is no apparent change in the "score". Is this the problem, that you described?

If there are further explanations needed, maybe you could write some simple pseudo-code to explain:

...
def objective_function(opt):
  ...
  return positive class / negative class
...

[erussell92]

Thank you so much for your response!

It's not that early stopping is being triggered because there is no change in the score, rather that we cannot get early stopping to be triggered in the negative case because the nature of the optimal solution presents unique circumstances where the parameters that define it are not unique.

The method I am working on is an explanation method for computer vision classification task. The objective function will return a loss score for an image and a bounding box. The objective function takes arguments in the form of the image and a vector of four parameters that define the bounding box with the first two elements being the coordinates of the top left corner and the last two elements being the height and width of the box. The objective function should be minimized for the optimal bounding box.

...
def objective_function(image, x , y , h , w):
    ...
    #computes score for the box defined by vector [x,y,h,w]

return score
...

In our case, either the image contains the object (positive class with non-empty optimal box) or does not (negative class with empty optimal box). In the case when the box is the empty set, either of the parameters h or w will be zero. Since the other parameters may change, this is where we are having issues with early stopping, because the representation of the empty set is not unique. The GFO should return a vector [x, y, h, w] and any box [x, y, 0, w] or [x, y, h, 0] would be the empty set (a box with 0 height and/or width). Since this is such a unique case, we were hoping there might be a way where we could trigger early stopping if the GFO returned an empty set multiple times in a row. For this, we were hoping if we could possibly define an early stopping criterion where the optimal parameters returned meet some certain criteria that represents a unique case n_times (such as h*w = 0).

Again, thank you so much for answering and please let me know if I can provide more context for clarity!

[SimonBlanke]

Thank you for this detailed explanation! It is nice to see such an interesting use-case.

So this problem is interesting, because you want to enable early-stopping by the condition of the parameters (but normally the condition is the score). And if I understood this correctly you could already solve this with some conditions in the objective-function:

import numpy as np
...
def objective_function(image, x , y , h , w):
    ...
    #computes score for the box defined by vector [x,y,h,w]

  if h == 0 or w == 0:
    return np.inf
return score
...

With this 'hack' that score gets automatically set to the worst possible value (GFO can handle inf/nan) if 'h' or 'w' are 0. So if we have the negative case n_iter_no_change-times in a row the early-stopping will kick in. Would that solve your problem or is there more to it?

[erussell92]

Thank you SO much! This did work well. I don't know why I didn't think to do this before. Thank you for all of your help and for your awesome module!

[SimonBlanke]

I am glad I was able to help you! It was also helpful to me, because questions from users give me inspiration for new features and improvements. So feel free to post questions/issues in the future :-)