Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework validation for measurements and pending_experiments #456

Merged
merged 22 commits into from
Feb 17, 2025

Conversation

Scienfitz
Copy link
Collaborator

@Scienfitz Scienfitz commented Jan 3, 2025

Fixes #453

Notes re measurements validation

  • Two utilities (one for parameters and one for targets) for input validation has been extracted. Additional validations for binary targets have been added. The utilities contain parts from add_measurements and fuzzy_row_match
  • As a result fuzzy_row_match does not perform any validation anymore. Whether numerical parameter measurements outside of numerical tolerances are allowed is now decided by outside of this function, it will always match numerical parts on the smallest distance
  • add_measurements now simply calls the utilities
  • tests for invalid parameter input have been extended
  • Any duplicated validation is excluded due to validated dataframes being wrapped in ValidatedDataFrame

Notes re pending_experiments validation

  • Campaign.recommend now performs validation of pending_experiments
  • tests for invalid pending_experiments have been added

Notes on recommenders

  • Pure recommenders now validate measurements and pending_experiments in recommend
  • Despite being derived from pure recommenders, bayesian recommenders have their own implementation of measurements and pending_experiments because the validation needs to happen before the call to the base class
  • meta recommenders dont perform any validation as at the base level there will always be a pure recommender which performs validation if still needed

Notes on numerical_measurements_must_be_within_tolerance

  • The only option to activate this behavior is for measurements in Campaign.add_measurements
  • In particular, i) pending_experiments are always valid independent of any numerical tolerance deviation (both in recommenders and campaign) and ii) measurements are always valid independent of numerical tolerances for recommenders

@Scienfitz Scienfitz added the enhancement Expand / change existing functionality label Jan 3, 2025
@Scienfitz Scienfitz self-assigned this Jan 3, 2025
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Scienfitz, thanks the refactor 🏗️ Below my comments

@Scienfitz Scienfitz force-pushed the feature/exp_input_validation branch 3 times, most recently from 0a42492 to a412305 Compare January 6, 2025 12:36
Copy link
Collaborator

@AVHopp AVHopp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Scienfitz Scienfitz force-pushed the feature/exp_input_validation branch from 7056db6 to 969dea4 Compare January 14, 2025 13:20
@Scienfitz Scienfitz requested a review from AdrianSosic January 14, 2025 13:21
@Scienfitz

This comment was marked as outdated.

@Scienfitz Scienfitz force-pushed the feature/exp_input_validation branch 3 times, most recently from 326a1b5 to d1f4c50 Compare February 10, 2025 17:02
@Scienfitz Scienfitz force-pushed the feature/exp_input_validation branch from 107e225 to c3ffc9f Compare February 12, 2025 17:26
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First round of very minor comments, you'll still get a full review

Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So now here's finally your full review 🙃

@Scienfitz Scienfitz force-pushed the feature/exp_input_validation branch from cddbee4 to bb5a8d6 Compare February 14, 2025 17:28
@Scienfitz Scienfitz merged commit 949b897 into main Feb 17, 2025
10 of 11 checks passed
@Scienfitz Scienfitz deleted the feature/exp_input_validation branch February 17, 2025 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Expand / change existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pending experiments with CustomDiscreteParameter value not in search space gives cryptic error
3 participants