Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add experimental features flag to be only accessible in core config #1003

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented Nov 15, 2024

add facility to unlock experimental CLI options if a core config flag is set

Verification

  • run python -m garak --help, experimental features should not be enabled
  • change system.enable_experimental to true in garak/resources/garak.core.yaml
  • run python -m garak --help, there should be a message between the usage summary and options that experimental features are enabled

Question - would this feature be better off in a site config? I'm concerned about garak.core.yaml's value getting accidentally clobbered in a development commit

@leondz leondz added enhancement Architectural upgrades cli Command-line interface functions labels Nov 15, 2024
@jmartin-tech
Copy link
Collaborator

Question - would this feature be better off in a site config? I'm concerned about garak.core.yaml's value getting accidentally clobbered in a development commit

One option is to add garak.core.yaml to the .gitignore, changes to it could still committed they would just require an additional flag force adding any file changes to a commit.

I could see allowing this to be in garak.site.yaml as well, depending on what level of accessibility we want exposed. The banner combined with the serialized system config in the report log makes it clear if the option is turned on.

I am wondering if we should make this more of a feature flag set, maybe we should accept a list of experimental features to enable vs boolean value. That may be overkill for now though.

@leondz
Copy link
Collaborator Author

leondz commented Nov 20, 2024

Alright. I think for now retaining the checks predicated on garak.core.yaml are more important than the odd mis-commit, so will proceed without a .gitignore. Site config is a great place for this flag but putting it there means changing the order of loading in a way that affects garak.cli processing, and I haven't taken a call on whether that's a good idea yet. So let's proceed as is

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli Command-line interface functions enhancement Architectural upgrades
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants