Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PackageHallucinationProbe shouldn't show in --list_probes #904

Open
leondz opened this issue Sep 6, 2024 · 0 comments
Open

PackageHallucinationProbe shouldn't show in --list_probes #904

leondz opened this issue Sep 6, 2024 · 0 comments
Labels
architecture Architectural upgrades cli Command-line interface functions probes Content & activity of LLM probes

Comments

@leondz
Copy link
Collaborator

leondz commented Sep 6, 2024

PackageHallucinationProbe is intended as an abstract class and shouldn't show in --list_probes, but does.

One option is to refactor it to be a mixin, a pattern we use some places.

Another pattern we use, that I'm not a huge fan of, is a "lead" probe that other, similar probes inherit from. My reservations with this pattern are (1) that the hierarchy is not always clear, i.e. it's uncertain which probe is the "lead" one when it has a name like PackageHallucinationRuby, for example; (2) that it doesn't make sense to inherit from a lead probe with one purpose and then to remove that purpose and add another.

Drawbacks with the mixin pattern - mixed inheritance isn't always so clean in python, and getting parent constructors to behave as intended can be non-trivial.

One alternative could be to use the _-prefix convention to denote plugin classes that shouldn't be shown in --list or loaded with load_plugin(), etc. E.g. in this instance we'd have probes.packagehallucination._PackageHallucinationProbe(Probe).

@leondz leondz added probes Content & activity of LLM probes ux Interface & interaction improvements architecture Architectural upgrades cli Command-line interface functions and removed ux Interface & interaction improvements labels Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
architecture Architectural upgrades cli Command-line interface functions probes Content & activity of LLM probes
Projects
None yet
Development

No branches or pull requests

1 participant