Basic question about basic.match:Match...

[CholoTook]

Sorry, noob looking at the code. Where does the actual match in basic.match:Match take place?

I have an eval in mind that should !match, so I want to tweak the code to test it.

Looking here https://github.com/openai/evals/blob/main/evals/elsuite/basic/match.py I can't see the code where the match actually takes place 😅

Thanks,

[CholoTook]

Sorry, found it... followed the ideal trail:
https://github.com/openai/evals/blob/main/evals/api.py#L55

[CholoTook]

Adding the words not match, does not contain for SEO

[CholoTook]

After spending some time to invert the logic of FuzzyMatch, I wonder if I should just create a new class, FuzzyDoesNotMatch?

[douglasmonsky]

@CholoTook maybe my answer for the issue #1282 might help clarify some of your confusion. I have quoted some of the key points from my other post below.

If you delve into any of the existing evals in the registry directory, you're likely to find a config snippet similar to this yaml file:
class:evals.elsuite.basic.includes:Includes
This essentially instructs the script to apply the evaluation logic located in evals/elsuite/basic/includes.py, specifically the Includes class. Most of the other classes found in the elsuite directory follow a similar pattern, though the details may vary.

All of the evaluation classes in elsuite inherit from the abstract base class Eval. Any class that inherits from this base class must implement a run() and an eval_sample() method to operate within the existing framework. This abstract base class also provides auxiliary methods to facilitate multi-threaded evaluations among other things such as helper methods.

However, an important note to keep in mind when contributing to this project:

"Important: Please note that we are currently not accepting Evals with custom code! While we ask you to not submit such evals at the moment, you can still submit modelgraded evals with custom modelgraded YAML files."

It's crucial to note that, although I am an active contributor, I'm not officially associated with OpenAI, nor am I a collaborator or code owner of this project. Therefore, if you prefer to wait for a response from a more authoritative source, it's completely understandable. I'm here to assist as much as I can, but for definitive answers, please refer to the official project leads or maintainers.

[CholoTook]

I saw the we are currently not accepting Evals with custom code and decided to ignore it...

I'll look at Includes to see if I can do 'not includes'.

Cheers,

[CholoTook]

Thanks @douglasmonsky I see the Includes thread I was following was just a generic pointer from your side.

For now I'll stick with my 'does not fuzzy match' hack, which is a PR somewhere... It's not perfect because the metrics are for matching, but it has the right logic at least.

Cheers,