Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mmlu #152

Merged
merged 71 commits into from
Feb 20, 2024
Merged

Mmlu #152

merged 71 commits into from
Feb 20, 2024

Conversation

kkahadze
Copy link
Collaborator

No description provided.

kkahadze and others added 30 commits November 28, 2023 21:23
@kkahadze
feat: backoff implemeted
386d286
@kkahadze
feat: implemented starting at any index in dataset.
9e4b97d
@kkahadze
feat: MMLU pipeline implemented with partial testing
f2174b9
@kkahadze
feat: removed human-sexuality section from MMLU dataset
a3aae71
@kkahadze
feat: MMLU experiment pipeline with non-role prompts.
51ea13d
@kkahadze
style: black
615d443
@kkahadze
chore: remove outdated comment
9e22602
@kkahadze
chore: remove random from reqs
7aa9888
@kkahadze
chore: commented out unfinished test
37c3366
@kkahadze
chore: blacken
549ae17
@kkahadze
feat: added correct order to answer and reasoning in json prompts
5344c22
@kkahadze
tests: evaluate_mmlu response and load_mmlu tests added
fbeba78
@kkahadze
feat: rereading implemented
5ee9355
@kkahadze
changed directory that benchmarking files save to
607a7ee
@kkahadze
Add .gitignore to keep benchmarking directory
3ea3a64
@kkahadze
feat: seeding and temperature implemented.
27e7bab
@kkahadze
chore: rename test for role prompting to benchmarking since it is all…
512426c 
… types of benchmarking now.
@kkahadze
test: modular prompts preliminary test, also changed src.prompt_syste…
1d20fb6 
…matic... to just prompt_systematic... because it wasn't working right.
@kkahadze
test: Added more modular prompts tests.
115e59c
@kkahadze
feat: modular prompting, removed logging to console on successful cal…
1c2bba2 
…ls to API.
@kkahadze
tests: More modular prompting tests.
5ad4b73
@kkahadze
feat: Added 10-shot prompts
f28fb73
@kkahadze
testing for 10-shot prompt
3d319e0
@kkahadze
feat: Reinforcement at user message for it to return json
5f125ae
@kkahadze
feat: Fixed typo in idiot promps
ceccea0
@kkahadze
Add resulting MMLU official dataset folder to gitignore.
f1777c8
@kkahadze
chore: most recent run settings
5cb2c14
@kkahadze
feat: mmlu download script
22b2867
@kkahadze
feat: prompt for mmlu without json mode
48c0991
@kkahadze
feat: checking correctness of outputs that are non-json when not in j…
bf4d8ae 
…son mode.
kkahadze added 19 commits January 19, 2024 16:05
@kkahadze
chore: Removed old useless prompts.
e3839b6
@kkahadze
test: fixed load_mmlu test.
d0d23d3
@kkahadze
feat: Fixed prompting order of question and prompt.
9425f09
@kkahadze
test: Added tests for new code.
a101422
@kkahadze
chore: added comments.
e8d5687
@kkahadze
feat: put newline between prompt and question.
160784e
@kkahadze
feat: first experiment script.
1ddeec4
@kkahadze
feat: fixed quotes to be parentheses.
1bcdafb
@kkahadze
feat: fixed wrong 0-shot CoT prompt.
d25bfa6
@kkahadze
tests: new answer extraction algorithm tests.
6baed76
@kkahadze
chore: fixed 0-shot CoT prompt construction.
9119b86
@kkahadze
feat: added new answer extraction.
7936d50
@kkahadze
tests: added and changed tests.
53e23ae
@kkahadze
feat: new prompts added to json. json currently not integrated with p…
629a645 
…rompting pipeling for newest prompts.
@kkahadze
feat: latest benchmarking updates.
a0aac45
@kkahadze
feat: self-con setup
c16e293
@kkahadze
chore: blacken
db36b48
@kkahadze
feat: benchmarking updated
ce96986
@kkahadze
chore: blacken
9501516
@kkahadze kkahadze self-assigned this Jan 31, 2024
@kkahadze kkahadze requested a review from trigaten January 31, 2024 23:59
@trigaten trigaten merged commit 612eb1c into main Feb 20, 2024
3 checks passed
@trigaten trigaten deleted the MMLU branch February 20, 2024 00:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@trigaten trigaten Awaiting requested review from trigaten

Assignees

@kkahadze kkahadze

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kkahadze @trigaten