Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use Yival to run Experiment #25

Merged
merged 43 commits into from
Jan 25, 2024
Merged

Use Yival to run Experiment #25

merged 43 commits into from
Jan 25, 2024

Conversation

semio
Copy link
Collaborator

@semio semio commented Nov 11, 2023 •
edited
Loading

Dependencies updates

  • Yival: I forked Yival: https://github.com/semio/YiVal, changed it to use latest litellm (for google palm support), and added caching to the openai based evaluator
  • set python version to 3.11, pydantic to v2 due to requirements in Yival

changes in automation_api

  • don't evaluate formulas in all sheets
  • adapted to pydantic v2
  • added Metric sheet

Yival experiment

  • custom functions/configurations are in yival_experiments/custom_configuration
  • the latest evaluation results were uploaded to yival_experiments/output

TODOs

  • add function to generate result table and upload to ai eval spreadsheet.

semio added 30 commits October 11, 2023 22:15
@semio
add yival to dependency
e61f841
@semio
migrate to pydantic v2
2307241 
because yival require v2
@semio
langchain still use pydantic v1
4f8b1cc
@semio
update pandera version and make the code work
69c20d1
@semio
experiment files so far
3f650a2
@semio
add readme
3339a12
@semio
some more files
0dac683
@semio
update readme
d8bb1c0
@semio
update readme
b51d444
@semio
add metrics data sheet
4a59740
@semio
update experiment sturcture
9c109df
@semio
minor changes
0276aba
@semio
have to set a callback to make Palm work
eb503d9
@semio
add replicate key
a757aaf
@semio
remove the hacking
0f1cff8 
I thought it's just my network issue
@semio
add rounds in model variation
06216ff 
and update data
@semio
update dependencies
5353d1c 
use my fork of yival now
@semio
Don't evaluate formulas
8757ee1 
I think we won't use formulas in these sheets. And if we enable
evaluate formulas, some text will be evaluated to incorrect values.
Such as "-6C", will result in "#ERROR"
@semio
Update github workflow
e2acc4c
@semio
update model compare function
5fc96c4 
- use redis cache
- use litellm directly instead of llm_completion from yival.
  llm_complete is just same as litellm.completion with some custom
  name mappings which we don't use.
@semio
strip question text
c999fb5
@semio
update generate result script
8c98ad5
@semio
Latest results
9b7b8e9
@semio
update example
12d935f
@semio
update dependencies
95e6447
@semio
add todo
d9c18a5
@semio
set model name for evaluator
365fee6
@semio
latest experiment yaml
9d2aaf6
@semio
questions
99fda7f
@semio
update README
4c77f24
@semio semio changed the title Yival Use Yival to run Experiment Nov 11, 2023
@motin motin merged commit 7e789cc into main Jan 25, 2024
2 checks passed
@motin motin deleted the yival branch January 25, 2024 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@motin motin motin approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@semio @motin