Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standardize qa dataset naming and structure #876

Merged
merged 20 commits into from
Nov 9, 2023
Merged

Standardize qa dataset naming and structure #876

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8aa46b5
standardize dataset
Prikshit7766 Nov 7, 2023
d80b4ef
datasource.py: updated _load_dataset method to load dataset
Prikshit7766 Nov 7, 2023
4f0b8c2
helpers.py :rename default_user_prompt keys
Prikshit7766 Nov 7, 2023
8fac1a1
fix: ASDiv dataset
Prikshit7766 Nov 7, 2023
9b1167d
updated task.py
Prikshit7766 Nov 7, 2023
cafcede
updated dataset-notebooks
Prikshit7766 Nov 7, 2023
8bb3ca6
rename the toxicity dataset
Prikshit7766 Nov 7, 2023
d5c7f28
updated helpers.py
Prikshit7766 Nov 7, 2023
5d2f8ea
langtest.py: updated the dataset checks
Prikshit7766 Nov 7, 2023
9a5bc77
updated datasource.py
Prikshit7766 Nov 7, 2023
6f2f92b
updated llm notebooks
Prikshit7766 Nov 7, 2023
b811165
updated task-specific-notebooks
Prikshit7766 Nov 7, 2023
999820b
notebook: updated Evaluation_Metrics
Prikshit7766 Nov 7, 2023
c131817
updated langtest.py
Prikshit7766 Nov 7, 2023
88f0645
updated datasource.py
Prikshit7766 Nov 7, 2023
c1071a0
updated test_translation.py
Prikshit7766 Nov 7, 2023
a723e84
updated errors.py
Prikshit7766 Nov 8, 2023
4798866
updated datasource.py
Prikshit7766 Nov 8, 2023
1302260
fix lint
Prikshit7766 Nov 8, 2023
5b5635b
updated setup.py
Prikshit7766 Nov 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 33 additions & 62 deletions demo/tutorials/llm_notebooks/AI21_QA_Summarization_Testing_Notebook.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,60 +162,30 @@
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "p_5nO14bvTzt",
"outputId": "cee6c5f4-6f32-4f72-e9db-440a410b59c7"
},
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"harness = Harness(task=\"question-answering\", model={\"model\": \"j2-jumbo-instruct\", \"hub\":\"ai21\"}, data={\"data_source\": 'BoolQ-test-tiny'})"
"harness = Harness(\n",
" task=\"question-answering\", \n",
" model={\"model\": \"j2-jumbo-instruct\", \"hub\":\"ai21\"}, \n",
" data={\"data_source\" :\"BBQ\",\n",
" \"split\":\"test-tiny\"}\n",
" )"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"id": "jWPAw9q0PwD1"
},
"metadata": {},
"source": [
"We have specified task as QA, hub as AI21 and model as `j2-jumbo-instruct`.\n",
"\n",
"For dataset we used `BoolQ-test-tiny` which includes 50 lines from BoolQ-test. Other available datasets are:\n",
"\n",
"#### BoolQ\n",
"* `BoolQ-test-tiny`\n",
"* `BoolQ-test`\n",
"* `BoolQ-combined`\n",
"#### NQ-open\n",
"* `NQ-open-test`\n",
"* `NQ-open-combined`\n",
"* `NQ-open-test-tiny`\n",
"#### TruthfulQA\n",
"* `TruthfulQA-combined`\n",
"* `TruthfulQA-test`\n",
"* `TruthfulQA-tiny`\n",
"#### MMLU\n",
"* `MMLU-test`\n",
"* `MMLU-test-tiny`\n",
"#### OpenBookQA\n",
"* `OpenBookQA-test`\n",
"* `OpenBookQA-test-tiny`\n",
"#### QUAC\n",
"* `Quac-test`\n",
"* `Quac-test-tiny`\n",
"#### NarrativeQA\n",
"* `NarrativeQA-test`\n",
"* `NarrativeQA-test-tiny`\n",
"#### HellaSwag\n",
"* `HellaSwag-test`\n",
"* `HellaSwag-test-tiny`\n",
"#### BBQ\n",
"* `BBQ-test`\n",
"* `BBQ-test-tiny`"
"We have specified task as QA, hub as AI21 and model as `j2-jumbo-instruct`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For dataset we used `BoolQ` dataset and `test-tiny` split which includes 50 samples. Other available datasets are: [Benchmark Datasets](https://langtest.org/docs/pages/docs/data#question-answering)"
]
},
{
Expand Down Expand Up @@ -1135,17 +1105,16 @@
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "oDh3Zaa9EDfZ",
"outputId": "10443ac6-8c92-4e86-ef4e-7050962c4255"
},
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"harness = Harness(task=\"question-answering\", model={\"model\": \"j2-jumbo-instruct\", \"hub\": \"ai21\"}, data={\"data_source\": 'NQ-open-test-tiny'})"
"harness = Harness(\n",
" task=\"question-answering\", \n",
" model={\"model\": \"j2-jumbo-instruct\", \"hub\": \"ai21\"}, \n",
" data={\"data_source\" :\"NQ-open\",\n",
" \"split\":\"test-tiny\"}\n",
" )"
]
},
{
Expand Down Expand Up @@ -1814,11 +1783,16 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"harness = Harness(task=\"summarization\", model={\"model\": \"j2-jumbo-instruct\", \"hub\": \"ai21\"}, data={\"data_source\": 'XSum-test-tiny'})"
"harness = Harness(\n",
" task=\"summarization\", \n",
" model={\"model\": \"j2-jumbo-instruct\", \"hub\": \"ai21\"},\n",
" data={\"data_source\" :\"XSum\",\n",
" \"split\":\"test-tiny\"}\n",
" )"
]
},
{
Expand All @@ -1829,10 +1803,7 @@
"We have specified task as summarization, hub as AI21 and model as `j2-jumbo-instruct`.\n",
"\n",
"\n",
"For dataset we used XSum-test-tiny which includes 50 lines from XSum-test. Available datasets for summarization are:\n",
"\n",
"* `XSum-test`\n",
"* `XSum-test-tiny`"
"For dataset we used `XSum` dataset and `test-tiny` split which includes 50 samples. Other available datasets are: [Benchmark Datasets](https://langtest.org/docs/pages/docs/data#summarization)"
]
},
{
Expand Down
82 changes: 25 additions & 57 deletions demo/tutorials/llm_notebooks/Azure_OpenAI_QA_Summarization_Testing_Notebook.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,17 +162,16 @@
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "p_5nO14bvTzt",
"outputId": "cee6c5f4-6f32-4f72-e9db-440a410b59c7"
},
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"harness = Harness(task=\"question-answering\", model={\"model\": \"text-davinci-003\", \"hub\":\"azure-openai\"} data={\"data_source\": 'BoolQ-test-tiny'})"
"harness = Harness(\n",
" task=\"question-answering\", \n",
" model={\"model\": \"text-davinci-003\",\"hub\":\"azure-openai\"}, \n",
" data={\"data_source\" :\"BoolQ\",\n",
" \"split\":\"test-tiny\"}\n",
" )"
]
},
{
Expand All @@ -184,38 +183,7 @@
"source": [
"We have specified task as QA, hub as OpenAI and model as text-davinci-003, text-davinci-002 whatever model available from azure openai services.\n",
"\n",
"For dataset we used `BoolQ-test-tiny` which includes 50 lines from BoolQ-test. Other available datasets are:\n",
"\n",
"#### BoolQ\n",
"* `BoolQ-test-tiny`\n",
"* `BoolQ-test`\n",
"* `BoolQ-combined`\n",
"#### NQ-open\n",
"* `NQ-open-test`\n",
"* `NQ-open-combined`\n",
"* `NQ-open-test-tiny`\n",
"#### TruthfulQA\n",
"* `TruthfulQA-combined`\n",
"* `TruthfulQA-test`\n",
"* `TruthfulQA-tiny`\n",
"#### MMLU\n",
"* `MMLU-test`\n",
"* `MMLU-test-tiny`\n",
"#### OpenBookQA\n",
"* `OpenBookQA-test`\n",
"* `OpenBookQA-test-tiny`\n",
"#### QUAC\n",
"* `Quac-test`\n",
"* `Quac-test-tiny`\n",
"#### NarrativeQA\n",
"* `NarrativeQA-test`\n",
"* `NarrativeQA-test-tiny`\n",
"#### HellaSwag\n",
"* `HellaSwag-test`\n",
"* `HellaSwag-test-tiny`\n",
"#### BBQ\n",
"* `BBQ-test`\n",
"* `BBQ-test-tiny`"
"For dataset we used `BoolQ` dataset and `test-tiny` split which includes 50 samples. Other available datasets are: [Benchmark Datasets](https://langtest.org/docs/pages/docs/data#question-answering)"
]
},
{
Expand Down Expand Up @@ -1120,18 +1088,16 @@
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "oDh3Zaa9EDfZ",
"outputId": "10443ac6-8c92-4e86-ef4e-7050962c4255"
},
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"harness = Harness(task=\"question-answering\", model={\"model\": \"text-davinci-003\",\"hub\":\"azure-openai\"} data={\"data_source\": \n",
"'NQ-open-test-tiny'})"
"harness = Harness(\n",
" task=\"question-answering\", \n",
" model={\"model\": \"text-davinci-003\",\"hub\":\"azure-openai\"}, \n",
" data={\"data_source\" :\"NQ-open\",\n",
" \"split\":\"test-tiny\"}\n",
" )"
]
},
{
Expand Down Expand Up @@ -1802,12 +1768,16 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"harness = Harness(task='summarization',model={\"model\": 'text-davinci-003', \"hub\": \"azure-openai\"}, data={\"data_source\": \n",
"'XSum-test-tiny'})"
"harness = Harness(\n",
" task=\"summarization\", \n",
" model={\"model\": \"text-davinci-003\",\"hub\":\"azure-openai\"}, \n",
" data={\"data_source\" :\"XSum\",\n",
" \"split\":\"test-tiny\"}\n",
" )"
]
},
{
Expand All @@ -1817,10 +1787,8 @@
"source": [
"We have specified task as Summarization, hub as Azure-OpenAI and model as text-davinci-003, text-davinci-002 whatever model available from azure openai services.\n",
"\n",
"For dataset we used XSum-test-tiny which includes 50 lines from XSum-test. Available datasets for summarization are:\n",
"\n",
"* `XSum-test`\n",
"* `XSum-test-tiny`"
"For dataset we used `XSum` dataset and `test-tiny` split which includes 50 samples. Other available datasets are: [Benchmark Datasets](https://langtest.org/docs/pages/docs/data#summarization)"
]
},
{
Expand Down
31 changes: 27 additions & 4 deletions demo/tutorials/llm_notebooks/Clinical_Tests.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@
"source": [
"import os\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = <ADD OPEN-AI-KEY>\n"
"os.environ[\"OPENAI_API_KEY\"] = \"<ADD OPEN-AI-KEY>\""
]
},
{
Expand Down Expand Up @@ -127,6 +127,19 @@
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### **Dataset** : **Clinical**\n",
"\n",
"**Data Splits**\n",
"\n",
"- `Medical-files` \n",
"- `Gastroenterology-files`\n",
"- `Oromaxillofacial-files`"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -173,7 +186,9 @@
],
"source": [
"model = {\"model\": \"text-davinci-003\", \"hub\": \"openai\"}\n",
"data = {\"data_source\": \"Medical-files\"}\n",
"\n",
"data = {\"data_source\": \"Clinical\", \"split\":\"Medical-files\"}\n",
"\n",
"harness = Harness(task=\"clinical-tests\", model=model, data=data)"
]
},
Expand Down Expand Up @@ -2619,7 +2634,11 @@
}
],
"source": [
"harness = Harness(task=\"clinical-tests\",model={\"model\": \"text-davinci-003\", \"hub\": \"openai\"},data = {\"data_source\": \"Gastroenterology-files\"})"
"model = {\"model\": \"text-davinci-003\", \"hub\": \"openai\"}\n",
"\n",
"data = {\"data_source\": \"Clinical\", \"split\":\"Gastroenterology-files\"}\n",
"\n",
"harness = Harness(task=\"clinical-tests\", model=model, data=data)"
]
},
{
Expand Down Expand Up @@ -4981,7 +5000,11 @@
}
],
"source": [
"harness = Harness(task=\"clinical-tests\", model={\"model\": \"text-davinci-003\", \"hub\": \"openai\"},data = {\"data_source\": \"Oromaxillofacial-files\"})"
"model = {\"model\": \"text-davinci-003\", \"hub\": \"openai\"}\n",
"\n",
"data = {\"data_source\": \"Clinical\", \"split\":\"Oromaxillofacial-files\"}\n",
"\n",
"harness = Harness(task=\"clinical-tests\", model=model, data=data)"
]
},
{
Expand Down
Loading