From ae1107a6b4999951e10ba7b62288deb97b8db464 Mon Sep 17 00:00:00 2001 From: ChengZi Date: Tue, 5 Dec 2023 15:10:35 +0800 Subject: [PATCH] fix topk, add comparision table Signed-off-by: ChengZi --- evaluation/README.md | 14 +++++++++++++- evaluation/evaluate_fiqa_customized_RAG.ipynb | 2 +- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/evaluation/README.md b/evaluation/README.md index b9db8eca6..422b45f8d 100644 --- a/evaluation/README.md +++ b/evaluation/README.md @@ -4,4 +4,16 @@ Evaluate the performance of RAG pipelines based on [ragas](https://github.com/ex - [evaluate_fiqa_openai.ipynb](evaluate_fiqa_openai.ipynb) Use Ragas to evaluate the OpenAI Assistant -- [evaluate_fiqa_customized_RAG.ipynb](evaluate_fiqa_customized_RAG.ipynb) Use Ragas to evaluate the customized RAG pipeline based on milvus \ No newline at end of file +- [evaluate_fiqa_customized_RAG.ipynb](evaluate_fiqa_customized_RAG.ipynb) Use Ragas to evaluate the customized RAG pipeline based on milvus + +The following outlines a comparison between two experimental setups: + +| | OpenAI assistant | Customized RAG pipeline | +| --- | --- | --- | +| LLM model | gpt-4-1106-preview | gpt-4-1106-preview | +| Vector DB | Not Disclosed | milvus | +| Embedding model | Not Disclosed | BAAI/bge-base-en | +| Chunk size | Not Disclosed | 1000 | +| Chunk overlap | Not Disclosed | 40 | +| topk | Not Disclosed | 5 | +| Use Agent | Yes | Yes | diff --git a/evaluation/evaluate_fiqa_customized_RAG.ipynb b/evaluation/evaluate_fiqa_customized_RAG.ipynb index cbf1e0463..82239912c 100644 --- a/evaluation/evaluate_fiqa_customized_RAG.ipynb +++ b/evaluation/evaluate_fiqa_customized_RAG.ipynb @@ -302,7 +302,7 @@ "outputs": [], "source": [ "def search_milvus(question, top_k=5):\n", - " contexts = vector_db.similarity_search(question)\n", + " contexts = vector_db.similarity_search(question, k=top_k)\n", " return contexts[:top_k]" ], "metadata": {