RAG_Response_synthesis.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIRfjqGfBoka"
      },
      "source": [
        "# Chapter 6\n",
        "\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/wandb/edu/blob/main/rag-advanced/notebooks/Chapter06.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>\n",
        "\n",
        "<!--- @wandbcode{rag-course-06} -->\n",
        "\n",
        "## Response Synthesis and Prompting\n",
        "\n",
        "Response synthesis is a critical component of RAG systems, responsible for generating coherent and accurate answers based on retrieved information. In this chapter, we'll explore techniques to improve response quality through iterative prompt engineering and model selection.\n",
        "\n",
        "Key concepts we'll cover:\n",
        "1. Baseline prompt evaluation\n",
        "2. Iterative prompt improvement\n",
        "3. Impact of model selection on response quality\n",
        "4. Comparative analysis of different prompting strategies\n",
        "\n",
        "This hands-on experience will deepen your understanding of advanced RAG concepts and prepare you to implement these techniques in your own projects.\n",
        "\n",
        "Let's begin by setting up our environment and importing the necessary libraries."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-oIQzNMJBokb"
      },
      "source": [
        "To begin, execute the following cell to clone the repository and install dependencies:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z7lCCT7mBokb"
      },
      "outputs": [],
      "source": [
        "!git clone https://github.com/wandb/edu.git\n",
        "%cd edu/rag-advanced\n",
        "!pip install -qqq -r requirements.txt\n",
        "%cd notebooks\n",
        "\n",
        "import nltk\n",
        "\n",
        "nltk.download(\"wordnet\")\n",
        "nltk.download(\"punkt_tab\")\n",
        "nltk.download(\"punkt\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1MbHZOVxBokc"
      },
      "source": [
        "With the setup complete, we can now proceed with the chapter content.\n",
        "\n",
        "Initial steps:\n",
        "1. Log in to Weights & Biases (W&B)\n",
        "2. Configure environment variables for API access\n",
        "\n",
        "To obtain your Cohere API key, visit the [Cohere API dashboard](https://dashboard.cohere.com/api-keys)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sg37CvWJBokc"
      },
      "outputs": [],
      "source": [
        "import getpass\n",
        "import os\n",
        "\n",
        "os.environ[\"COHERE_API_KEY\"] = getpass.getpass(\"Please enter your COHERE_API_KEY\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eE_waaf1Bokc"
      },
      "outputs": [],
      "source": [
        "import asyncio\n",
        "\n",
        "import nest_asyncio\n",
        "\n",
        "nest_asyncio.apply()\n",
        "\n",
        "import weave"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u05vYt3fBokd"
      },
      "outputs": [],
      "source": [
        "WANDB_PROJECT = \"rag-course\"\n",
        "\n",
        "weave_client = weave.init(WANDB_PROJECT)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u4qMKWUvBokd"
      },
      "source": [
        "# Data loading\n",
        "We'll start by loading the semantically chunked data from Chapter 3. As a reminder, semantic chunking is an technique that groups related sentences together, preserving context and improving retrieval relevance.\n",
        "\n",
        "This chunked data will serve as the input for the knowledge base for our RAG pipeline, allowing us to compare the effectiveness of our response synthesis techniques against a baseline system.\n",
        "\n",
        "Let's load the data and take a look at the first few chunks:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "n7IIcpV4Bokd"
      },
      "outputs": [],
      "source": [
        "# Reload the data from Chapter 3\n",
        "chunked_data = weave.ref(\n",
        "    \"weave:///rag-course/rag-course/object/chunked_data:Ij9KThmiZQ9ljpCm8rVXTJlCaAbY2qC0zX6UJkBWHQ0\"\n",
        ").get()\n",
        "# uncomment the next line to get the chunked data from weave from your own project instead\n",
        "# chunked_data = weave.ref(\"chunked_data:latest\").get()\n",
        "\n",
        "chunked_data.rows[:2]\n",
        "chunked_data = list(map(dict, chunked_data.rows[:]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vkIP4PEQBokd"
      },
      "source": [
        "Next, let's load the query enhancer, hybrid retriever, response generator and RAG pipeline from the previous chapters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lrvBpYKBBokd"
      },
      "outputs": [],
      "source": [
        "import cohere\n",
        "\n",
        "from scripts.query_enhancer import QueryEnhancer\n",
        "from scripts.rag_pipeline import QueryEnhancedRAGPipeline\n",
        "from scripts.response_generator import QueryEnhanedResponseGenerator\n",
        "from scripts.retriever import HybridRetrieverReranker\n",
        "\n",
        "query_enhancer = QueryEnhancer()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q9ZSCABBBokd"
      },
      "source": [
        "## Prompt iteration\n",
        "\n",
        "Prompt engineering is a crucial skill in developing effective RAG systems. By carefully crafting prompts, we can guide the model to produce more accurate, relevant, and coherent responses. We'll explore several iterations of prompt improvements:\n",
        "\n",
        "1. Baseline prompt\n",
        "2. Adding precise instructions\n",
        "3. Including response format examples\n",
        "4. Incorporating model reasoning\n",
        "\n",
        "For each iteration, we'll evaluate the impact on response quality using our established metrics.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B7VBeLClBokd"
      },
      "outputs": [],
      "source": [
        "eval_dataset = weave.ref(\n",
        "    \"weave:///rag-course/dev/object/Dataset:Qj4IFICc2EbdXu5A5UuhkPiWgxM1GvJMIvXEyv1DYnM\"\n",
        ").get()\n",
        "\n",
        "print(eval_dataset.rows[:2])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JQf3Dv-FBoke"
      },
      "outputs": [],
      "source": [
        "from scripts.response_metrics import ALL_METRICS as RESPONSE_METRICS\n",
        "\n",
        "response_evaluations = weave.Evaluation(\n",
        "    name=\"Response_Evaluation\",\n",
        "    dataset=eval_dataset,\n",
        "    scorers=RESPONSE_METRICS,\n",
        "    preprocess_model_input=lambda x: {\"query\": x[\"question\"]},\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0ZUr1iUFBoke"
      },
      "outputs": [],
      "source": [
        "hybrid_retriever = HybridRetrieverReranker()\n",
        "hybrid_retriever.index_data(chunked_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2S3XpK6Boke"
      },
      "source": [
        "### Baseline Prompt Evaluation\n",
        "\n",
        "We are now ready to evaluate the performance of the RAG pipeline while iterating over different prompt improvemtns.\n",
        "For comparison, let's begin our evaluation of the baseline RAG pipeline.\n",
        "\n",
        "This simple prompt serves as our starting point. It provides basic instructions for the model to answer questions about W&B using only the provided context. However, it lacks specific guidance on response structure, tone, or level of detail. As we iterate, we'll see how more detailed prompts can improve response quality and relevance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evWihebKBoke"
      },
      "outputs": [],
      "source": [
        "INITIAL_PROMPT = open(\"prompts/initial_system.txt\").read()\n",
        "\n",
        "print(INITIAL_PROMPT)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yAsawibeBoke"
      },
      "outputs": [],
      "source": [
        "baseline_response_generator = QueryEnhanedResponseGenerator(\n",
        "    model=\"command-r\", prompt=INITIAL_PROMPT, client=cohere.AsyncClientV2()\n",
        ")\n",
        "\n",
        "\n",
        "class BaselineRAGPipeline(QueryEnhancedRAGPipeline):\n",
        "    pass\n",
        "\n",
        "\n",
        "baseline_rag_pipeline = BaselineRAGPipeline(\n",
        "    query_enhancer=query_enhancer,\n",
        "    retriever=hybrid_retriever,\n",
        "    response_generator=baseline_response_generator,\n",
        ")\n",
        "\n",
        "\n",
        "baseline_response_scores = asyncio.run(\n",
        "    response_evaluations.evaluate(baseline_rag_pipeline)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Iw-t8WMBoke"
      },
      "source": [
        "**Tip**: When designing your initial prompt, aim for clarity and simplicity. However, be prepared to iterate and refine based on the results.\n",
        "\n",
        "**Best Practice**: Always establish a baseline performance to measure improvements against."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9VkSckaMBoke"
      },
      "source": [
        "### Improved Prompt V1: Adding Precise Instructions\n",
        "\n",
        "In our first iteration, let's enhance the prompt by providing more detailed instructions to the AI assistant. We'll focus on:\n",
        "1. Defining a clear role for the AI as a W&B specialist\n",
        "2. Incorporating dynamic elements like language and intent recognition\n",
        "3. Outlining a structured approach to formulating responses\n",
        "4. Specifying formatting requirements, including markdown usage\n",
        "5. Addressing edge cases, such as insufficient information or off-topic queries\n",
        "\n",
        "By adding these elements, we aim to guide the model towards generating more coherent, relevant, and well-structured responses. This approach should help maintain accuracy while ensuring proper citation of sources. As we progress, we'll evaluate how these changes impact the quality of the generated answers."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YCDXQNCsBoke"
      },
      "outputs": [],
      "source": [
        "# Can we improve the prompt with mode precise instructions ?\n",
        "\n",
        "IMPROVED_PROMPT_V1 = open(\"prompts/improved_prompt_v1.txt\").read()\n",
        "\n",
        "print(IMPROVED_PROMPT_V1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8wD0TSFBoke"
      },
      "source": [
        "**Tip**: Adding specific instructions and defining the AI's role can significantly improve response quality.\n",
        "\n",
        "**Best Practice**: Include guidelines for handling edge cases, such as insufficient information or off-topic queries, in your prompt design."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0eehEJBkBoke"
      },
      "outputs": [],
      "source": [
        "improved_v1_response_generator = QueryEnhanedResponseGenerator(\n",
        "    model=\"command-r\", prompt=IMPROVED_PROMPT_V1, client=cohere.AsyncClientV2()\n",
        ")\n",
        "\n",
        "\n",
        "class ImprovedV1RAGPipeline(QueryEnhancedRAGPipeline):\n",
        "    pass\n",
        "\n",
        "\n",
        "improved_v1_rag_pipeline = ImprovedV1RAGPipeline(\n",
        "    query_enhancer=query_enhancer,\n",
        "    retriever=hybrid_retriever,\n",
        "    response_generator=improved_v1_response_generator,\n",
        ")\n",
        "\n",
        "\n",
        "improved_v1_response_scores = asyncio.run(\n",
        "    response_evaluations.evaluate(improved_v1_rag_pipeline)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2HKS3_n5Boke"
      },
      "source": [
        "### Improved Prompt V2: Including Response Format Examples\n",
        "\n",
        "In this iteration, we further refine our prompt by incorporating a concrete example of a well-structured response. This addition serves several purposes:\n",
        "\n",
        "1. It demonstrates the desired formatting and structure, including proper use of markdown and code blocks.\n",
        "2. It shows how to integrate citations and reference relevant documentation.\n",
        "3. It illustrates the appropriate level of detail and explanation expected in responses.\n",
        "4. It provides a model for balancing technical accuracy with user-friendly explanations.\n",
        "\n",
        "By including this example, we aim to guide the model towards producing more consistent, well-formatted, and informative responses. This approach should help improve the overall quality and usefulness of the generated answers, making them more accessible to users with varying levels of technical expertise."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WSQn6bCUBoke"
      },
      "outputs": [],
      "source": [
        "# Can we improve the prompt with a example of the response format ?\n",
        "\n",
        "IMPROVED_PROMPT_V2 = open(\"prompts/improved_prompt_v2.txt\").read()\n",
        "print(IMPROVED_PROMPT_V2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UyZ8hZGKBokf"
      },
      "source": [
        "**Tip**: Providing concrete examples in your prompt can help guide the model towards the desired output format and structure.\n",
        "\n",
        "**Best Practice**: When including examples, ensure they demonstrate key aspects like proper citation, use of markdown, and appropriate level of detail."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZFMV-Hz4Bokf"
      },
      "outputs": [],
      "source": [
        "improved_v2_response_generator = QueryEnhanedResponseGenerator(\n",
        "    model=\"command-r\", prompt=IMPROVED_PROMPT_V2, client=cohere.AsyncClientV2()\n",
        ")\n",
        "\n",
        "\n",
        "class ImprovedV2RAGPipeline(QueryEnhancedRAGPipeline):\n",
        "    pass\n",
        "\n",
        "\n",
        "improved_v2_rag_pipeline = ImprovedV2RAGPipeline(\n",
        "    query_enhancer=query_enhancer,\n",
        "    retriever=hybrid_retriever,\n",
        "    response_generator=improved_v2_response_generator,\n",
        ")\n",
        "improved_v2_response_scores = asyncio.run(\n",
        "    response_evaluations.evaluate(improved_v2_rag_pipeline)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C-4kS8e3Bokf"
      },
      "source": [
        "### Improved Prompt V3: Incorporating Model Reasoning\n",
        "\n",
        "In this iteration, we focus on enhancing the model's reasoning process and transparency:\n",
        "\n",
        "1. We introduce a structured approach to breaking down and addressing complex queries.\n",
        "2. The prompt now explicitly requests the model to explain its thought process for each step.\n",
        "3. We emphasize the importance of providing detailed explanations, including the relevance and functionality of code elements.\n",
        "4. The example response demonstrates a clear, step-by-step structure with explanations at each stage.\n",
        "5. We've added instructions for handling edge cases more comprehensively.\n",
        "\n",
        "By encouraging the model to \"show its work,\" we aim to produce more transparent, logical, and comprehensive responses. This approach can help users better understand the reasoning behind the answers, potentially leading to improved learning outcomes and increased trust in the AI assistant's capabilities. Additionally, this structured reasoning process may help the model catch and correct its own errors, leading to more accurate and reliable responses."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Jvns2afBokf"
      },
      "outputs": [],
      "source": [
        "# Can we further improve the prompt to inlcude model reasoning ?\n",
        "\n",
        "\n",
        "IMPROVED_PROMPT_V3 = open(\"prompts/improved_prompt_v3.txt\").read()\n",
        "\n",
        "print(IMPROVED_PROMPT_V3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMox2qXJBokf"
      },
      "source": [
        "**Tip**: Encouraging the model to explain its reasoning process can lead to more transparent and logical responses.\n",
        "\n",
        "**Best Practice**: Structure your prompt to guide the model through a step-by-step approach for complex queries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KjpMApMaBokf"
      },
      "outputs": [],
      "source": [
        "improved_v3_response_generator = QueryEnhanedResponseGenerator(\n",
        "    model=\"command-r\", prompt=IMPROVED_PROMPT_V3, client=cohere.AsyncClientV2()\n",
        ")\n",
        "\n",
        "\n",
        "class ImprovedV3RAGPipeline(QueryEnhancedRAGPipeline):\n",
        "    pass\n",
        "\n",
        "\n",
        "improved_v3_rag_pipeline = ImprovedV3RAGPipeline(\n",
        "    query_enhancer=query_enhancer,\n",
        "    retriever=hybrid_retriever,\n",
        "    response_generator=improved_v3_response_generator,\n",
        ")\n",
        "\n",
        "improved_v3_response_scores = asyncio.run(\n",
        "    response_evaluations.evaluate(improved_v3_rag_pipeline)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MFznYhYuBokf"
      },
      "source": [
        "### Model Improvement: Leveraging Advanced Language Models\n",
        "\n",
        "After iterating on our prompt engineering, we now take the next step by utilizing a more advanced language model (command-r-plus). This change demonstrates an important principle in RAG system development: the synergy between prompt design and model capability. By combining our refined prompt with a more sophisticated model, we aim to:\n",
        "\n",
        "1. Improve the overall quality and coherence of generated responses\n",
        "2. Enhance the model's ability to understand and follow complex instructions\n",
        "3. Potentially increase the accuracy and depth of domain-specific knowledge\n",
        "4. Better handle nuanced queries and edge cases\n",
        "\n",
        "This step allows us to explore how model selection interacts with prompt engineering to affect response quality. As we evaluate the results, we'll gain insights into the relative impact of prompt refinement versus model capabilities in our RAG pipeline."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZjKiNau2Bokf"
      },
      "source": [
        "**Tip**: Don't rely solely on prompt engineering; consider the capabilities of different models in your iterative improvement process.\n",
        "\n",
        "**Best Practice**: Balance the trade-off between response quality and latency based on your specific use-case requirements."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0IoU_tjNBokf"
      },
      "outputs": [],
      "source": [
        "# Can we further imporve by using a better model to generate the response ?\n",
        "\n",
        "improved_v4_response_generator = QueryEnhanedResponseGenerator(\n",
        "    model=\"command-r-plus\", prompt=IMPROVED_PROMPT_V3, client=cohere.AsyncClientV2()\n",
        ")\n",
        "\n",
        "\n",
        "class ImprovedV4RAGPipeline(QueryEnhancedRAGPipeline):\n",
        "    pass\n",
        "\n",
        "\n",
        "improved_v4_rag_pipeline = ImprovedV4RAGPipeline(\n",
        "    query_enhancer=query_enhancer,\n",
        "    retriever=hybrid_retriever,\n",
        "    response_generator=improved_v4_response_generator,\n",
        ")\n",
        "\n",
        "improved_v4_response_scores = asyncio.run(\n",
        "    response_evaluations.evaluate(improved_v4_rag_pipeline)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t9RQn2YcBokf"
      },
      "source": [
        "## Comparing Evaluations\n",
        "\n",
        "Comparing the performance of different RAG pipeline iterations is crucial for understanding the impact of our prompt engineering efforts. By comparing metrics across various versions, we can identify trends, improvements, and potential trade-offs. This comparative analysis helps us make informed decisions about which prompting strategies are most effective for our specific use case. It's important to consider both quantitative metrics (like accuracy scores) and qualitative aspects (such as response relevance) when assessing overall performance improvements.\n",
        "\n",
        "**Tip**: Use multiple evaluation metrics to get a comprehensive view of your system's performance.\n",
        "\n",
        "**Best Practice**: Regularly reassess and refine your prompts as you gather more data on user queries and system performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QGj-sX90Bokf"
      },
      "source": [
        "![compare_retriever_responses](https://github.com/wandb/edu/blob/main/rag-advanced/images/06_compare_prompts.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XC1ETFtuBokf"
      },
      "source": [
        "### Comparing RAG Pipeline Iterations\n",
        "\n",
        "Here are a few key insights from the evaluation of the RAG pipeline iterations:\n",
        "\n",
        "1. **Response Quality Improvement**: The ImprovedV3 pipelines significantly outperformed earlier versions in LLM Response Scorer metrics (0.95 vs 0.75 for baseline), indicating substantial improvements in response quality and correctness.\n",
        "\n",
        "2. **Trade-off Between Quality and Latency**: While the later iterations (V3 and V4) produced higher quality responses, they also exhibited increased latency. This highlights a common trade-off in AI systems between performance and computational efficiency.\n",
        "\n",
        "3. **Incremental Gains**: Each iteration showed improvements in various metrics, demonstrating the value of iterative refinement in prompt engineering and model selection.\n",
        "\n",
        "4. **Metric Variability**: Some metrics (e.g., Levenshtein distance) showed unexpected increases in later iterations, reminding us that different evaluation metrics can capture different aspects of performance.\n",
        "\n",
        "### Learnings\n",
        "\n",
        "1. Prompt engineering can significantly impact response quality without changing the underlying model.\n",
        "2. Combining refined prompts with more advanced models (as in V4) can lead to synergistic improvements.\n",
        "3. The choice of evaluation metrics is crucial; a holistic view using multiple metrics provides a more comprehensive understanding of system performance.\n",
        "4. In real-world applications, the balance between response quality and latency must be carefully considered based on specific use-case requirements.\n",
        "\n",
        "This evaluation underscores the complexity of optimizing RAG systems and the importance of comprehensive, multi-faceted assessment approaches in AI development.\n",
        "\n",
        "\n",
        "**Overall Best Practice**: \"Iterative improvement is key in RAG system development. Continuously analyze results, gather feedback, and refine both prompts and model selection.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEMHzJBzBokg"
      },
      "source": [
        "## Key Takeaways\n",
        "\n",
        "1. Iterative Prompt Engineering: Systematic refinement of prompts can significantly enhance response quality without changing the underlying model.\n",
        "\n",
        "2. Structured Instructions: Clear, detailed prompts with specific roles, formatting guidelines, and edge case handling improve response coherence and relevance.\n",
        "\n",
        "3. Example Integration: Including well-crafted examples in prompts helps guide the model towards desired output structure and content quality.\n",
        "\n",
        "4. Reasoning Transparency: Prompting the model to explain its thought process leads to more logical, comprehensive, and trustworthy responses.\n",
        "\n",
        "5. Model-Prompt Synergy: Combining refined prompts with more advanced language models can yield synergistic improvements in response quality.\n",
        "\n",
        "6. Performance Trade-offs: Higher quality responses often come at the cost of increased latency. Balance these factors based on specific use-case requirements.\n",
        "\n",
        "7. Multifaceted Evaluation: Use a combination of metrics to comprehensively assess improvements, as different aspects of performance may not all improve uniformly.\n",
        "\n",
        "8. Continuous Optimization: RAG system development is an ongoing process. Regularly reassess and refine prompts based on performance data and user feedback.\n",
        "\n",
        "9. Scalability and Efficiency: As prompt complexity increases, consider the impact on system efficiency and scalability in real-world applications."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "rag-edu",
      "language": "python",
      "name": "python3"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}