diff --git a/tutorials/01_Basic_QA_Pipeline.ipynb b/tutorials/01_Basic_QA_Pipeline.ipynb index 7000b94..7162d35 100644 --- a/tutorials/01_Basic_QA_Pipeline.ipynb +++ b/tutorials/01_Basic_QA_Pipeline.ipynb @@ -7,7 +7,7 @@ "source": [ "# Tutorial: Build Your First Question Answering System\n", "\n", - "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) and [Build an Extractive QA Pipeline](https://haystack.deepset.ai/tutorials/34_extractive_qa_pipeline). \n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) and [Build an Extractive QA Pipeline](https://haystack.deepset.ai/tutorials/34_extractive_qa_pipeline). \n", ">\n", "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", diff --git a/tutorials/02_Finetune_a_model_on_your_data.ipynb b/tutorials/02_Finetune_a_model_on_your_data.ipynb index 3620e16..07d2bec 100644 --- a/tutorials/02_Finetune_a_model_on_your_data.ipynb +++ b/tutorials/02_Finetune_a_model_on_your_data.ipynb @@ -7,6 +7,10 @@ "source": [ "# Tutorial: Fine-Tuning a Model on Your Own Data\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook)\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 15 minutes\n", "- **Nodes Used**: `FARMReader`\n", diff --git a/tutorials/03_Scalable_QA_System.ipynb b/tutorials/03_Scalable_QA_System.ipynb index 810889f..ecd098d 100644 --- a/tutorials/03_Scalable_QA_System.ipynb +++ b/tutorials/03_Scalable_QA_System.ipynb @@ -7,6 +7,10 @@ "source": [ "# Tutorial: Build a Scalable Question Answering System\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook)\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Beginner\n", "- **Time to complete**: 20 minutes\n", "- **Nodes Used**: `ElasticsearchDocumentStore`, `BM25Retriever`, `FARMReader`\n", diff --git a/tutorials/04_FAQ_style_QA.ipynb b/tutorials/04_FAQ_style_QA.ipynb index 4bb2cf3..382463c 100644 --- a/tutorials/04_FAQ_style_QA.ipynb +++ b/tutorials/04_FAQ_style_QA.ipynb @@ -1,736 +1,741 @@ { - "cells": [ - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "id": "jUbPUmtaozIP" - }, - "source": [ - "# Utilizing existing FAQs for Question Answering\n", - "- **Level**: Beginner\n", - "- **Time to complete**: 15 minutes\n", - "- **Nodes Used**: `InMemoryDocumentStore`, `EmbeddingRetriever`\n", - "- **Goal**: Learn how to use the `EmbeddingRetriever` in a `FAQPipeline` to answer incoming questions by matching them to the most similar questions in your existing FAQ.\n", - "\n", - "# Overview\n", - "While *extractive Question Answering* works on pure texts and is therefore more generalizable, there's also a common alternative that utilizes existing FAQ data.\n", - "\n", - "**Pros**:\n", - "\n", - "- Very fast at inference time\n", - "- Utilize existing FAQ data\n", - "- Quite good control over answers\n", - "\n", - "**Cons**:\n", - "\n", - "- Generalizability: We can only answer questions that are similar to existing ones in FAQ\n", - "\n", - "In some use cases, a combination of extractive QA and FAQ-style can also be an interesting option." - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "collapsed": false, - "id": "zBOtphIMozIT" - }, - "source": [ - "\n", - "## Preparing the Colab Environment\n", - "\n", - "- [Enable GPU Runtime](https://docs.haystack.deepset.ai/docs/enabling-gpu-acceleration#enabling-the-gpu-in-colab)\n" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "id": "ENpLjBejozIW" - }, - "source": [ - "## Installing Haystack\n", - "\n", - "To start, let's install the latest release of Haystack with `pip`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "q_y78_4LozIW" - }, - "outputs": [], - "source": [ - "%%bash\n", - "\n", - "pip install --upgrade pip\n", - "pip install farm-haystack[colab,inference]" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Enabling Telemetry \n", - "Knowing you're using this tutorial helps us decide where to invest our efforts to build a better product but you can always opt out by commenting the following line. See [Telemetry](https://docs.haystack.deepset.ai/docs/telemetry) for more details." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "from haystack.telemetry import tutorial_running\n", - "\n", - "tutorial_running(4)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "collapsed": false, - "id": "Wl9Q6E3hozIW", - "pycharm": { - "name": "#%% md\n" - } - }, - "source": [ - "Set the logging level to INFO:" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "id": "Edvocv1ZozIX", - "pycharm": { - "name": "#%%\n" - } - }, - "outputs": [], - "source": [ - "import logging\n", - "\n", - "logging.basicConfig(format=\"%(levelname)s - %(name)s - %(message)s\", level=logging.WARNING)\n", - "logging.getLogger(\"haystack\").setLevel(logging.INFO)" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "id": "noVtM20ZozIX" - }, - "source": [ - "### Create a simple DocumentStore\n", - "The InMemoryDocumentStore is good for quick development and prototyping. For more scalable options, check-out the [docs](https://docs.haystack.deepset.ai/docs/document_store)." - ] + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "id": "jUbPUmtaozIP" + }, + "source": [ + "# Utilizing existing FAQs for Question Answering\n", + "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", + "- **Level**: Beginner\n", + "- **Time to complete**: 15 minutes\n", + "- **Nodes Used**: `InMemoryDocumentStore`, `EmbeddingRetriever`\n", + "- **Goal**: Learn how to use the `EmbeddingRetriever` in a `FAQPipeline` to answer incoming questions by matching them to the most similar questions in your existing FAQ.\n", + "\n", + "# Overview\n", + "While *extractive Question Answering* works on pure texts and is therefore more generalizable, there's also a common alternative that utilizes existing FAQ data.\n", + "\n", + "**Pros**:\n", + "\n", + "- Very fast at inference time\n", + "- Utilize existing FAQ data\n", + "- Quite good control over answers\n", + "\n", + "**Cons**:\n", + "\n", + "- Generalizability: We can only answer questions that are similar to existing ones in FAQ\n", + "\n", + "In some use cases, a combination of extractive QA and FAQ-style can also be an interesting option." + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "id": "zBOtphIMozIT" + }, + "source": [ + "\n", + "## Preparing the Colab Environment\n", + "\n", + "- [Enable GPU Runtime](https://docs.haystack.deepset.ai/docs/enabling-gpu-acceleration#enabling-the-gpu-in-colab)\n" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "id": "ENpLjBejozIW" + }, + "source": [ + "## Installing Haystack\n", + "\n", + "To start, let's install the latest release of Haystack with `pip`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "q_y78_4LozIW" + }, + "outputs": [], + "source": [ + "%%bash\n", + "\n", + "pip install --upgrade pip\n", + "pip install farm-haystack[colab,inference]" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Enabling Telemetry \n", + "Knowing you're using this tutorial helps us decide where to invest our efforts to build a better product but you can always opt out by commenting the following line. See [Telemetry](https://docs.haystack.deepset.ai/docs/telemetry) for more details." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "from haystack.telemetry import tutorial_running\n", + "\n", + "tutorial_running(4)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "id": "Wl9Q6E3hozIW", + "pycharm": { + "name": "#%% md\n" + } + }, + "source": [ + "Set the logging level to INFO:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "Edvocv1ZozIX", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "import logging\n", + "\n", + "logging.basicConfig(format=\"%(levelname)s - %(name)s - %(message)s\", level=logging.WARNING)\n", + "logging.getLogger(\"haystack\").setLevel(logging.INFO)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "id": "noVtM20ZozIX" + }, + "source": [ + "### Create a simple DocumentStore\n", + "The InMemoryDocumentStore is good for quick development and prototyping. For more scalable options, check-out the [docs](https://docs.haystack.deepset.ai/docs/document_store)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zeVfvRLZozIY" + }, + "outputs": [], + "source": [ + "from haystack.document_stores import InMemoryDocumentStore\n", + "\n", + "document_store = InMemoryDocumentStore()" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "id": "zHevRxxaozIa" + }, + "source": [ + "### Create a Retriever using embeddings\n", + "Instead of retrieving via Elasticsearch's plain BM25, we want to use vector similarity of the questions (user question vs. FAQ ones).\n", + "We can use the `EmbeddingRetriever` for this purpose and specify a model that we use for the embeddings." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "oFNXb3kIozIb", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "from haystack.nodes import EmbeddingRetriever\n", + "\n", + "retriever = EmbeddingRetriever(\n", + " document_store=document_store,\n", + " embedding_model=\"sentence-transformers/all-MiniLM-L6-v2\",\n", + " use_gpu=True,\n", + " scale_score=False,\n", + ")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "id": "uLv8ysluozIb" + }, + "source": [ + "### Prepare & Index FAQ data\n", + "We create a pandas dataframe containing some FAQ data (i.e curated pairs of question + answer) and index those in our documentstore.\n", + "Here: We download some question-answer pairs related to COVID-19" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "AHiSltp4ozIb", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "\n", + "from haystack.utils import fetch_archive_from_http\n", + "\n", + "\n", + "# Download\n", + "doc_dir = \"data/tutorial4\"\n", + "s3_url = \"https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/small_faq_covid.csv.zip\"\n", + "fetch_archive_from_http(url=s3_url, output_dir=doc_dir)\n", + "\n", + "# Get dataframe with columns \"question\", \"answer\" and some custom metadata\n", + "df = pd.read_csv(f\"{doc_dir}/small_faq_covid.csv\")\n", + "# Minimal cleaning\n", + "df.fillna(value=\"\", inplace=True)\n", + "df[\"question\"] = df[\"question\"].apply(lambda x: x.strip())\n", + "print(df.head())\n", + "\n", + "# Create embeddings for our questions from the FAQs\n", + "# In contrast to most other search use cases, we don't create the embeddings here from the content of our documents,\n", + "# but rather from the additional text field \"question\" as we want to match \"incoming question\" <-> \"stored question\".\n", + "questions = list(df[\"question\"].values)\n", + "df[\"embedding\"] = retriever.embed_queries(queries=questions).tolist()\n", + "df = df.rename(columns={\"question\": \"content\"})\n", + "\n", + "# Convert Dataframe to list of dicts and index them in our DocumentStore\n", + "docs_to_index = df.to_dict(orient=\"records\")\n", + "document_store.write_documents(docs_to_index)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": { + "collapsed": false, + "id": "MXteNgYRozIb" + }, + "source": [ + "### Ask questions\n", + "Initialize a Pipeline (this time without a reader) and ask questions" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "metadata": { + "id": "F5O7r3poozIb", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [], + "source": [ + "from haystack.pipelines import FAQPipeline\n", + "\n", + "pipe = FAQPipeline(retriever=retriever)" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 709, + "referenced_widgets": [ + "070f7d6a12804647b2c4f5ec98241ced", + "8678507de5e748219ba28bb7970c0e63", + "35855d91133f474092381950bdbfce58", + "0656e34a277141d184aef005e4d39f88", + "612af309a6a94477b56dcea22c7a0940", + "dc9c54def7bf47d39819a97b7ceed839", + "09f4ba018a514f1ca2929ece4d0335e2", + "58f3458cddc747d7b1a1c05f8f0664ed", + "cd2614a0933c48a391966cb572044710", + "52abbb2d8eb043a0924d705a99577303", + "04495cdbd0e04e02a91ae3b026ef4c46" + ] }, + "id": "QX6qbic2ozIc", + "outputId": "af0a8eda-f7f6-4c97-cda7-13566ff888b1", + "pycharm": { + "name": "#%%\n" + } + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "zeVfvRLZozIY" + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "070f7d6a12804647b2c4f5ec98241ced", + "version_major": 2, + "version_minor": 0 }, - "outputs": [], - "source": [ - "from haystack.document_stores import InMemoryDocumentStore\n", - "\n", - "document_store = InMemoryDocumentStore()" + "text/plain": [ + "Batches: 0%| | 0/1 [00:00 \"stored question\".\n", - "questions = list(df[\"question\"].values)\n", - "df[\"embedding\"] = retriever.embed_queries(queries=questions).tolist()\n", - "df = df.rename(columns={\"question\": \"content\"})\n", - "\n", - "# Convert Dataframe to list of dicts and index them in our DocumentStore\n", - "docs_to_index = df.to_dict(orient=\"records\")\n", - "document_store.write_documents(docs_to_index)" - ] + "09f4ba018a514f1ca2929ece4d0335e2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": { - "collapsed": false, - "id": "MXteNgYRozIb" - }, - "source": [ - "### Ask questions\n", - "Initialize a Pipeline (this time without a reader) and ask questions" - ] + "35855d91133f474092381950bdbfce58": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_58f3458cddc747d7b1a1c05f8f0664ed", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_cd2614a0933c48a391966cb572044710", + "value": 1 + } }, - { - "cell_type": "code", - "execution_count": 29, - "metadata": { - "id": "F5O7r3poozIb", - "pycharm": { - "name": "#%%\n" - } - }, - "outputs": [], - "source": [ - "from haystack.pipelines import FAQPipeline\n", - "\n", - "pipe = FAQPipeline(retriever=retriever)" - ] + "52abbb2d8eb043a0924d705a99577303": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } }, - { - "cell_type": "code", - "execution_count": 34, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 709, - "referenced_widgets": [ - "070f7d6a12804647b2c4f5ec98241ced", - "8678507de5e748219ba28bb7970c0e63", - "35855d91133f474092381950bdbfce58", - "0656e34a277141d184aef005e4d39f88", - "612af309a6a94477b56dcea22c7a0940", - "dc9c54def7bf47d39819a97b7ceed839", - "09f4ba018a514f1ca2929ece4d0335e2", - "58f3458cddc747d7b1a1c05f8f0664ed", - "cd2614a0933c48a391966cb572044710", - "52abbb2d8eb043a0924d705a99577303", - "04495cdbd0e04e02a91ae3b026ef4c46" - ] - }, - "id": "QX6qbic2ozIc", - "outputId": "af0a8eda-f7f6-4c97-cda7-13566ff888b1", - "pycharm": { - "name": "#%%\n" - } - }, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "070f7d6a12804647b2c4f5ec98241ced", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Batches: 0%| | 0/1 [00:00 This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Evaluating RAG Pipelines](https://haystack.deepset.ai/tutorials/35_evaluating_rag_pipelines). \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "To be able to make a statement about the quality of results a question-answering pipeline or any other pipeline in haystack produces, it is important to evaluate it. Furthermore, evaluation allows determining which components of the pipeline can be improved.\n", "The results of the evaluation can be saved as CSV files, which contain all the information to calculate additional metrics later on or inspect individual predictions." ] diff --git a/tutorials/06_Better_Retrieval_via_Embedding_Retrieval.ipynb b/tutorials/06_Better_Retrieval_via_Embedding_Retrieval.ipynb index 8263ba6..db82999 100644 --- a/tutorials/06_Better_Retrieval_via_Embedding_Retrieval.ipynb +++ b/tutorials/06_Better_Retrieval_via_Embedding_Retrieval.ipynb @@ -9,7 +9,7 @@ "source": [ "# Better Retrieval via \"Embedding Retrieval\"\n", "\n", - "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) and [Build an Extractive QA Pipeline](https://haystack.deepset.ai/tutorials/34_extractive_qa_pipeline). \n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) and [Build an Extractive QA Pipeline](https://haystack.deepset.ai/tutorials/34_extractive_qa_pipeline). \n", ">\n", "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", diff --git a/tutorials/07_RAG_Generator.ipynb b/tutorials/07_RAG_Generator.ipynb index 2b239d3..b7dffe2 100644 --- a/tutorials/07_RAG_Generator.ipynb +++ b/tutorials/07_RAG_Generator.ipynb @@ -15,7 +15,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "> As of version 1.16, `RAGenerator` has been deprecated in Haystack and completely removed from Haystack as of v1.18. We recommend following the tutorial on [Creating a Generative QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/22_pipeline_with_promptnode) instead. For more details about this deprecation, check out [our announcement](https://github.com/deepset-ai/haystack/discussions/4816) on Github." + "> As of version 1.16 (`farm-haystack`), `RAGenerator` has been deprecated in Haystack and completely removed from Haystack as of v1.18. We recommend using Haystack 2.x (`haystack-ai`) and following the tutorial on [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) instead. \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release)." ] }, { diff --git a/tutorials/08_Preprocessing.ipynb b/tutorials/08_Preprocessing.ipynb index a0be3bc..a24168b 100644 --- a/tutorials/08_Preprocessing.ipynb +++ b/tutorials/08_Preprocessing.ipynb @@ -8,7 +8,7 @@ "source": [ "# Preprocessing\n", "\n", - "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Preprocessing Different File Types](https://haystack.deepset.ai/tutorials/30_file_type_preprocessing_index_pipeline). \n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Preprocessing Different File Types](https://haystack.deepset.ai/tutorials/30_file_type_preprocessing_index_pipeline). \n", ">\n", "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", diff --git a/tutorials/09_DPR_training.ipynb b/tutorials/09_DPR_training.ipynb index 611450a..90ac044 100644 --- a/tutorials/09_DPR_training.ipynb +++ b/tutorials/09_DPR_training.ipynb @@ -12,6 +12,10 @@ "source": [ "# Training Your Own \"Dense Passage Retrieval\" Model\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "Haystack contains all the tools needed to train your own Dense Passage Retrieval model.\n", "This tutorial will guide you through the steps required to create a retriever that is specifically tailored to your domain." ] diff --git a/tutorials/10_Knowledge_Graph.ipynb b/tutorials/10_Knowledge_Graph.ipynb index 5f728a6..198aba3 100644 --- a/tutorials/10_Knowledge_Graph.ipynb +++ b/tutorials/10_Knowledge_Graph.ipynb @@ -11,7 +11,9 @@ "source": [ "# Question Answering on a Knowledge Graph\n", "\n", - "> Starting from version 1.15, `BaseKnowledgeGraph`, `GraphDBKnowledgeGraph`, `InMemoryKnowledgeGraph`, and `Text2SparqlRetriever` are being deprecated and will be removed from Haystack as of version 1.17. For more details about this deprecation, check out [our announcement](https://github.com/deepset-ai/haystack/discussions/4882) on Github. \n", + "> Starting from version 1.15 (`farm-haystack`), `BaseKnowledgeGraph`, `GraphDBKnowledgeGraph`, `InMemoryKnowledgeGraph`, and `Text2SparqlRetriever` are being deprecated and will be removed from Haystack as of version 1.17. For more details about this deprecation, check out [our announcement](https://github.com/deepset-ai/haystack/discussions/4882) on Github. \n", + ">\n", + "> If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook). For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", "Haystack allows storing and querying knowledge graphs with the help of pre-trained models that translate text queries to SPARQL queries.\n", "This tutorial demonstrates how to load an existing knowledge graph into haystack, load a pre-trained retriever, and execute text queries on the knowledge graph.\n", diff --git a/tutorials/11_Pipelines.ipynb b/tutorials/11_Pipelines.ipynb index 86a9f7c..381b906 100644 --- a/tutorials/11_Pipelines.ipynb +++ b/tutorials/11_Pipelines.ipynb @@ -11,7 +11,11 @@ } }, "source": [ - "# Tutorial: How to Use Pipelines\n", + "# How to Use Pipelines\n", + "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", "In this tutorial, you will learn how the `Pipeline` connects the different components in Haystack. Whether you are using a Reader, Summarizer\n", "or Retriever (or 2), the `Pipeline` class will help you build a Directed Acyclic Graph (DAG) that\n", diff --git a/tutorials/12_LFQA.ipynb b/tutorials/12_LFQA.ipynb index c70f96c..787bc9b 100644 --- a/tutorials/12_LFQA.ipynb +++ b/tutorials/12_LFQA.ipynb @@ -15,7 +15,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "> As of version 1.16, `Seq2SeqGenerator` has been deprecated in Haystack and completely removed from Haystack as of v1.18. We recommend following the tutorial on [Creating a Generative QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/22_pipeline_with_promptnode) instead. For more details about this deprecation, check out [our announcement](https://github.com/deepset-ai/haystack/discussions/4816) on Github." + "> As of version 1.16 (`farm-haystack`), `Seq2SeqGenerator` has been deprecated in Haystack and completely removed from Haystack as of v1.18. We recommend using Haystack 2.x (`haystack-ai`) and following the tutorial on [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) instead. \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release)." ] }, { diff --git a/tutorials/13_Question_generation.ipynb b/tutorials/13_Question_generation.ipynb index 1f50b5d..0a78d8d 100644 --- a/tutorials/13_Question_generation.ipynb +++ b/tutorials/13_Question_generation.ipynb @@ -9,6 +9,10 @@ "source": [ "# Question Generation\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "This is a bare bones tutorial showing what is possible with the QuestionGenerator Nodes and Pipelines which automatically\n", "generate questions which the question generation model thinks can be answered by a given document." ] diff --git a/tutorials/14_Query_Classifier.ipynb b/tutorials/14_Query_Classifier.ipynb index a2a4da9..09d8798 100644 --- a/tutorials/14_Query_Classifier.ipynb +++ b/tutorials/14_Query_Classifier.ipynb @@ -9,6 +9,10 @@ "source": [ "# Tutorial: Query Classifier\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 15 minutes\n", "- **Nodes Used**: `TransformersQueryClassifier`, `InMemoryDocumentStore`, `BM25Retriever`, `EmbeddingRetriever`, `FARMReader`\n", diff --git a/tutorials/15_TableQA.ipynb b/tutorials/15_TableQA.ipynb index ed8fea8..77bc51f 100644 --- a/tutorials/15_TableQA.ipynb +++ b/tutorials/15_TableQA.ipynb @@ -9,6 +9,10 @@ "source": [ "# Open-Domain QA on Tables\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "This tutorial shows you how to perform question-answering on tables using the `EmbeddingRetriever` or `BM25Retriever` as retriever node and the `TableReader` as reader node." ] }, diff --git a/tutorials/16_Document_Classifier_at_Index_Time.ipynb b/tutorials/16_Document_Classifier_at_Index_Time.ipynb index 57b23ca..6f132c4 100644 --- a/tutorials/16_Document_Classifier_at_Index_Time.ipynb +++ b/tutorials/16_Document_Classifier_at_Index_Time.ipynb @@ -7,6 +7,10 @@ "source": [ "# Extending your Metadata using DocumentClassifiers at Index Time\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "With DocumentClassifier it's possible to automatically enrich your documents with categories, sentiments, topics or whatever metadata you like. This metadata could be used for efficient filtering or further processing. Say you have some categories your users typically filter on. If the documents are tagged manually with these categories, you could automate this process by training a model. Or you can leverage the full power and flexibility of zero shot classification. All you need to do is pass your categories to the classifier, no labels required. This tutorial shows how to integrate it in your indexing pipeline." ] }, diff --git a/tutorials/17_Audio.ipynb b/tutorials/17_Audio.ipynb index cd15a2f..f460f80 100644 --- a/tutorials/17_Audio.ipynb +++ b/tutorials/17_Audio.ipynb @@ -9,7 +9,9 @@ "source": [ "# Tutorial: Make Your QA Pipelines Talk!\n", "\n", - ">⚠️**Update:** This tutorial is now outdated and we recommend moving to Haystack >= 2.0 and checking out the new tutorials [here](https://haystack.deepset.ai/tutorials). AnswerToSpeech lives in the [text2speech](https://github.com/deepset-ai/haystack-extras/tree/main/nodes/text2speech) package. Main [Haystack](https://github.com/deepset-ai/haystack) repository doesn't include it anymore.\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 15 minutes\n", @@ -319,7 +321,7 @@ }, "gpuClass": "standard", "kernelspec": { - "display_name": "Python 3.9.6 64-bit", + "display_name": "Python 3", "language": "python", "name": "python3" }, @@ -334,11 +336,6 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" - }, - "vscode": { - "interpreter": { - "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" - } } }, "nbformat": 4, diff --git a/tutorials/18_GPL.ipynb b/tutorials/18_GPL.ipynb index 9ca9246..187a757 100644 --- a/tutorials/18_GPL.ipynb +++ b/tutorials/18_GPL.ipynb @@ -12,6 +12,10 @@ "source": [ "# Generative Pseudo Labeling for Domain Adaptation of Dense Retrievals\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "*Note: Adapted to Haystack from Nils Reimers' original [notebook](https://colab.research.google.com/gist/jamescalam/d2c888775c87f9882bb7c379a96adbc8/gpl-domain-adaptation.ipynb#scrollTo=183ff7ab)\n", "\n", "The NLP models we use every day were trained on a corpus of data that reflects the world from the past. In the meantime, we've experienced world-changing events, like the COVID pandemics, and we'd like our models to know about them. Training a model from scratch is tedious work but what if we could just update the models with new data? Generative Pseudo Labeling comes to the rescue.\n", diff --git a/tutorials/19_Text_to_Image_search_pipeline_with_MultiModal_Retriever.ipynb b/tutorials/19_Text_to_Image_search_pipeline_with_MultiModal_Retriever.ipynb index 2438680..52ef270 100644 --- a/tutorials/19_Text_to_Image_search_pipeline_with_MultiModal_Retriever.ipynb +++ b/tutorials/19_Text_to_Image_search_pipeline_with_MultiModal_Retriever.ipynb @@ -9,18 +9,17 @@ "source": [ "# Text-To-Image Search Pipeline with Multimodal Retriever\n", "\n", - "**Level**: Intermediate\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", "\n", + "**Level**: Intermediate\n", "**Time to complete**: 20 minutes\n", - "\n", "**Prerequisites**: This tutorial assumes basic knowledge of Haystack Retrievers and Pipelines. If you want to learn about them, have a look at our tutorials on [Build Your First QA System](https://github.com/deepset-ai/haystack-tutorials/blob/main/tutorials/01_Basic_QA_Pipeline.ipynb) and [Fine-Tuning a Model on Your Own Data](https://github.com/deepset-ai/haystack-tutorials/blob/main/tutorials/02_Finetune_a_model_on_your_data.ipynb).\n", "\n", "Prepare the Colab environment (see links below).\n", - "\n", "**Nodes Used**: InMemoryDocumentStore, MultiModalRetriever\n", - "\n", "**Goal**: After completing this tutorial, you will have built a search system that retrieves images as answers to a text query.\n", - "\n", "**Description**: In this tutorial, you'll download a set of images that you'll then turn into embeddings using a transformers model, OpenAI CLIP. You'll then use the same model to embed the text query. Finally, you'll perform a nearest neighbor search to retrieve the images relevant to the text query.\n", "\n", "Let's build a text-to-image search pipeline using a small animal dataset!" diff --git a/tutorials/20_Using_Haystack_with_REST_API.ipynb b/tutorials/20_Using_Haystack_with_REST_API.ipynb index 55d6629..57a79ed 100644 --- a/tutorials/20_Using_Haystack_with_REST_API.ipynb +++ b/tutorials/20_Using_Haystack_with_REST_API.ipynb @@ -7,6 +7,10 @@ "source": [ "# Tutorial: Using Haystack with REST API\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Advanced\n", "- **Time to complete**: 30 minutes\n", "- **Prerequisites**: Basic understanding of Docker and basic knowledge of Haystack pipelines. \n", diff --git a/tutorials/21_Customizing_PromptNode.ipynb b/tutorials/21_Customizing_PromptNode.ipynb index e031a8f..078bacb 100644 --- a/tutorials/21_Customizing_PromptNode.ipynb +++ b/tutorials/21_Customizing_PromptNode.ipynb @@ -9,14 +9,14 @@ "source": [ "# Tutorial: Customizing PromptNode for NLP Tasks\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 20 minutes\n", "- **Nodes Used**: `PromptNode`, `PromptTemplate`\n", - "- **Goal**: After completing this tutorial, you will have learned the basics of using PromptNode and PromptTemplates and you'll have added titles to articles from The Guardian and categorized them. \n", - "\n", - "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline). \n", - ">\n", - "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release)." + "- **Goal**: After completing this tutorial, you will have learned the basics of using PromptNode and PromptTemplates and you'll have added titles to articles from The Guardian and categorized them. " ] }, { diff --git a/tutorials/22_Pipeline_with_PromptNode.ipynb b/tutorials/22_Pipeline_with_PromptNode.ipynb index fedbf40..d7ea329 100644 --- a/tutorials/22_Pipeline_with_PromptNode.ipynb +++ b/tutorials/22_Pipeline_with_PromptNode.ipynb @@ -9,21 +9,16 @@ "source": [ "# Tutorial: Creating a Generative QA Pipeline with Retrieval-Augmentation\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline) \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 15 minutes\n", "- **Nodes Used**: `InMemoryDocumentStore`, `BM25Retriever`, `PromptNode`, `PromptTemplate`\n", "- **Goal**: After completing this tutorial, you'll have created a generative question answering search system that uses a large language model through PromptNode with PromptTemplate." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Creating Your First QA Pipeline with Retrieval-Augmentation](https://haystack.deepset.ai/tutorials/27_first_rag_pipeline). \n", - ">\n", - "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release)." - ] - }, { "attachments": {}, "cell_type": "markdown", diff --git a/tutorials/23_Answering_Multihop_Questions_with_Agents.ipynb b/tutorials/23_Answering_Multihop_Questions_with_Agents.ipynb index 12f7b5f..86275a9 100644 --- a/tutorials/23_Answering_Multihop_Questions_with_Agents.ipynb +++ b/tutorials/23_Answering_Multihop_Questions_with_Agents.ipynb @@ -9,6 +9,10 @@ "source": [ "# Tutorial: Answering Multihop Questions with Agents\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), and would like to follow the updated version of this tutorial, check out [Cookbook: Newsletter Sending Agent](https://haystack.deepset.ai/cookbook/newsletter-agent) or refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 10 minutes\n", "- **Nodes Used**: `Agent`, `PromptNode`, `InMemoryDocumentStore`, `FARMReader` and `ExtractiveQAPipeline`\n", diff --git a/tutorials/24_Building_Chat_App.ipynb b/tutorials/24_Building_Chat_App.ipynb index ee15a6a..b6399b3 100644 --- a/tutorials/24_Building_Chat_App.ipynb +++ b/tutorials/24_Building_Chat_App.ipynb @@ -8,6 +8,10 @@ "source": [ "# Tutorial: Building a Conversational Chat App\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`) and would like to follow the updated version of this tutorial, check out [Building a Chat Application with Function Calling](https://haystack.deepset.ai/tutorials/40_building_chat_application_with_function_calling).\n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 10 minutes\n", "- **Nodes Used**: `PromptNode`, `ConversationalAgent` and `ConversationSummaryMemory`\n", diff --git a/tutorials/25_Customizing_Agent.ipynb b/tutorials/25_Customizing_Agent.ipynb index cd076d6..08d592e 100644 --- a/tutorials/25_Customizing_Agent.ipynb +++ b/tutorials/25_Customizing_Agent.ipynb @@ -9,6 +9,10 @@ "source": [ "# Tutorial: Customizing Agent to Chat with Your Documents\n", "\n", + "> This tutorial is based on Haystack 1.x (`farm-haystack`). If you're using Haystack 2.x (`haystack-ai`), refer to the [Haystack 2.x tutorials](https://haystack.deepset.ai/tutorials) or [Haystack Cookbook](https://haystack.deepset.ai/cookbook).\n", + ">\n", + "> For more information on Haystack 2.x, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Advanced\n", "- **Time to complete**: 20 minutes\n", "- **Nodes Used**: `BM25Retriever`, `PromptNode`, `Agent`, and `Memory`\n", diff --git a/tutorials/26_Hybrid_Retrieval.ipynb b/tutorials/26_Hybrid_Retrieval.ipynb index 0ef23f7..d56fbfb 100644 --- a/tutorials/26_Hybrid_Retrieval.ipynb +++ b/tutorials/26_Hybrid_Retrieval.ipynb @@ -8,21 +8,16 @@ "source": [ "# Tutorial: Creating a Hybrid Retrieval Pipeline\n", "\n", + "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.x and would like to follow the updated version of this tutorial, check out [Creating a Hybrid Pipeline](https://haystack.deepset.ai/tutorials/33_hybrid_retrieval). \n", + ">\n", + "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release).\n", + "\n", "- **Level**: Intermediate\n", "- **Time to complete**: 15 minutes\n", "- **Nodes Used**: `EmbeddingRetriever`, `BM25Retriever`, `JoinDocuments`, `SentenceTransformersRanker` and `InMemoryDocumentStore`\n", "- **Goal**: After completing this tutorial, you will have learned about creating your first hybrid retrieval and when it's useful." ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "> This tutorial is based on Haystack 1.x. If you're using Haystack 2.0 and would like to follow the updated version of this tutorial, check out [Creating a Hybrid Pipeline](https://haystack.deepset.ai/tutorials/33_hybrid_retrieval). \n", - ">\n", - "> For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release)." - ] - }, { "cell_type": "markdown", "metadata": { diff --git a/tutorials/27_First_RAG_Pipeline.ipynb b/tutorials/27_First_RAG_Pipeline.ipynb index d8a4fc4..cedbcaa 100644 --- a/tutorials/27_First_RAG_Pipeline.ipynb +++ b/tutorials/27_First_RAG_Pipeline.ipynb @@ -13,8 +13,7 @@ "- **Components Used**: [`InMemoryDocumentStore`](https://docs.haystack.deepset.ai/docs/inmemorydocumentstore), [`SentenceTransformersDocumentEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformersdocumentembedder), [`SentenceTransformersTextEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformerstextembedder), [`InMemoryEmbeddingRetriever`](https://docs.haystack.deepset.ai/docs/inmemoryembeddingretriever), [`PromptBuilder`](https://docs.haystack.deepset.ai/docs/promptbuilder), [`OpenAIChatGenerator`](https://docs.haystack.deepset.ai/docs/openaichatgenerator)\n", "- **Prerequisites**: You must have an [OpenAI API Key](https://platform.openai.com/api-keys).\n", "- **Goal**: After completing this tutorial, you'll have learned the new prompt syntax and how to use PromptBuilder and OpenAIChatGenerator to build a generative question-answering pipeline with retrieval-augmentation.\n", - "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)." + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro)." ] }, { @@ -50,7 +49,7 @@ "source": [ "## Installing Haystack\n", "\n", - "Install Haystack 2.0 and other required packages with `pip`:" + "Install Haystack and other required packages with `pip`:" ] }, { @@ -85,7 +84,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": { "id": "A76B4S49O-qa" }, @@ -192,7 +191,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -359,7 +358,9 @@ "from haystack.components.builders import ChatPromptBuilder\n", "from haystack.dataclasses import ChatMessage\n", "\n", - "template = [ChatMessage.from_user(\"\"\"\n", + "template = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", "Given the following information, answer the question.\n", "\n", "Context:\n", @@ -369,7 +370,9 @@ "\n", "Question: {{question}}\n", "Answer:\n", - "\"\"\")]\n", + "\"\"\"\n", + " )\n", + "]\n", "\n", "prompt_builder = ChatPromptBuilder(template=template)" ] @@ -449,7 +452,7 @@ "basic_rag_pipeline.add_component(\"text_embedder\", text_embedder)\n", "basic_rag_pipeline.add_component(\"retriever\", retriever)\n", "basic_rag_pipeline.add_component(\"prompt_builder\", prompt_builder)\n", - "basic_rag_pipeline.add_component(\"llm\", chat_generator)\n" + "basic_rag_pipeline.add_component(\"llm\", chat_generator)" ] }, { diff --git a/tutorials/28_Structured_Output_With_Loop.ipynb b/tutorials/28_Structured_Output_With_Loop.ipynb index 217d1f0..90b6d37 100644 --- a/tutorials/28_Structured_Output_With_Loop.ipynb +++ b/tutorials/28_Structured_Output_With_Loop.ipynb @@ -14,7 +14,7 @@ "- **Components Used**: `PromptBuilder`, `OpenAIChatGenerator`, `OutputValidator` (Custom component)\n", "- **Goal**: After completing this tutorial, you will have built a system that extracts unstructured data, puts it in a JSON schema, and automatically corrects errors in the JSON output from a large language model (LLM) to make sure it follows the specified structure.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)..\n", + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", "\n", "## Overview\n", "This tutorial demonstrates how to use Haystack 2.0's advanced [looping pipelines](https://docs.haystack.deepset.ai/docs/pipelines#loops) with LLMs for more dynamic and flexible data processing. You'll learn how to extract structured data from unstructured data using an LLM, and to validate the generated output against a predefined schema.\n", @@ -282,7 +282,9 @@ "from haystack.components.builders import ChatPromptBuilder\n", "\n", "\n", - "prompt_template = [ChatMessage.from_user(\"\"\"\n", + "prompt_template = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", "Create a JSON object from the information present in this passage: {{passage}}.\n", "Only use information that is present in the passage. Follow this JSON schema, but only return the actual instances without any additional schema definition:\n", "{{schema}}\n", @@ -292,7 +294,9 @@ " However, this doesn't comply with the format requirements from above and triggered this Python exception: {{error_message}}\n", " Correct the output and try again. Just return the corrected output without any extra explanations.\n", "{% endif %}\n", - "\"\"\")]\n", + "\"\"\"\n", + " )\n", + "]\n", "prompt_builder = ChatPromptBuilder(template=prompt_template)" ] }, @@ -520,7 +524,8 @@ "provenance": [] }, "kernelspec": { - "display_name": "Python 3", + "display_name": ".venv", + "language": "python", "name": "python3" }, "language_info": { diff --git a/tutorials/29_Serializing_Pipelines.ipynb b/tutorials/29_Serializing_Pipelines.ipynb index cb59ba0..fdfb544 100644 --- a/tutorials/29_Serializing_Pipelines.ipynb +++ b/tutorials/29_Serializing_Pipelines.ipynb @@ -14,7 +14,7 @@ "- **Prerequisites**: None\n", "- **Goal**: After completing this tutorial, you'll understand how to serialize and deserialize between YAML and Python code.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)." + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro)." ] }, { @@ -52,7 +52,7 @@ "source": [ "## Installing Haystack\n", "\n", - "Install Haystack 2.0 with `pip`:" + "Install Haystack with `pip`:" ] }, { diff --git a/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb b/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb index 8fc936b..81573db 100644 --- a/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb +++ b/tutorials/30_File_Type_Preprocessing_Index_Pipeline.ipynb @@ -12,7 +12,7 @@ "- **Time to complete**: 15 minutes\n", "- **Goal**: After completing this tutorial, you'll have learned how to build an indexing pipeline that will preprocess files based on their file type, using the `FileTypeRouter`.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", "\n", "> 💡 (Optional): After creating the indexing pipeline in this tutorial, there is an optional section that shows you how to create a RAG pipeline on top of the document store you just created. You must have a [Hugging Face API Key](https://huggingface.co/settings/tokens) for this section\n", "\n", @@ -448,7 +448,9 @@ "from haystack.dataclasses import ChatMessage\n", "from haystack.components.generators.chat import HuggingFaceAPIChatGenerator\n", "\n", - "template = [ChatMessage.from_user(\"\"\"\n", + "template = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", "Answer the questions based on the given context.\n", "\n", "Context:\n", @@ -458,14 +460,18 @@ "\n", "Question: {{ question }}\n", "Answer:\n", - "\"\"\")]\n", + "\"\"\"\n", + " )\n", + "]\n", "pipe = Pipeline()\n", "pipe.add_component(\"embedder\", SentenceTransformersTextEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\"))\n", "pipe.add_component(\"retriever\", InMemoryEmbeddingRetriever(document_store=document_store))\n", "pipe.add_component(\"chat_prompt_builder\", ChatPromptBuilder(template=template))\n", "pipe.add_component(\n", " \"llm\",\n", - " HuggingFaceAPIChatGenerator(api_type=\"serverless_inference_api\", api_params={\"model\": \"HuggingFaceH4/zephyr-7b-beta\"}),\n", + " HuggingFaceAPIChatGenerator(\n", + " api_type=\"serverless_inference_api\", api_params={\"model\": \"HuggingFaceH4/zephyr-7b-beta\"}\n", + " ),\n", ")\n", "\n", "pipe.connect(\"embedder.embedding\", \"retriever.query_embedding\")\n", @@ -512,13 +518,7 @@ " \"What ingredients would I need to make vegan keto eggplant lasagna, vegan persimmon flan, and vegan hemp cheese?\"\n", ")\n", "\n", - "pipe.run(\n", - " {\n", - " \"embedder\": {\"text\": question},\n", - " \"chat_prompt_builder\": {\"question\": question},\n", - " \n", - " }\n", - ")" + "pipe.run({\"embedder\": {\"text\": question}, \"chat_prompt_builder\": {\"question\": question}})" ] }, { @@ -555,7 +555,8 @@ "provenance": [] }, "kernelspec": { - "display_name": "Python 3", + "display_name": ".venv", + "language": "python", "name": "python3" }, "language_info": { diff --git a/tutorials/31_Metadata_Filtering.ipynb b/tutorials/31_Metadata_Filtering.ipynb index 4c5d05f..d46f67f 100644 --- a/tutorials/31_Metadata_Filtering.ipynb +++ b/tutorials/31_Metadata_Filtering.ipynb @@ -14,7 +14,7 @@ "- **Prerequisites**: None\n", "- **Goal**: Filter documents in a document store based on given metadata\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)." + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro)." ] }, { @@ -50,7 +50,7 @@ "source": [ "## Installing Haystack\n", "\n", - "Install Haystack 2.0 with `pip`:" + "Install Haystack with `pip`:" ] }, { diff --git a/tutorials/32_Classifying_Documents_and_Queries_by_Language.ipynb b/tutorials/32_Classifying_Documents_and_Queries_by_Language.ipynb index d82c604..632dbb4 100644 --- a/tutorials/32_Classifying_Documents_and_Queries_by_Language.ipynb +++ b/tutorials/32_Classifying_Documents_and_Queries_by_Language.ipynb @@ -14,7 +14,7 @@ "- **Goal**: After completing this tutorial, you'll have learned how to build a Haystack pipeline to classify documents based on the (human) language they were written in.\n", "- Optionally, at the end you'll also incorporate language clasification and query routing into a RAG pipeline, so you can query documents based on the language a question was written in.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro).\n" + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n" ] }, { @@ -199,9 +199,11 @@ "outputs": [], "source": [ "language_classifier = DocumentLanguageClassifier(languages=[\"en\", \"fr\", \"es\"])\n", - "router_rules = {\"en\": {\"field\": \"meta.language\", \"operator\": \"==\", \"value\": \"en\"}, \n", - " \"fr\": {\"field\": \"meta.language\", \"operator\": \"==\", \"value\": \"fr\"}, \n", - " \"es\": {\"field\": \"meta.language\", \"operator\": \"==\", \"value\": \"es\"}}\n", + "router_rules = {\n", + " \"en\": {\"field\": \"meta.language\", \"operator\": \"==\", \"value\": \"en\"},\n", + " \"fr\": {\"field\": \"meta.language\", \"operator\": \"==\", \"value\": \"fr\"},\n", + " \"es\": {\"field\": \"meta.language\", \"operator\": \"==\", \"value\": \"es\"},\n", + "}\n", "router = MetadataRouter(rules=router_rules)" ] }, @@ -425,7 +427,9 @@ "from haystack.dataclasses import ChatMessage\n", "from haystack.components.routers import TextLanguageRouter\n", "\n", - "prompt_template = [ChatMessage.from_user(\"\"\"\n", + "prompt_template = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", "You will be provided with reviews for an accommodation.\n", "Answer the question concisely based solely on the given reviews.\n", "Reviews:\n", @@ -434,7 +438,9 @@ " {% endfor %}\n", "Question: {{ query}}\n", "Answer:\n", - "\"\"\")]" + "\"\"\"\n", + " )\n", + "]" ] }, { diff --git a/tutorials/33_Hybrid_Retrieval.ipynb b/tutorials/33_Hybrid_Retrieval.ipynb index 4813d3d..6c136ab 100644 --- a/tutorials/33_Hybrid_Retrieval.ipynb +++ b/tutorials/33_Hybrid_Retrieval.ipynb @@ -14,7 +14,7 @@ "- **Prerequisites**: None\n", "- **Goal**: After completing this tutorial, you will have learned about creating a hybrid retrieval and when it's useful.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)." + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro)." ] }, { @@ -50,7 +50,7 @@ "source": [ "## Installing Haystack\n", "\n", - "Install Haystack 2.0 and other required packages with `pip`:" + "Install Haystack and other required packages with `pip`:" ] }, { diff --git a/tutorials/34_Extractive_QA_Pipeline.ipynb b/tutorials/34_Extractive_QA_Pipeline.ipynb index 6414265..eaa10c1 100644 --- a/tutorials/34_Extractive_QA_Pipeline.ipynb +++ b/tutorials/34_Extractive_QA_Pipeline.ipynb @@ -13,7 +13,7 @@ "- **Components Used**: [`ExtractiveReader`](https://docs.haystack.deepset.ai/docs/extractivereader), [`InMemoryDocumentStore`](https://docs.haystack.deepset.ai/docs/inmemorydocumentstore), [`InMemoryEmbeddingRetriever`](https://docs.haystack.deepset.ai/docs/inmemoryembeddingretriever), [`DocumentWriter`](https://docs.haystack.deepset.ai/docs/documentwriter), [`SentenceTransformersDocumentEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformersdocumentembedder), [`SentenceTransformersTextEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformerstextembedder)\n", "- **Goal**: After completing this tutorial, you'll have learned how to build a Haystack pipeline that uses an extractive model to display where the answer to your query is.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", "\n" ] }, diff --git a/tutorials/35_Evaluating_RAG_Pipelines.ipynb b/tutorials/35_Evaluating_RAG_Pipelines.ipynb index 5ebb551..630c76d 100644 --- a/tutorials/35_Evaluating_RAG_Pipelines.ipynb +++ b/tutorials/35_Evaluating_RAG_Pipelines.ipynb @@ -14,7 +14,7 @@ "- **Prerequisites**: You must have an API key from an active OpenAI account as this tutorial is using the gpt-4o-mini model by OpenAI: https://platform.openai.com/api-keys\n", "- **Goal**: After completing this tutorial, you'll have learned how to evaluate your RAG pipelines both with model-based, and statistical metrics available in the Haystack evaluation offering. You'll also see which other evaluation frameworks are integrated with Haystack.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)." + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro)." ] }, { @@ -75,12 +75,12 @@ "source": [ "## Installing Haystack\n", "\n", - "Install Haystack 2.0 and [datasets](https://pypi.org/project/datasets/) with `pip`:" + "Install Haystack and [datasets](https://pypi.org/project/datasets/) with `pip`:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -88,153 +88,7 @@ "id": "UQbU8GUfO-qZ", "outputId": "80fe52ea-108b-4bb4-cb1d-fe79373c86f3" }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Defaulting to user installation because normal site-packages is not writeable\n", - "Requirement already satisfied: haystack-ai in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (2.8.0)\n", - "Requirement already satisfied: haystack-experimental in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (0.3.0)\n", - "Requirement already satisfied: jinja2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (3.1.4)\n", - "Requirement already satisfied: lazy-imports in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (0.3.1)\n", - "Requirement already satisfied: more-itertools in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (10.2.0)\n", - "Requirement already satisfied: networkx in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (3.2.1)\n", - "Requirement already satisfied: numpy in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (1.26.4)\n", - "Requirement already satisfied: openai>=1.1.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (1.31.1)\n", - "Requirement already satisfied: pandas in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (2.2.2)\n", - "Requirement already satisfied: posthog in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (3.5.0)\n", - "Requirement already satisfied: python-dateutil in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (2.9.0.post0)\n", - "Requirement already satisfied: pyyaml in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (6.0.1)\n", - "Requirement already satisfied: requests in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (2.32.3)\n", - "Requirement already satisfied: tenacity!=8.4.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (8.3.0)\n", - "Requirement already satisfied: tqdm in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (4.66.4)\n", - "Requirement already satisfied: typing-extensions>=4.7 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from haystack-ai) (4.12.1)\n", - "Requirement already satisfied: anyio<5,>=3.5.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from openai>=1.1.0->haystack-ai) (4.4.0)\n", - "Requirement already satisfied: distro<2,>=1.7.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from openai>=1.1.0->haystack-ai) (1.9.0)\n", - "Requirement already satisfied: httpx<1,>=0.23.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from openai>=1.1.0->haystack-ai) (0.27.0)\n", - "Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from openai>=1.1.0->haystack-ai) (2.7.3)\n", - "Requirement already satisfied: sniffio in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from openai>=1.1.0->haystack-ai) (1.3.1)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from jinja2->haystack-ai) (2.1.5)\n", - "Requirement already satisfied: pytz>=2020.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pandas->haystack-ai) (2024.1)\n", - "Requirement already satisfied: tzdata>=2022.7 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pandas->haystack-ai) (2024.1)\n", - "Requirement already satisfied: six>=1.5 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (from python-dateutil->haystack-ai) (1.15.0)\n", - "Requirement already satisfied: monotonic>=1.5 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from posthog->haystack-ai) (1.6)\n", - "Requirement already satisfied: backoff>=1.10.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from posthog->haystack-ai) (2.2.1)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->haystack-ai) (3.3.2)\n", - "Requirement already satisfied: idna<4,>=2.5 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->haystack-ai) (3.7)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->haystack-ai) (1.26.18)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->haystack-ai) (2024.6.2)\n", - "Requirement already satisfied: exceptiongroup>=1.0.2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from anyio<5,>=3.5.0->openai>=1.1.0->haystack-ai) (1.2.1)\n", - "Requirement already satisfied: httpcore==1.* in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from httpx<1,>=0.23.0->openai>=1.1.0->haystack-ai) (1.0.5)\n", - "Requirement already satisfied: h11<0.15,>=0.13 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai>=1.1.0->haystack-ai) (0.14.0)\n", - "Requirement already satisfied: annotated-types>=0.4.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pydantic<3,>=1.9.0->openai>=1.1.0->haystack-ai) (0.7.0)\n", - "Requirement already satisfied: pydantic-core==2.18.4 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pydantic<3,>=1.9.0->openai>=1.1.0->haystack-ai) (2.18.4)\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49m/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Defaulting to user installation because normal site-packages is not writeable\n", - "Requirement already satisfied: datasets>=2.6.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (3.1.0)\n", - "Requirement already satisfied: filelock in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (3.14.0)\n", - "Requirement already satisfied: numpy>=1.17 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (1.26.4)\n", - "Requirement already satisfied: pyarrow>=15.0.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (18.1.0)\n", - "Requirement already satisfied: dill<0.3.9,>=0.3.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (0.3.8)\n", - "Requirement already satisfied: pandas in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (2.2.2)\n", - "Requirement already satisfied: requests>=2.32.2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (2.32.3)\n", - "Requirement already satisfied: tqdm>=4.66.3 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (4.66.4)\n", - "Requirement already satisfied: xxhash in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (3.5.0)\n", - "Requirement already satisfied: multiprocess<0.70.17 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (0.70.16)\n", - "Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets>=2.6.1) (2024.6.0)\n", - "Requirement already satisfied: aiohttp in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (3.11.10)\n", - "Requirement already satisfied: huggingface-hub>=0.23.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (0.23.3)\n", - "Requirement already satisfied: packaging in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (24.0)\n", - "Requirement already satisfied: pyyaml>=5.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from datasets>=2.6.1) (6.0.1)\n", - "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (2.4.4)\n", - "Requirement already satisfied: aiosignal>=1.1.2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (1.3.1)\n", - "Requirement already satisfied: async-timeout<6.0,>=4.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (5.0.1)\n", - "Requirement already satisfied: attrs>=17.3.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (24.2.0)\n", - "Requirement already satisfied: frozenlist>=1.1.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (1.5.0)\n", - "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (6.1.0)\n", - "Requirement already satisfied: propcache>=0.2.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (0.2.1)\n", - "Requirement already satisfied: yarl<2.0,>=1.17.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from aiohttp->datasets>=2.6.1) (1.18.3)\n", - "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.23.0->datasets>=2.6.1) (4.12.1)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests>=2.32.2->datasets>=2.6.1) (3.3.2)\n", - "Requirement already satisfied: idna<4,>=2.5 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests>=2.32.2->datasets>=2.6.1) (3.7)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests>=2.32.2->datasets>=2.6.1) (1.26.18)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests>=2.32.2->datasets>=2.6.1) (2024.6.2)\n", - "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pandas->datasets>=2.6.1) (2.9.0.post0)\n", - "Requirement already satisfied: pytz>=2020.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pandas->datasets>=2.6.1) (2024.1)\n", - "Requirement already satisfied: tzdata>=2022.7 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from pandas->datasets>=2.6.1) (2024.1)\n", - "Requirement already satisfied: six>=1.5 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (from python-dateutil>=2.8.2->pandas->datasets>=2.6.1) (1.15.0)\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49m/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip\u001b[0m\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Defaulting to user installation because normal site-packages is not writeable\n", - "Requirement already satisfied: sentence-transformers>=3.0.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (3.0.0)\n", - "Requirement already satisfied: transformers<5.0.0,>=4.34.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (4.41.2)\n", - "Requirement already satisfied: tqdm in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (4.66.4)\n", - "Requirement already satisfied: torch>=1.11.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (2.3.1)\n", - "Requirement already satisfied: numpy in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (1.26.4)\n", - "Requirement already satisfied: scikit-learn in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (1.5.0)\n", - "Requirement already satisfied: scipy in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (1.13.1)\n", - "Requirement already satisfied: huggingface-hub>=0.15.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (0.23.3)\n", - "Requirement already satisfied: Pillow in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sentence-transformers>=3.0.0) (10.3.0)\n", - "Requirement already satisfied: filelock in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (3.14.0)\n", - "Requirement already satisfied: fsspec>=2023.5.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (2024.6.0)\n", - "Requirement already satisfied: packaging>=20.9 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (24.0)\n", - "Requirement already satisfied: pyyaml>=5.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (6.0.1)\n", - "Requirement already satisfied: requests in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (2.32.3)\n", - "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (4.12.1)\n", - "Requirement already satisfied: sympy in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from torch>=1.11.0->sentence-transformers>=3.0.0) (1.12.1)\n", - "Requirement already satisfied: networkx in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from torch>=1.11.0->sentence-transformers>=3.0.0) (3.2.1)\n", - "Requirement already satisfied: jinja2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from torch>=1.11.0->sentence-transformers>=3.0.0) (3.1.4)\n", - "Requirement already satisfied: regex!=2019.12.17 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers>=3.0.0) (2024.5.15)\n", - "Requirement already satisfied: tokenizers<0.20,>=0.19 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers>=3.0.0) (0.19.1)\n", - "Requirement already satisfied: safetensors>=0.4.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from transformers<5.0.0,>=4.34.0->sentence-transformers>=3.0.0) (0.4.3)\n", - "Requirement already satisfied: joblib>=1.2.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from scikit-learn->sentence-transformers>=3.0.0) (1.4.2)\n", - "Requirement already satisfied: threadpoolctl>=3.1.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from scikit-learn->sentence-transformers>=3.0.0) (3.5.0)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from jinja2->torch>=1.11.0->sentence-transformers>=3.0.0) (2.1.5)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (3.3.2)\n", - "Requirement already satisfied: idna<4,>=2.5 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (3.7)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (1.26.18)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from requests->huggingface-hub>=0.15.1->sentence-transformers>=3.0.0) (2024.6.2)\n", - "Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /Users/amna.mubashar/Library/Python/3.9/lib/python/site-packages (from sympy->torch>=1.11.0->sentence-transformers>=3.0.0) (1.3.0)\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.3.1\u001b[0m\n", - "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49m/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip\u001b[0m\n" - ] - } - ], + "outputs": [], "source": [ "%%bash\n", "\n", @@ -642,7 +496,9 @@ "if \"OPENAI_API_KEY\" not in os.environ:\n", " os.environ[\"OPENAI_API_KEY\"] = getpass(\"Enter OpenAI API key:\")\n", "\n", - "template = [ChatMessage.from_user(\"\"\"\n", + "template = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", " You have to answer the following question based on the given context information only.\n", "\n", " Context:\n", @@ -652,7 +508,9 @@ "\n", " Question: {{question}}\n", " Answer:\n", - " \"\"\")]\n", + " \"\"\"\n", + " )\n", + "]\n", "\n", "rag_pipeline = Pipeline()\n", "rag_pipeline.add_component(\n", diff --git a/tutorials/36_Building_Fallbacks_with_Conditional_Routing.ipynb b/tutorials/36_Building_Fallbacks_with_Conditional_Routing.ipynb index 7653633..80b6c36 100644 --- a/tutorials/36_Building_Fallbacks_with_Conditional_Routing.ipynb +++ b/tutorials/36_Building_Fallbacks_with_Conditional_Routing.ipynb @@ -14,7 +14,7 @@ "- **Prerequisites**: You must have an [OpenAI API Key](https://platform.openai.com/api-keys) and a [Serper API Key](https://serper.dev/api-key) for this tutorial\n", "- **Goal**: After completing this tutorial, you'll have learned how to create a pipeline with conditional routing that can fallback to websearch if the answer is not present in your dataset.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro).\n" + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n" ] }, { @@ -59,7 +59,7 @@ "source": [ "### Install Haystack\n", "\n", - "Install Haystack 2.0 with `pip`:" + "Install Haystack with `pip`:" ] }, { @@ -204,7 +204,9 @@ "from haystack.dataclasses import ChatMessage\n", "from haystack.components.generators.chat import OpenAIChatGenerator\n", "\n", - "prompt_template = [ChatMessage.from_user(\"\"\"\n", + "prompt_template = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", "Answer the following query given the documents.\n", "If the answer is not contained within the documents reply with 'no_answer'\n", "Query: {{query}}\n", @@ -212,7 +214,9 @@ "{% for document in documents %}\n", " {{document.content}}\n", "{% endfor %}\n", - "\"\"\")]\n", + "\"\"\"\n", + " )\n", + "]\n", "\n", "prompt_builder = ChatPromptBuilder(template=prompt_template)\n", "llm = OpenAIChatGenerator(model=\"gpt-4o-mini\")" @@ -239,10 +243,11 @@ }, "outputs": [], "source": [ - "\n", "from haystack.components.websearch.serper_dev import SerperDevWebSearch\n", "\n", - "prompt_for_websearch = [ChatMessage.from_user(\"\"\"\n", + "prompt_for_websearch = [\n", + " ChatMessage.from_user(\n", + " \"\"\"\n", "Answer the following query given the documents retrieved from the web.\n", "Your answer shoud indicate that your answer was generated from websearch.\n", "\n", @@ -251,7 +256,9 @@ "{% for document in documents %}\n", " {{document.content}}\n", "{% endfor %}\n", - "\"\"\")]\n", + "\"\"\"\n", + " )\n", + "]\n", "\n", "websearch = SerperDevWebSearch()\n", "prompt_builder_for_websearch = ChatPromptBuilder(template=prompt_for_websearch)\n", @@ -550,7 +557,8 @@ "provenance": [] }, "kernelspec": { - "display_name": "Python 3", + "display_name": ".venv", + "language": "python", "name": "python3" }, "language_info": { diff --git a/tutorials/37_Simplifying_Pipeline_Inputs_with_Multiplexer.ipynb b/tutorials/37_Simplifying_Pipeline_Inputs_with_Multiplexer.ipynb index 1f7f04e..91552ee 100644 --- a/tutorials/37_Simplifying_Pipeline_Inputs_with_Multiplexer.ipynb +++ b/tutorials/37_Simplifying_Pipeline_Inputs_with_Multiplexer.ipynb @@ -17,7 +17,7 @@ "\n", "> As of version 2.2.0, `Multiplexer` has been deprecated in Haystack and will be completely removed from Haystack as of v2.4.0. We recommend using [BranchJoiner](https://docs.haystack.deepset.ai/docs/branchjoiner) instead. For more details about this deprecation, check out [Haystack 2.2.0 release notes](https://github.com/deepset-ai/haystack/releases/tag/v2.2.0) on Github.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro)." + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro)." ] }, { @@ -56,7 +56,7 @@ "source": [ "### Install Haystack\n", "\n", - "Install Haystack 2.0 with `pip`:" + "Install Haystack with `pip`:" ] }, { diff --git a/tutorials/39_Embedding_Metadata_for_Improved_Retrieval.ipynb b/tutorials/39_Embedding_Metadata_for_Improved_Retrieval.ipynb index c591c1e..7b48edc 100644 --- a/tutorials/39_Embedding_Metadata_for_Improved_Retrieval.ipynb +++ b/tutorials/39_Embedding_Metadata_for_Improved_Retrieval.ipynb @@ -12,7 +12,7 @@ "- **Components Used**: [`InMemoryDocumentStore`](https://docs.haystack.deepset.ai/docs/inmemorydocumentstore), [`InMemoryEmbeddingRetriever`](https://docs.haystack.deepset.ai/docs/inmemoryembeddingretriever), [`SentenceTransformersDocumentEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformersdocumentembedder), [`SentenceTransformersTextEmbedder`](https://docs.haystack.deepset.ai/docs/sentencetransformerstextembedder)\n", "- **Goal**: After completing this tutorial, you'll have learned how to embed metadata information while indexing documents, to improve retrieval.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n", "\n", "> ⚠️ Note of caution: The method showcased in this tutorial is not always the right approach for all types of metadata. This method works best when the embedded metadata is meaningful. For example, here we're showcasing embedding the \"title\" meta field, which can also provide good context for the embedding model." ] @@ -45,73 +45,14 @@ "source": [ "### Install Haystack\n", "\n", - "Install Haystack 2.0 and other required packages with `pip`:" + "Install Haystack and other required packages with `pip`:" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: haystack-ai in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (2.0.0b7)\n", - "Collecting wikipedia\n", - " Downloading wikipedia-1.4.0.tar.gz (27 kB)\n", - " Preparing metadata (setup.py): started\n", - " Preparing metadata (setup.py): finished with status 'done'\n", - "Requirement already satisfied: boilerpy3 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (1.0.7)\n", - "Requirement already satisfied: haystack-bm25 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (1.0.2)\n", - "Requirement already satisfied: jinja2 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (3.1.3)\n", - "Requirement already satisfied: lazy-imports in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (0.3.1)\n", - "Requirement already satisfied: more-itertools in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (10.2.0)\n", - "Requirement already satisfied: networkx in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (3.2.1)\n", - "Requirement already satisfied: openai>=1.1.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (1.12.0)\n", - "Requirement already satisfied: pandas in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (2.2.0)\n", - "Requirement already satisfied: posthog in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (3.4.1)\n", - "Requirement already satisfied: pyyaml in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (6.0.1)\n", - "Requirement already satisfied: tenacity in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (8.2.3)\n", - "Requirement already satisfied: tqdm in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (4.66.2)\n", - "Requirement already satisfied: typing-extensions in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-ai) (4.9.0)\n", - "Collecting beautifulsoup4 (from wikipedia)\n", - " Using cached beautifulsoup4-4.12.3-py3-none-any.whl.metadata (3.8 kB)\n", - "Requirement already satisfied: requests<3.0.0,>=2.0.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from wikipedia) (2.31.0)\n", - "Requirement already satisfied: anyio<5,>=3.5.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from openai>=1.1.0->haystack-ai) (4.2.0)\n", - "Requirement already satisfied: distro<2,>=1.7.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from openai>=1.1.0->haystack-ai) (1.9.0)\n", - "Requirement already satisfied: httpx<1,>=0.23.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from openai>=1.1.0->haystack-ai) (0.26.0)\n", - "Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from openai>=1.1.0->haystack-ai) (1.10.9)\n", - "Requirement already satisfied: sniffio in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from openai>=1.1.0->haystack-ai) (1.3.0)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.3.2)\n", - "Requirement already satisfied: idna<4,>=2.5 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.6)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2.2.0)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2024.2.2)\n", - "Collecting soupsieve>1.2 (from beautifulsoup4->wikipedia)\n", - " Using cached soupsieve-2.5-py3-none-any.whl.metadata (4.7 kB)\n", - "Requirement already satisfied: numpy in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from haystack-bm25->haystack-ai) (1.26.4)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from jinja2->haystack-ai) (2.1.5)\n", - "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from pandas->haystack-ai) (2.8.2)\n", - "Requirement already satisfied: pytz>=2020.1 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from pandas->haystack-ai) (2024.1)\n", - "Requirement already satisfied: tzdata>=2022.7 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from pandas->haystack-ai) (2024.1)\n", - "Requirement already satisfied: six>=1.5 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from posthog->haystack-ai) (1.16.0)\n", - "Requirement already satisfied: monotonic>=1.5 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from posthog->haystack-ai) (1.6)\n", - "Requirement already satisfied: backoff>=1.10.0 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from posthog->haystack-ai) (2.2.1)\n", - "Requirement already satisfied: httpcore==1.* in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from httpx<1,>=0.23.0->openai>=1.1.0->haystack-ai) (1.0.2)\n", - "Requirement already satisfied: h11<0.15,>=0.13 in /Users/tuanacelik/opt/anaconda3/envs/mistral/lib/python3.12/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai>=1.1.0->haystack-ai) (0.14.0)\n", - "Using cached beautifulsoup4-4.12.3-py3-none-any.whl (147 kB)\n", - "Using cached soupsieve-2.5-py3-none-any.whl (36 kB)\n", - "Building wheels for collected packages: wikipedia\n", - " Building wheel for wikipedia (setup.py): started\n", - " Building wheel for wikipedia (setup.py): finished with status 'done'\n", - " Created wheel for wikipedia: filename=wikipedia-1.4.0-py3-none-any.whl size=11678 sha256=17926b00d77f1d294e927f20b0e52e7a137fcd6b219ca85e63570f3b5c7d58f3\n", - " Stored in directory: /Users/tuanacelik/Library/Caches/pip/wheels/63/47/7c/a9688349aa74d228ce0a9023229c6c0ac52ca2a40fe87679b8\n", - "Successfully built wikipedia\n", - "Installing collected packages: soupsieve, beautifulsoup4, wikipedia\n", - "Successfully installed beautifulsoup4-4.12.3 soupsieve-2.5 wikipedia-1.4.0\n" - ] - } - ], + "outputs": [], "source": [ "%%bash\n", "\n", diff --git a/tutorials/40_Building_Chat_Application_with_Function_Calling.ipynb b/tutorials/40_Building_Chat_Application_with_Function_Calling.ipynb index 6bc53f8..36df0b2 100644 --- a/tutorials/40_Building_Chat_Application_with_Function_Calling.ipynb +++ b/tutorials/40_Building_Chat_Application_with_Function_Calling.ipynb @@ -14,7 +14,7 @@ "- **Prerequisites**: You must have an [OpenAI API Key](https://platform.openai.com/api-keys) and be familiar with [creating pipelines](https://docs.haystack.deepset.ai/docs/creating-pipelines)\n", "- **Goal**: After completing this tutorial, you will have learned how to build chat applications that demonstrate agent-like behavior using OpenAI's function calling feature.\n", "\n", - "> This tutorial uses Haystack 2.0. To learn more, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack 2.0 Documentation](https://docs.haystack.deepset.ai/docs/intro).\n" + "> This tutorial uses the latest version of Haystack 2.x (`haystack-ai`). For more information on Haystack 2.0, read the [Haystack 2.0 announcement](https://haystack.deepset.ai/blog/haystack-2-release) or visit the [Haystack Documentation](https://docs.haystack.deepset.ai/docs/intro).\n" ] }, { @@ -45,7 +45,7 @@ "source": [ "## Setting up the Development Environment\n", "\n", - "Install Haystack 2.0 and [sentence-transformers](https://pypi.org/project/sentence-transformers/) using pip:" + "Install Haystack and [sentence-transformers](https://pypi.org/project/sentence-transformers/) using pip:" ] }, { @@ -481,7 +481,9 @@ "from haystack.dataclasses import ChatMessage\n", "from haystack.components.generators.chat import OpenAIChatGenerator\n", "\n", - "template = [ChatMessage.from_system(\"\"\"\n", + "template = [\n", + " ChatMessage.from_system(\n", + " \"\"\"\n", "Answer the questions based on the given context.\n", "\n", "Context:\n", @@ -490,7 +492,9 @@ "{% endfor %}\n", "Question: {{ question }}\n", "Answer:\n", - "\"\"\")]\n", + "\"\"\"\n", + " )\n", + "]\n", "rag_pipe = Pipeline()\n", "rag_pipe.add_component(\"embedder\", SentenceTransformersTextEmbedder(model=\"sentence-transformers/all-MiniLM-L6-v2\"))\n", "rag_pipe.add_component(\"retriever\", InMemoryEmbeddingRetriever(document_store=document_store))\n",