ch3/jupyter-notebooks/embedding-components.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding components\n",
    "\n",
    "In this example, `docs` is a list of `Document` objects with text content to be embedded. The `OpenAIDocumentEmbedder` is initialized with an OpenAI API key and is used to generate embeddings for each document. The embeddings are then printed out for each document in the `docs` list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade haystack-ai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenAIDocumentEmbedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.dataclasses import Document\n",
    "from haystack.components.embedders import OpenAIDocumentEmbedder\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "\n",
    "# Load the .env file\n",
    "load_dotenv(\"./../../.env\")\n",
    "api_key = os.getenv(\"OPENAI_API_KEY\")\n",
    "\n",
    "# List of documents to embed\n",
    "docs = [Document(content=\"The quick brown fox jumps over the lazy dog.\"), \n",
    "        Document(content=\"To be or not to be, that is the question.\")]\n",
    "\n",
    "# Initialize the embedder with your OpenAI API key\n",
    "document_embedder = OpenAIDocumentEmbedder()\n",
    "\n",
    "# Run the embedder to get embeddings\n",
    "result = document_embedder.run(docs)\n",
    "\n",
    "# Access the embeddings stored in the documents\n",
    "for doc in result['documents']:\n",
    "    print(doc.embedding[0:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking a look at the result data structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metadata shows the model and usage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['meta']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### OpenAITextEmbedder\n",
    "\n",
    "In this snippet, `text_embedder` is created with an OpenAI API key and used to generate an embedding for the string \"I love pizza!\". The resulting embedding and associated metadata are then printed out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.components.embedders import OpenAITextEmbedder\n",
    "\n",
    "# Initialize the text embedder with your OpenAI API key\n",
    "text_embedder = OpenAITextEmbedder()\n",
    "\n",
    "# Text you want to embed\n",
    "text_to_embed = \"I love pizza!\"\n",
    "\n",
    "# Embed the text and print the result\n",
    "result_text= text_embedder.run(text_to_embed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_text.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we can access the embeddings through the embedding key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_text['embedding'][0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_text['meta']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SentenceTransformersDocumentEmbedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.components.embedders import SentenceTransformersDocumentEmbedder\n",
    "\n",
    "# Initialize the document embedder with a model from the Sentence Transformers library\n",
    "doc_embedder = SentenceTransformersDocumentEmbedder(model=\"sentence-transformers/all-mpnet-base-v2\")\n",
    "doc_embedder.warm_up()\n",
    "\n",
    "# Create a document to embed\n",
    "doc = Document(content=\"I love pizza!\")\n",
    "\n",
    "# Embed the document and print the embedding\n",
    "result = doc_embedder.run([doc])\n",
    "print(result['documents'][0].embedding[0:2])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result['documents'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SentenceTransformersTextEmbedder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.components.embedders import SentenceTransformersTextEmbedder\n",
    "\n",
    "# Initialize the text embedder with a specific model from Sentence Transformers\n",
    "text_embedder = SentenceTransformersTextEmbedder(model=\"sentence-transformers/all-mpnet-base-v2\")\n",
    "\n",
    "# Warm up the model before use\n",
    "text_embedder.warm_up()\n",
    "\n",
    "# Define the text you want to embed\n",
    "text_to_embed = \"I love pizza!\"\n",
    "\n",
    "# Embed the text and retrieve the embedding\n",
    "result = text_embedder.run(text_to_embed)\n",
    "\n",
    "# Print the embedding vector\n",
    "print(result['embedding'][0:2])\n",
    "# Output: List of floats representing the embedded vector\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result.keys()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-pipelines",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}