ch3/jupyter-notebooks/writing-components.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing content into a Document Store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### `DocumentWriter`\n",
    "\n",
    "#### Writing regular documents\n",
    "\n",
    "We can write `Document` objects into a Document Store using the `DocumentWriter` class. In this example, we create a `DocumentStore` and write a `Document` object into it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade haystack-ai\n",
    "!pip install \"sentence-transformers>=2.2.0\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.components.writers import DocumentWriter\n",
    "from haystack.document_stores.in_memory.document_store import InMemoryDocumentStore\n",
    "from haystack.dataclasses import Document\n",
    "\n",
    "# Initialize an in-memory document store\n",
    "doc_store = InMemoryDocumentStore()\n",
    "\n",
    "# Create the DocumentWriter component with the document store\n",
    "document_writer = DocumentWriter(document_store=doc_store)\n",
    "\n",
    "# Define a list of documents to write\n",
    "documents_to_write = [\n",
    "    Document(content=\"Document 1 content\"),\n",
    "    Document(content=\"Document 2 content\"),\n",
    "]\n",
    "\n",
    "# Use the DocumentWriter component to write documents to the store\n",
    "result = document_writer.run(documents=documents_to_write)\n",
    "\n",
    "# Print the number of documents written\n",
    "print(f\"Documents written: {result['documents_written']}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_store.count_documents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_store.filter_documents()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Writing embedded documents\n",
    "\n",
    "There may be times in which, either due to the size of the data, or to preserve semantic meaning while leveraging embedding models, that we may want to work with embeddings instead. \n",
    "\n",
    "We can follow the next key steps.\n",
    "\n",
    "* Compute Embeddings: Use either the `OpenAIDocumentEmbedder` or `SentenceTransformersDocumentEmbedder`, or other Haystack embedding model integration, to compute the embeddings for your documents.\n",
    "\n",
    "* Store Embeddings: The computed embeddings are stored in the embedding field of the Document objects.\n",
    "\n",
    "* Write to DocumentStore: Use the DocumentWriter component to write these Document objects, now with embeddings, into a DocumentStore.\n",
    "\n",
    "Here's an example code snippet that demonstrates how to use the SentenceTransformersDocumentEmbedder to write embeddings into a document store:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from haystack.document_stores.in_memory.document_store import InMemoryDocumentStore\n",
    "from haystack.components.writers import DocumentWriter\n",
    "from haystack.components.embedders import SentenceTransformersDocumentEmbedder\n",
    "from haystack.dataclasses import Document\n",
    "\n",
    "# Initialize document store and components\n",
    "doc_store = InMemoryDocumentStore()\n",
    "doc_embedder = SentenceTransformersDocumentEmbedder(model=\"sentence-transformers/all-mpnet-base-v2\")\n",
    "document_writer = DocumentWriter(document_store=doc_store)\n",
    "\n",
    "# Example document\n",
    "documents = [\n",
    "    Document(content=\"The quick brown fox jumps over the lazy dog.\"),\n",
    "    Document(content=\"When it comes to natural language processing, context is key.\")\n",
    "]\n",
    "\n",
    "# Warm up the embedder and compute embeddings\n",
    "doc_embedder.warm_up()\n",
    "embedded_docs = doc_embedder.run(documents)['documents']\n",
    "\n",
    "# Write documents with embeddings to the document store\n",
    "document_writer.run(documents=embedded_docs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Showing the document content and their embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve all documents\n",
    "all_documents = doc_store.filter_documents()\n",
    "\n",
    "# Print details of each document, including the embedding if it exists\n",
    "for doc in all_documents:\n",
    "    print(f\"Document ID: {doc.id}\")\n",
    "    print(f\"Content: {doc.content}\")\n",
    "    if doc.embedding:\n",
    "        print(f\"Embedding: {doc.embedding[:5]}...\")  # Displaying first 5 values of the embedding for brevity\n",
    "    print(\"\\n\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-pipelines",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}