src/documentation/document_index.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from os import getenv\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "from intelligence_layer.connectors import (\n",
    "    CollectionPath,\n",
    "    DocumentContents,\n",
    "    DocumentIndexClient,\n",
    "    DocumentIndexRetriever,\n",
    "    DocumentPath,\n",
    "    IndexConfiguration,\n",
    "    IndexPath,\n",
    "    InstructableEmbed,\n",
    "    LimitedConcurrencyClient,\n",
    "    ResourceNotFound,\n",
    "    SemanticEmbed,\n",
    ")\n",
    "from intelligence_layer.core import InMemoryTracer, LuminousControlModel\n",
    "from intelligence_layer.examples import MultipleChunkRetrieverQa, RetrieverBasedQaInput\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Index\n",
    "\n",
    "Answering questions given a known text may not be sufficient for your use case.\n",
    "At some point, you will probably want to search through, or answer questions about, your own knowledge base.\n",
    "You can leverage Aleph Alpha's DocumentIndex (DI) – a robust semantic search tool – to pinpoint sections in documents that align closely with your query.\n",
    "Simply upload your knowledge base, internal documents, manuals, SharePoint, and more to the DI for efficient searching.\n",
    "\n",
    "The DI splits documents into smaller parts called 'chunks', generates semantic embedding vectors for each, and stores them alongside each other.\n",
    "To find segments that closely match your query, the system identifies chunks with embedding vectors that best align semantically with your question.\n",
    "The DI seamlessly manages document updates (using document names), determines the ideal chunk size, and optimizes the vector space search process.\n",
    "\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "In this notebook, we will show you how to upload your own documents to the DI, how to search through your documents, and how to build a question-answering system based on your DI-knowledge base.\n",
    "To make proper use of the search and question-answering task, it is necessary to evaluate the results in an iterative way, to ensure it satisfies your requirements.\n",
    "For an example of how such an evaluation can look like, refer to [evaluation.ipynb](./evaluation.ipynb).\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload documents to the Document Index\n",
    "\n",
    "To search through the DI, you'll first need to upload the documents to it. We assume that the URL of your DI instance is available under the `DOCUMENT_INDEX_URL` environment variable, and that you already have a namespace and a token to access it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change this to your namespace\n",
    "NAMESPACE = \"Search\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_index = DocumentIndexClient(\n",
    "    token=getenv(\"AA_TOKEN\"),\n",
    "    base_document_index_url=getenv(\"DOCUMENT_INDEX_URL\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to start working with the DI, you need to execute five simple steps:\n",
    "\n",
    "1. Create a new collection.\n",
    "2. Assign an index to the collection. This makes it searchable.\n",
    "3. Define a few documents we will put in our collection.\n",
    "4. Upload the documents to the collection.\n",
    "5. Verify that the documents are successfully uploaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change this value if you want to use a collection of a different name\n",
    "COLLECTION = \"intelligence-layer-sdk-demo-collection\"\n",
    "\n",
    "collection_path = CollectionPath(namespace=NAMESPACE, collection=COLLECTION)\n",
    "\n",
    "document_index.create_collection(collection_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's create an index and assign it to this collection. You can do this before or after populating the collection with documents; the Document Index automatically updates semantic indexes in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change this value if you want to use an index of a different name\n",
    "INDEX = \"intelligence-layer-sdk-demo-index\"\n",
    "\n",
    "index_path = IndexPath(namespace=NAMESPACE, index=INDEX)\n",
    "\n",
    "# customise the parameters of the index here\n",
    "index_configuration = IndexConfiguration(\n",
    "    chunk_size=64,\n",
    "    chunk_overlap=0,\n",
    "    embedding=SemanticEmbed(model_name=\"luminous-base\", representation=\"asymmetric\"),\n",
    ")\n",
    "\n",
    "# create the namespace-wide index resource\n",
    "document_index.create_index(index_path, index_configuration)\n",
    "\n",
    "# assign the index to the collection\n",
    "document_index.assign_index_to_collection(collection_path, INDEX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define three documents based on Wikipedia articles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \"https://en.wikipedia.org/wiki/Robert_Moses\"\n",
    "document_1 = {\n",
    "    \"name\": \"robert_moses\",\n",
    "    \"content\": \"\"\"Robert Moses''' (December 18, 1888 – July 29, 1981) was an American [[urban planner]] and public official who worked in the [[New York metropolitan area]] during the early to mid 20th century. Despite never being elected to any office, Moses is regarded as one of the most powerful and influential individuals in the history of New York City and New York State. The grand scale of his infrastructural projects and his philosophy of urban development influenced a generation of engineers, architects, and urban planners across the United States.<ref name=\":0\" />\n",
    "\n",
    "Moses held various positions throughout his more than forty-year long career. He at times held up to 12 titles simultaneously, including [[New York City Parks Commissioner]] and chairman of the [[Long Island State Park Commission]].<ref>{{Cite web|url=https://www.pbs.org/wnet/need-to-know/environment/the-legacy-of-robert-moses/16018/|title=The legacy of Robert Moses|last=Sarachan|first=Sydney|date=January 17, 2013|website=Need to Know {{!}} PBS|language=en-US|access-date=December 3, 2019}}</ref> Having worked closely with New York governor [[Al Smith]] early in his career, Moses became expert in writing laws and navigating and manipulating the inner workings of state government. He created and led numerous semi-autonomous [[Public authority|public authorities]], through which he controlled millions of dollars in revenue and directly issued [[Bond (finance)|bonds]] to fund new ventures with little outside input or oversight.\n",
    "\n",
    "Moses's projects transformed the New York area and revolutionized the way cities in the U.S. were designed and built. As Long Island State Park Commissioner, Moses oversaw the construction of [[Jones Beach State Park]], the most visited public beach in the United States,<ref name=\"Jones Beach\">{{cite news |url=http://www.longislandexchange.com/jones-beach.html |website=Long Island Exchange |title=Jones Beach |access-date=November 21, 2012 |archive-url=https://web.archive.org/web/20130121130008/http://www.longislandexchange.com/jones-beach.html |archive-date=January 21, 2013 |url-status=dead }}</ref> and was the primary architect of the [[Parkways in New York|New York State Parkway System]]. As head of the [[MTA Bridges and Tunnels|Triborough Bridge Authority]], Moses had near-complete control over bridges and tunnels in New York City as well as the tolls collected from them, and built, among others, the [[Robert F. Kennedy Bridge|Triborough Bridge]], the [[Brooklyn–Battery Tunnel]], and the [[Throgs Neck Bridge]], as well as several major highways. These roadways and bridges, alongside [[urban renewal]] efforts that saw the destruction of huge swaths of tenement housing and their replacement with large [[New York City Housing Authority|public housing projects]], transformed the physical fabric of New York and inspired other cities to undertake similar development endeavors.\n",
    "\n",
    "Moses's reputation declined following the publication of [[Robert Caro]]'s [[Pulitzer Prize]]-winning biography ''[[The Power Broker]]'' (1974), which cast doubt on the purported benefits of many of Moses's projects and further cast Moses as racist. In large part because of ''The Power Broker'', Moses is today considered a controversial figure in the history of New York City.\n",
    "\n",
    "==Early life and career==\n",
    "Moses was born in [[New Haven, Connecticut]], on December 18, 1888, to [[German Jewish]] parents, Bella (Silverman) and Emanuel Moses.<ref>{{cite news | url=https://www.nytimes.com/learning/general/onthisday/bday/1218.html | title=Robert Moses, Master Builder, is Dead at 92| newspaper=The New York Times |archive-url=https://web.archive.org/web/20160305003155/https://www.nytimes.com/learning/general/onthisday/bday/1218.html |archive-date=March 5, 2016 |url-status=dead}}</ref>{{sfn|Caro|1974|p=25}} He spent the first nine years of his life living at 83 Dwight Street in New Haven, two blocks from [[Yale University]]. In 1897, the Moses family moved to New York City,{{sfn|Caro|1974|pp=29}} where they lived on East 46th Street off Fifth Avenue.<ref>{{cite web |url=http://www.newsday.com/community/guide/lihistory/ny-history-hs722a,0,7092161.story |title=The Master Builder |access-date=April 4, 2007 |last=DeWan |first=George |year=2007 |website=Long Island History |publisher=Newsday |archive-url=https://web.archive.org/web/20061211045554/http://www.newsday.com/community/guide/lihistory/ny-history-hs722a%2C0%2C7092161.story |archive-date=December 11, 2006 |url-status=dead  }}</ref> Moses's father was a successful department store owner and [[real estate]] speculator in New Haven. In order for the family to move to New York City, he sold his real estate holdings and store, then retired.{{sfn|Caro|1974|pp=29}} Moses's mother was active in the [[settlement movement]], with her own love of building. Robert Moses and his brother Paul attended several schools for their elementary and [[secondary education]], including the [[Ethical Culture Fieldston School|Ethical Culture School]], the [[Dwight School]] and the [[Mohegan Lake, New York#Historic places|Mohegan Lake School]], a military academy near [[Peekskill, New York|Peekskill]].{{sfn|Caro|1974|pp=35}}\n",
    "\n",
    "After graduating from [[Yale College]] (B.A., 1909) and [[Wadham College]], [[Oxford University|Oxford]] (B.A., Jurisprudence, 1911; M.A., 1913), and earning a Ph.D. in [[political science]] from [[Columbia University]] in 1914, Moses became attracted to New York City reform politics.<ref>{{Cite web|url=http://c250.columbia.edu/c250_celebrates/remarkable_columbians/robert_moses.html|title = Robert Moses}}</ref> A committed [[idealism|idealist]], he developed several plans to rid New York of [[Patronage#Politics|patronage hiring]] practices, including being the lead author of a 1919 proposal to reorganize the New York state government. None went very far, but Moses, due to his intelligence, caught the notice of [[Belle Moskowitz]], a friend and trusted advisor to Governor [[Al Smith]].{{sfn|Caro|1974}}  When the state [[Secretary of State of New York|Secretary of State's]] position became appointive rather than elective, Smith named Moses. He served from 1927 to 1929.<ref>{{cite news |date=December 19, 1928 |title=Moses Resigns State Position |url=http://cdsun.library.cornell.edu/cgi-bin/cornell?a=d&d=CDS19281219.2.63.7# |newspaper=Cornell Daily Sun |location=Ithaca, NY |page=8}}</ref>\n",
    "\n",
    "Moses rose to power with Smith, who was elected as governor in 1918, and then again in 1922. With Smith's support, Moses set in motion a sweeping consolidation of the New York State government. During that period Moses began his first foray into large-scale public work initiatives, while drawing on Smith's political power to enact legislation. This helped create the new [[Long Island State Park Commission]] and the State Council of Parks.<ref>{{cite web|last=Gutfreund|first=Owen|title=Moses, Robert|url=http://www.anb.org/articles/07/07-00375.html|publisher=Anb.org|access-date=December 24, 2014}}</ref> In 1924, Governor Smith appointed Moses chairman of the State Council of Parks and president of the Long Island State Park Commission.<ref>{{Cite book|title=Encyclopedia of the City|url=https://archive.org/details/encyclopediacity00cave|url-access=limited|last=Caves|first=R. W.|publisher=Routledge|year=2004|isbn=978-0-415-25225-6|pages=[https://archive.org/details/encyclopediacity00cave/page/n512 472]}}</ref> This centralization allowed Smith to run a government later used as a model for Franklin D. Roosevelt's [[New Deal]] federal government.{{or|date=October 2022}} Moses also received numerous commissions that he carried out efficiently, such as the development of [[Jones Beach State Park]].{{cn|date=October 2022}} Displaying a strong command of [[law]] as well as matters of [[engineering]], Moses became known for his skill in drafting legislation, and was called \"the best bill drafter in [[Albany, New York|Albany]]\".<ref name=\":0\">{{cite news |title=Annals of Power |first=Robert A. |last=Caro |author-link=Robert Caro |url=http://archives.newyorker.com/?i=1974-07-22#folio=032 |magazine=[[The New Yorker]] |date=July 22, 1974 |access-date=September 1, 2011}}</ref> At a time when the public was accustomed to [[Tammany Hall]] corruption and incompetence, Moses was seen as a savior of government.{{sfn|Caro|1974}}\n",
    "\n",
    "Shortly after [[President of the United States|President]] [[Franklin Delano Roosevelt|Franklin D. Roosevelt's]] [[First inauguration of Franklin D. Roosevelt|inauguration]] in 1933, the [[United States federal government|federal government]] found itself with millions of [[New Deal]] dollars to spend, yet states and cities had few projects ready. Moses was one of the few local officials who had projects [[shovel ready]]. For that reason, New York City was able to obtain significant [[Works Progress Administration]] (WPA), [[Civilian Conservation Corps]] (CCC), and other Depression-era funding. One of his most influential and longest-lasting positions was that of Parks Commissioner of New York City, a role he served from January 18, 1934, to May 23, 1960.<ref>{{Cite web|url=https://www.nycgovparks.org/about/history/commissioners|title=New York City Parks Commissioners : NYC Parks|website=www.nycgovparks.org|language=en|access-date=March 29, 2018}}</ref>\n",
    "\n",
    "==Offices held==\n",
    "The many offices and professional titles that Moses held gave him unusually broad power to shape urban development in the New York metropolitan region. These include, according to the New York Preservation Archive Project:<ref>{{Cite web|url=http://www.nypap.org/preservation-history/robert-moses/|title=Robert Moses {{!}}|website=www.nypap.org|language=en-US|access-date=March 29, 2018}}</ref>\n",
    "*[[Long Island State Park Commission]] (President, 1924–1963)\n",
    "* New York State Council of Parks (Chairman, 1924–1963)\n",
    "*[[Secretary of State of New York|New York Secretary of State]] (1927–1928)\n",
    "* Bethpage State Park Authority (President, 1933–1963)\n",
    "* Emergency Public Works Commission (Chairman, 1933–1934)\n",
    "* Jones Beach Parkway Authority (President, 1933–1963)\n",
    "*[[New York City Department of Parks and Recreation|New York City Department of Parks]] (Commissioner, 1934–1960)\n",
    "* [[Triborough Bridge]] and Tunnel Authority (Chairman, 1934–1968)\n",
    "* New York City Planning Commission (Commissioner, 1942–1960)\n",
    "* New York State Power Authority (Chairman, 1954–1962)\n",
    "* [[1964 New York World's Fair|New York's World Fair]] (President, 1960–1966)\n",
    "* Office of the Governor of New York (Special Advisor on Housing, 1974–1975)\n",
    "\n",
    "==Influence==\n",
    "During the 1920s, Moses sparred with [[Franklin D. Roosevelt]], then head of the Taconic State Park Commission, who favored the prompt construction of a [[parkway]] through the [[Hudson Valley]]. Moses succeeded in diverting funds to his Long Island parkway projects (the [[Northern State Parkway]], the [[Southern State Parkway]] and the [[Wantagh State Parkway]]), although the [[Taconic State Parkway]] was later completed as well.<ref>{{cite web|url=http://www.nycroads.com/roads/taconic/ |title=Taconic State Parkway |website=NYCRoads.com |access-date=May 25, 2006}}</ref> Moses helped build Long Island's [[Meadowbrook State Parkway]]. It was the first fully divided limited access highway in the world.<ref name=\"Leonard 1991 339\">{{cite book|last=Leonard|first=Wallock|title=The Myth of The Master Builder|year=1991|publisher=Journal of Urban History|page=339}}</ref>\n",
    "\n",
    "Moses was a highly influential figure in the initiation of many of the reforms that restructured New York state's government during the 1920s. A 'Reconstruction Commission' headed by Moses produced a highly influential report that provided recommendations that would largely be adopted, including the consolidation of 187 existing agencies under 18 departments, a new executive budget system, and the four-year term limit for the governorship.{{sfn|Caro|1974|pp=106, 260}}\"\"\",\n",
    "}\n",
    "\n",
    "# \"https://en.wikipedia.org/wiki/Jane_Jacobs\"\n",
    "document_2 = {\n",
    "    \"name\": \"jane_jacobs\",\n",
    "    \"content\": \"\"\"Jane Jacobs OC OOnt (née Butzner; 4 May 1916 – 25 April 2006) was an American-Canadian journalist, author, theorist, and activist who influenced urban studies, sociology, and economics. Her book The Death and Life of Great American Cities (1961) argued that \"urban renewal\" and \"slum clearance\" did not respect the needs of city-dwellers.[1][2]\n",
    "\n",
    "Jacobs organized grassroots efforts to protect neighborhoods from urban renewal and slum clearance – in particular plans by Robert Moses to overhaul her own Greenwich Village neighborhood. She was instrumental in the eventual cancellation of the Lower Manhattan Expressway,[3] which would have passed directly through the area of Manhattan that would later become known as SoHo, as well as part of Little Italy and Chinatown.[4] She was arrested in 1968 for inciting a crowd at a public hearing on that project.[5] After moving to Toronto in 1968, she joined the opposition to the Spadina Expressway and the associated network of expressways in Toronto that were planned and under construction.[6][7]\n",
    "\n",
    "As a woman and a writer who criticized experts in the male-dominated field of urban planning,[8][9] Jacobs endured scorn from established figures.[who?] Routinely, she was described first as a housewife,[10] as she did not have a college degree or any formal training in urban planning; as a result, her lack of credentials was seized upon as grounds for criticism.[11][12] However, the influence of her concepts eventually was acknowledged by highly respected professionals such as Richard Florida and Robert Lucas.[13] \"\"\",\n",
    "}\n",
    "\n",
    "# \"https://en.wikipedia.org/wiki/Nelson_Rockefeller\"\n",
    "document_3 = {\n",
    "    \"name\": \"nelson_rockefeller\",\n",
    "    \"content\": \"\"\"Nelson Aldrich Rockefeller (July 8, 1908 – January 26, 1979), sometimes referred to by his nickname Rocky,[1] was an American businessman and politician who served as the 41st vice president of the United States from 1974 to 1977 under President Gerald Ford. A member of the Republican Party and the wealthy Rockefeller family, he previously served as the 49th governor of New York from 1959 to 1973. Rockefeller also served as assistant secretary of State for American Republic Affairs for Presidents Franklin D. Roosevelt and Harry S. Truman (1944–1945) as well as under secretary of Health, Education and Welfare (HEW) under Dwight D. Eisenhower from 1953 to 1954. A son of John D. Rockefeller Jr. and Abby Aldrich Rockefeller as well as a grandson of Standard Oil co-founder John D. Rockefeller, he was a noted art collector and served as administrator of Rockefeller Center in Manhattan, New York City.\n",
    "\n",
    "Rockefeller was often considered to be liberal, progressive,[2] or moderate. In an agreement that was termed the Treaty of Fifth Avenue, he persuaded Richard Nixon to alter the Republican Party platform just before the 1960 Republican Convention. In his time, liberals in the Republican Party were called \"Rockefeller Republicans\". As Governor of New York from 1959 to 1973, Rockefeller's achievements included the expansion of the State University of New York (SUNY), efforts to protect the environment, the construction of the Empire State Plaza in Albany, increased facilities and personnel for medical care, and the creation of the New York State Council on the Arts.\n",
    "\n",
    "After unsuccessfully seeking the Republican presidential nomination in 1960, 1964, and 1968, he was appointed vice president of the United States under President Gerald Ford, who was appointed Vice President by President Richard Nixon after the resignation of Spiro Agnew, and who ascended to the presidency following Nixon's August 1974 resignation. Rockefeller was the second vice president appointed to the position under the 25th Amendment, following Ford himself. Rockefeller did not seek a full term on the 1976 Republican ticket with Ford. He retired from politics in 1977 and died two years later.\n",
    "\n",
    "As a businessman, Rockefeller was president and later chair of Rockefeller Center, Inc., and he formed the International Basic Economy Corporation in 1947. Rockefeller assembled a significant art collection and promoted public access to the arts. He served as trustee, treasurer, and president of the Museum of Modern Art and founded the Museum of Primitive Art in 1954. In the area of philanthropy, he founded the Rockefeller Brothers Fund in 1940 with his four brothers and established the American International Association for Economic and Social Development in 1946. \"\"\",\n",
    "}\n",
    "\n",
    "documents = [document_1, document_2, document_3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's upload the documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in documents:\n",
    "    document_path = DocumentPath(\n",
    "        collection_path=collection_path, document_name=doc[\"name\"]\n",
    "    )\n",
    "    document_index.add_document(\n",
    "        document_path, contents=DocumentContents.from_text(doc[\"content\"])\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's check if the documents are in the collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_index.documents(collection_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once a document is indexed, we can also have a look at its chunks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    chunks = document_index.chunks(\n",
    "        DocumentPath(collection_path=collection_path, document_name=document_1[\"name\"]),\n",
    "        index_name=INDEX,\n",
    "    )\n",
    "    print(chunks)\n",
    "except ResourceNotFound:\n",
    "    pass  # This is expected if the document is still embedding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search\n",
    "\n",
    "Now that we have uploaded our documents, we can search through them using the semantic similarities between a given query and each chunk.\n",
    "\n",
    "To do so, let's use the `DocumentIndexRetriever`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_index_retriever = DocumentIndexRetriever(\n",
    "    document_index=document_index,\n",
    "    index_name=INDEX,\n",
    "    namespace=NAMESPACE,\n",
    "    collection=COLLECTION,\n",
    "    k=5,\n",
    ")\n",
    "\n",
    "document_index_retriever.get_relevant_documents_with_scores(\n",
    "    query=\"The influence of Robert Moses\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hybrid Search\n",
    "\n",
    "The Document Index supports hybrid search, which combines results of semantic search and keyword search.\n",
    "\n",
    "In order to use hybrid search, we need to create a hybrid index and assign it to the collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change this value if you want to use an index of a different name\n",
    "HYBRID_INDEX = \"intelligence-layer-sdk-demo-hybrid-index\"\n",
    "\n",
    "index_path = IndexPath(namespace=NAMESPACE, index=HYBRID_INDEX)\n",
    "\n",
    "# customise the parameters of the index here\n",
    "index_configuration = IndexConfiguration(\n",
    "    chunk_size=64,\n",
    "    chunk_overlap=0,\n",
    "    hybrid_index=\"bm25\",\n",
    "    embedding=SemanticEmbed(model_name=\"luminous-base\", representation=\"asymmetric\"),\n",
    ")\n",
    "\n",
    "# create the namespace-wide index resource\n",
    "document_index.create_index(index_path, index_configuration)\n",
    "\n",
    "# assign the index to the collection\n",
    "document_index.assign_index_to_collection(collection_path, HYBRID_INDEX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we now search on the hybrid index, we will not only get chunks with a semantic similarity but also chunks that match the keywords in the query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_index_retriever = DocumentIndexRetriever(\n",
    "    document_index=document_index,\n",
    "    index_name=HYBRID_INDEX,\n",
    "    namespace=NAMESPACE,\n",
    "    collection=COLLECTION,\n",
    "    k=5,\n",
    "    threshold=0.5,\n",
    ")\n",
    "\n",
    "document_index_retriever.get_relevant_documents_with_scores(query=\"25 April\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Instructable Embeddings\n",
    "\n",
    "As well as supporting custom embedding models, the Document Index also supports instructable embeddings. This lets you prompt embedding models like `pharia-1-embedding-4608-control` with custom instructions for queries and documents. Steering the model like this can help the model understand nuances of your specific data and ultimately lead to embeddings that are more useful for your use-case. To use default instructions, leave the instruction fields unspecified.\n",
    "\n",
    "To use an instructable embedding model, create an index as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change this value if you want to use an index of a different name\n",
    "INSTRUCTABLE_EMBEDDING_INDEX = \"intelligence-layer-sdk-demo-instructable-embedding\"\n",
    "\n",
    "index_path = IndexPath(namespace=NAMESPACE, index=INSTRUCTABLE_EMBEDDING_INDEX)\n",
    "\n",
    "# customise the parameters of the index here\n",
    "index_configuration = IndexConfiguration(\n",
    "    chunk_size=64,\n",
    "    chunk_overlap=0,\n",
    "    embedding=InstructableEmbed(\n",
    "        model_name=\"pharia-1-embedding-4608-control\",\n",
    "        query_instruction=\"Represent the user's question about rivers to find a relevant wikipedia paragraph\",\n",
    "        document_instruction=\"Represent the document so that it can be matched to a user's question about rivers\",\n",
    "    ),\n",
    ")\n",
    "\n",
    "# create the namespace-wide index resource\n",
    "document_index.create_index(index_path, index_configuration)\n",
    "\n",
    "# assign the index to the collection\n",
    "document_index.assign_index_to_collection(collection_path, INSTRUCTABLE_EMBEDDING_INDEX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search with Metadata filtering\n",
    "\n",
    "The DocumentIndex also supports filter-indexes, which gives us the ability to provide specific filters in case we want to filter our search based on each document's metadata.\n",
    "\n",
    "To do so, let's first upload another version of our documents but this time with some metadata e.g. the \"title\" field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in documents:\n",
    "    document_path = DocumentPath(\n",
    "        collection_path=collection_path, document_name=doc[\"name\"]\n",
    "    )\n",
    "    document_index.add_document(\n",
    "        document_path,\n",
    "        contents=DocumentContents(\n",
    "            contents=[doc[\"content\"]], metadata={\"title\": doc[\"name\"]}\n",
    "        ),\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparation of the Filter Index\n",
    "To be able to use metadata filtering, we need to first check the following:\n",
    "1. Check if we already have a search index assigned. If not, we need to assign one because filter-indexes can be defined at the namespace level but can only be assigned to already existing search indexes \n",
    "2. Define a new filter-index configuration for our specific collection metadata.\n",
    "3. Assign the filter-index that we created to a search index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1\n",
    "# list all the assigned search indexes for our collection\n",
    "document_index.list_assigned_index_names(collection_path=collection_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2\n",
    "# define a new filter-index\n",
    "document_index.create_filter_index_in_namespace(\n",
    "    namespace=collection_path.namespace,\n",
    "    filter_index_name=\"demo-title\",  # this is how our filter-index is identified in our namespace\n",
    "    field_name=\"title\",  # this is the name of the field to which we want to apply our filter\n",
    "    field_type=\"string\",  # type of the field we want to apply our filter to. Must be one of \"string\", \"integer\", \"float\", \"boolean\" or \"datetime\"\n",
    ")\n",
    "\n",
    "# let's check if our index is present now\n",
    "\"demo-title\" in document_index.list_filter_indexes_in_namespace(\n",
    "    namespace=collection_path.namespace\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3\n",
    "# assign our new filter-index to our collection\n",
    "document_index.assign_filter_index_to_search_index(\n",
    "    collection_path=collection_path,\n",
    "    index_name=INDEX,  # we assign it to intelligence-layer-sdk-demo-index\n",
    "    filter_index_name=\"demo-title\",\n",
    ")\n",
    "\n",
    "# check if our filter-index is assigned to our collection\n",
    "document_index.list_assigned_filter_index_names(\n",
    "    collection_path=collection_path, index_name=INDEX\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Now, as we have the filter-index enabled, we need to initialize a new `DocumentIndexRetriever` with the search index for which we added the filter-index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "document_index_retriever = DocumentIndexRetriever(\n",
    "    document_index=document_index,\n",
    "    index_name=INDEX,\n",
    "    namespace=NAMESPACE,\n",
    "    collection=COLLECTION,\n",
    "    k=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining and Using a Filter\n",
    "Before we perform the filtered search we have to define a filter.\n",
    "Filters are composed of the following elements:\n",
    "- `filter_type` which can be one of \"with\", \"without\" or \"with_one_of\"\n",
    "- `filter_fields`, which defines the actual filtering criteria over a certain value for our chosen field\n",
    "\n",
    "If we want a filter that accepts only documents with the value of the \"title\" field equal to the \"name\" field of `document_1`, we define the filter as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from intelligence_layer.connectors import FilterField, FilterOps, Filters\n",
    "\n",
    "filters = Filters(\n",
    "    filter_type=\"with\",  # we want to only return documents matching our filter\n",
    "    fields=[\n",
    "        FilterField(\n",
    "            field_name=\"title\",  # this is the key we used in our metadata dict\n",
    "            field_value=document_1[\n",
    "                \"name\"\n",
    "            ],  # this is what we used as a value in the metadata dict\n",
    "            criteria=FilterOps.EQUAL_TO,  # we want to match exactly\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's use the filters with our query to restrict the search to documents with the title \"robert_moses\"\n",
    "document_index_retriever.get_relevant_documents_with_scores(\n",
    "    query=\"Robert Moses\", filters=[filters]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We only get document with the \"robert_moses\" as the exact title in the metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exclusion of Documents\n",
    "We can also modify our filter such that we only get documents that do *not* match the specified filter fields. This is as simply as replacing the \"with\" filter_type with a \"without\": "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's now try to exclude the document with the title \"robert_moses\"\n",
    "filters_without = Filters(\n",
    "    filter_type=\"without\",  # we change this to \"without\" to exclude the document\n",
    "    fields=[\n",
    "        FilterField(\n",
    "            field_name=\"title\",  # this is the key we used in our metadata dict\n",
    "            field_value=document_1[\n",
    "                \"name\"\n",
    "            ],  # this is what we used as a value in the metadata dict\n",
    "            criteria=FilterOps.EQUAL_TO,  # we want to match exactly\n",
    "        ),\n",
    "    ],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's use the filters with our query to exclude the document with the title \"robert_moses\"\n",
    "document_index_retriever.get_relevant_documents_with_scores(\n",
    "    query=\"Robert Moses\", filters=[filters_without]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, how we only get results where the `document_name` is not \"robert_moses\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question Answering\n",
    "\n",
    "Finally, since we have a ready-to use 'Retriever', we can employ it to do something more complicated than just search; it can serve as the basis for a question-answering system.\n",
    "\n",
    "To do so, let's run a `MultipleChunkRetrieverQa` task:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = LimitedConcurrencyClient.from_env()\n",
    "retriever_qa = MultipleChunkRetrieverQa(\n",
    "    document_index_retriever, insert_chunk_number=3, model=LuminousControlModel()\n",
    ")\n",
    "\n",
    "\n",
    "input = RetrieverBasedQaInput(\n",
    "    question=\"What is the name of the book about Robert Moses?\"\n",
    ")\n",
    "tracer = InMemoryTracer()\n",
    "\n",
    "output = retriever_qa.run(input, tracer)\n",
    "\n",
    "output.answer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn how the answer was produced, you can see it step-by-step in the `tracer`.\n",
    "Here, we record the input and output of every individual step up to the final answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracer"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython"
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}