api-entities/works/search-works.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![GitHub Repository](https://img.shields.io/badge/GitHub-Repository-181717?style=for-the-badge&logo=GitHub&link=https://github.com/Mearman/openalex-docs/tree/main)](https://github.com/Mearman/openalex-docs/tree/main)[![Open in GitHub](https://img.shields.io/badge/Open%20in-GitHub-181717?style=for-the-badge&logo=github&link=https://github.com/Mearman/openalex-docs/blob/main/api-entities/works/search-works.ipynb)](https://github.com/Mearman/openalex-docs/blob/main/api-entities/works/search-works.ipynb)[![Open in Colab](https://img.shields.io/badge/Open%20in-Colab-F9AB00?style=for-the-badge&logo=Google%20Colab&link=https://colab.research.google.com/github/Mearman/openalex-docs/blob/main/api-entities/works/search-works.ipynb)](https://colab.research.google.com/github/Mearman/openalex-docs/blob/main/api-entities/works/search-works.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install --upgrade \"git+https://github.com/Mearman/openalex-python-pydantic-v1.git\"\n",
        "%pip install pandasai"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from openalex_api import Configuration, ApiClient, AutocompleteApi, AuthorsApi, ConceptsApi, FundersApi, InstitutionsApi, PublishersApi, SourcesApi, WorksApi\n",
        "\n",
        "configuration = Configuration(host=\"https://api.openalex.org\")\n",
        "autocomplete_api = AutocompleteApi(ApiClient(configuration))\n",
        "authors_api = AuthorsApi(ApiClient(configuration))\n",
        "concepts_api = ConceptsApi(ApiClient(configuration))\n",
        "funders_api = FundersApi(ApiClient(configuration))\n",
        "institutions_api = InstitutionsApi(ApiClient(configuration))\n",
        "publishers_api = PublishersApi(ApiClient(configuration))\n",
        "sources_api = SourcesApi(ApiClient(configuration))\n",
        "works_api = WorksApi(ApiClient(configuration))\n",
        "\n",
        "from pandasai import SmartDataframe\n",
        "from pandasai.llm import OpenAI"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title  { run: \"auto\", display-mode: \"form\" }\n",
        "openapi_token = \"\" # @param {type:\"string\"}"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Search works\n",
        "\n",
        "The best way to search for works is to use the `search` query parameter, which searches across [titles](./work-object/README.md#title), [abstracts](./work-object/README.md#abstract_inverted_index), and [fulltext](./work-object/README.md#has_fulltext). Example:\n",
        "\n",
        "* Get works with search term \"dna\" in the title, abstract, or fulltext:\n",
        "  [`https://api.openalex.org/works?search=dna`](https://api.openalex.org/works?search=dna)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/works?search=dna\n",
        "search=\"dna\" # @param {type: \"string\"}\n",
        "\n",
        "response = works_api.get_works(\n",
        "\tsearch=search\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Fulltext search is available for a subset of works, see [`Work.has_fulltext`](./work-object/README.md#has_fulltext) for more details.\n",
        "\n",
        "{% hint style=\"info\" %}\n",
        "You can read more about search [here](./../../how-to-use-the-api/get-lists-of-entities/search-entities.ipynb). It will show you how relevance score is calculated, how words are stemmed to improve search results, and how to do complex boolean searches.\n",
        "{% endhint %}\n",
        "\n",
        "## Search a specific field\n",
        "\n",
        "You can use search as a [filter](./../../how-to-use-the-api/get-lists-of-entities/filter-entity-lists.ipynb), allowing you to fine-tune the fields you're searching over. To do this, you append `.search` to the end of the property you are filtering for:\n",
        "\n",
        "* Get works with \"cubist\" in the title:\n",
        "  [`https://api.openalex.org/works?filter=title.search:cubist`](https://api.openalex.org/works?filter=title.search:cubist)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/works?filter=title.search:cubist\n",
        "filter=\"title.search:cubist\" # @param {type: \"string\"}\n",
        "\n",
        "response = works_api.get_works(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The following fields can be searched within works:\n",
        "\n",
        "| Search filter                                                                        | Field that is searched                                                  |\n",
        "| ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- |\n",
        "| [`abstract.search`](./filter-works.md#abstract.search)                                 | [`abstract_inverted_index`](./work-object/README.md#abstract_inverted_index)     |\n",
        "| [`display_name.search`](./filter-works.md#display_name.search-alias-title.search)     | [`display_name`](./work-object/README.md#display_name)                            |\n",
        "| [`fulltext.search`](./filter-works.md#fulltext.search)                                 | fulltext via [`n-grams`](./get-n-grams.ipynb)                                |\n",
        "| [`title.search`](./filter-works.md#display_name.search-alias-title.search)            | [`display_name`](./work-object/README.md#display_name)                            |\n",
        "| [`title_and_abstract.search`](./filter-works.md#title_and_abstractsearch)            | [`display_name`](./work-object/README.md#display_name) and [`abstract_inverted_index`](./work-object/README.md#abstract_inverted_index)  |\n",
        "\n",
        "You can also use the filter `default.search`, which works the same as using the [`search` parameter](./README.md#search-works).\n",
        "\n",
        "### Why can't I search by name of related entity (author name, institution name, etc.)?\n",
        "\n",
        "Rather than searching for the _names_ of entities related to works—such as authors, institutions, and sources—you need to search by a more unique identifier for that entity, like the OpenAlex ID. This means that there is a 2 step process:\n",
        "\n",
        "1. Find the ID of the related entity. For example, if you're interested in works associated with NYU, you could search the `/institutions` endpoint for that name: [`https://api.openalex.org/institutions?search=nyu`](https://api.openalex.org/institutions?search=nyu). Looking at the first result, you'll see that the OpenAlex ID for NYU is `I57206974`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/institutions?search=nyu\n",
        "search=\"nyu\" # @param {type: \"string\"}\n",
        "\n",
        "response = institutions_api.get_institutions(\n",
        "\tsearch=search\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "2. Use a [filter ](./filter-works.ipynb)with the `/works` endpoint to get all of the works: [`https://api.openalex.org/works?filter=institutions.id:I57206974`](https://api.openalex.org/works?filter=institutions.id:I57206974)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/works?filter=institutions.id:I57206974\n",
        "filter=\"institutions.id:I57206974\" # @param {type: \"string\"}\n",
        "\n",
        "response = works_api.get_works(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Why can't you do this in just one step? Well, if you use the search term, \"NYU,\" you might end up missing the ones that use the full name \"New York University,\" rather than the initials. Sure, you could try to think of all possible variants and search for all of them, but you might miss some, and you risk putting in search terms that let in works that you're not interested in. Figuring out which works are actually associated with the \"NYU\" you're interested shouldn't be your responsibility—that's our job! We've done that work for you, so all the relevant works should be associated with one unique ID.\n",
        "\n",
        "## Autocomplete works\n",
        "\n",
        "You can autocomplete works to create a very fast type-ahead style search function:\n",
        "\n",
        "* Autocomplete works with \"tigers\" in the title:\n",
        "  [`https://api.openalex.org/autocomplete/works?q=tigers`](https://api.openalex.org/autocomplete/works?q=tigers)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/autocomplete/works?q=tigers\n",
        "q=\"tigers\" # @param {type: \"string\"}\n",
        "\n",
        "response = autocomplete_api.get_autocomplete_works(\n",
        "\tq=q\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This returns a list of works titles with the author of each work set as the hint:\n",
        "\n",
        "<pre class=\"language-json\"><code class=\"lang-json\">{ \n",
        "  \"results\": [\n",
        "    {\n",
        "      \"id\": \"https://openalex.org/W2125098916\",\n",
        "      \"display_name\": \"Crouching tigers, hidden prey: Sumatran tiger and prey populations in a tropical forest landscape\",\n",
        "      \"hint\": \"Timothy G. O'Brien, Margaret F. Kinnaird, Hariyo T. Wibisono\",\n",
        "      \"cited_by_count\": 620,\n",
        "      \"works_count\": null,\n",
        "      \"entity_type\": \"work\",\n",
        "      \"external_id\": \"https://doi.org/10.1017/s1367943003003172\"\n",
        "    },\n",
        "    ...\n",
        "<strong>  ]\n",
        "</strong><strong>}\n",
        "</strong></code></pre>\n",
        "\n",
        "{% hint style=\"info\" %}\n",
        "Read more about [autocomplete](./../../how-to-use-the-api/get-lists-of-entities/autocomplete-entities.ipynb).\n",
        "{% endhint %}"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}