claude_researcher_with_map.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/alexfazio/firecrawl-quickstarts/blob/main/claude_researcher_with_map.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8f015733",
      "metadata": {
        "id": "8f015733"
      },
      "source": [
        "# Firecrawl Web Crawling with OpenAI and Anthropic\n",
        "This notebook demonstrates how to use the Firecrawl API along with OpenAI's Anthropic to search for specific information on a website. It takes a user-defined objective and website URL, then attempts to find relevant pages and extract information based on the objective.\n",
        "\n",
        "### Requirements\n",
        "1. **Firecrawl API key**: Obtain from your Firecrawl account.\n",
        "2. **Anthropic API key**: Obtain from Anthropic if you're leveraging their models.\n",
        "3. **AgentOps API key**: If using AgentOps, include its API key.\n",
        "\n",
        "Set up your API keys as environment variables or directly in the notebook for ease of access.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "TuuO7HAFyuq9",
      "metadata": {
        "id": "TuuO7HAFyuq9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f025d6d5-8b78-4e21-a464-da21217fa515"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/946.0 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m450.6/946.0 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m942.1/946.0 kB\u001b[0m \u001b[31m18.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m946.0/946.0 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h\u001b[?25l   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/50.8 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.8/50.8 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.0/53.0 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m288.2/288.2 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m164.1/164.1 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "%pip install -q firecrawl-py anthropic agentops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "8e2be400",
      "metadata": {
        "id": "8e2be400"
      },
      "outputs": [],
      "source": [
        "from getpass import getpass\n",
        "from firecrawl import FirecrawlApp\n",
        "import os, re, json, anthropic, agentops"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "RJZQ-gYpGOl9",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RJZQ-gYpGOl9",
        "outputId": "fa8698d2-7126-4a89-f9f8-46c0b79e7d16"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FIRECRAWL_API_KEY··········\n",
            "ANTHROPIC_API_KEY··········\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False\n",
            "DEBUG:httpx:load_verify_locations cafile='/usr/local/lib/python3.10/dist-packages/certifi/cacert.pem'\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "AGENTOPS_API_KEY··········\n"
          ]
        }
      ],
      "source": [
        "# Initialize the FirecrawlApp, OpenAI client, and AgentOps\n",
        "app = FirecrawlApp(api_key=getpass('FIRECRAWL_API_KEY'))\n",
        "client = anthropic.Anthropic(api_key=getpass('ANTHROPIC_API_KEY'))\n",
        "AGENTOPS_API_KEY = getpass('AGENTOPS_API_KEY')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c625af64",
      "metadata": {
        "id": "c625af64"
      },
      "source": [
        "### Custom Color-Coded Logging Configuration\n",
        "This cell sets up a custom logging configuration with color-coded output for different log levels, enhancing readability for various messages.\n",
        "\n",
        "The `CustomFormatter` class applies specific colors to log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL) and resets colors after each log message.\n",
        "\n",
        "A `StreamHandler` is added to the root logger with this custom formatter, displaying messages in the notebook's output stream.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "98ec7066",
      "metadata": {
        "id": "98ec7066"
      },
      "outputs": [],
      "source": [
        "import logging\n",
        "\n",
        "# Set up colored logging\n",
        "class CustomFormatter(logging.Formatter):\n",
        "    COLORS = {\n",
        "        'DEBUG': '\\033[94m',   # Blue\n",
        "        'INFO': '\\033[92m',    # Green\n",
        "        'WARNING': '\\033[93m', # Yellow\n",
        "        'ERROR': '\\033[91m',   # Red\n",
        "        'CRITICAL': '\\033[95m' # Magenta\n",
        "    }\n",
        "    RESET = '\\033[0m'\n",
        "    FORMAT = \"[%(levelname)s] %(message)s\"\n",
        "\n",
        "    def format(self, record):\n",
        "        log_color = self.COLORS.get(record.levelname, self.RESET)\n",
        "        log_fmt = log_color + self.FORMAT + self.RESET\n",
        "        formatter = logging.Formatter(log_fmt)\n",
        "        return formatter.format(record)\n",
        "\n",
        "# Configure the root logger\n",
        "logger = logging.getLogger()\n",
        "logger.setLevel(logging.INFO)\n",
        "\n",
        "ch = logging.StreamHandler()\n",
        "ch.setLevel(logging.INFO)\n",
        "ch.setFormatter(CustomFormatter())\n",
        "\n",
        "# Add handler if not already added\n",
        "if not logger.hasHandlers():\n",
        "    logger.addHandler(ch)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "e732d54d",
      "metadata": {
        "id": "e732d54d"
      },
      "source": [
        "### Step 1: Finding the Relevant Page\n",
        "The function `find_relevant_page_via_map` takes an objective and a website URL. It then uses the Anthropic client to generate search parameters for the Firecrawl API to map the website and identify relevant pages based on the objective.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "13f88d4a",
      "metadata": {
        "id": "13f88d4a"
      },
      "outputs": [],
      "source": [
        "# Find the page that most likely contains the objective using Firecrawl's Map\n",
        "def find_relevant_page_via_map(objective, url, app, client):\n",
        "    \"\"\"\n",
        "    Identifies the page most likely to contain the specified objective using Firecrawl's Map.\n",
        "\n",
        "    Args:\n",
        "        objective (str): The objective to search for within the website pages.\n",
        "        url (str): The base URL of the website to be crawled.\n",
        "        app (object): The application instance for conducting the crawl.\n",
        "        client (object): The client used to make requests to the pages.\n",
        "\n",
        "    Returns:\n",
        "        str or None: Returns the URL of the page that most likely contains the objective if found; otherwise, returns None.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        logger.info(f\"{Colors.CYAN}Objective: {objective}{Colors.RESET}\")\n",
        "        logger.info(f\"{Colors.CYAN}Initiating search on the website: {url}{Colors.RESET}\")\n",
        "\n",
        "        map_prompt = f\"\"\"\n",
        "        The map function generates a list of URLs from a website and accepts a search parameter.\n",
        "        Based on the objective of: {objective}, suggest a 1-2 word search parameter.\n",
        "        \"\"\"\n",
        "\n",
        "        completion = client.messages.create(\n",
        "            model='claude-3-5-sonnet-20241022',\n",
        "            max_tokens=1000,\n",
        "            temperature=0,\n",
        "            system=\"Expert web crawler\",\n",
        "            messages=[{'role': 'user', 'content': map_prompt}]\n",
        "        )\n",
        "\n",
        "        map_search_parameter = completion.content[0].text\n",
        "        map_website = app.map_url(url, params={'search': map_search_parameter})\n",
        "\n",
        "        logger.info(f\"{Colors.GREEN}Mapping completed. Links found: {len(map_website['links'])}{Colors.RESET}\")\n",
        "        return map_website['links']\n",
        "    except Exception as e:\n",
        "        logger.info(f\"{Colors.RED}Error: {str(e)}{Colors.RESET}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c49f669d",
      "metadata": {
        "id": "c49f669d"
      },
      "source": [
        "### Step 2: Examining Top Pages using Firewcrawl's [Map](https://docs.firecrawl.dev/features/map)\n",
        "The function `find_objective_in_top_pages` examines the top pages from the website map, attempting to fulfill the user's objective using scraped content. If the objective is met, it returns the relevant data in JSON format.\n",
        "\n",
        "**Note:** Firecrawl's Map Response will be an ordered list from the most relevant to the least relevant. By selecting only the first three elements (`[:2]`), the function focuses on analyzing just the top three most relevant pages identified during the mapping stage..\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "812fd739",
      "metadata": {
        "id": "812fd739"
      },
      "outputs": [],
      "source": [
        "def find_objective_in_top_pages(map_website, objective, app, client):\n",
        "    \"\"\"\n",
        "    Scrapes the top 3 pages of a given website to check if the specified objective is met.\n",
        "\n",
        "    Args:\n",
        "        map_website (str): The website to be scraped.\n",
        "        objective (str): The objective to look for within the pages.\n",
        "        app (object): The application instance for web scraping.\n",
        "        client (object): The client used to request page content.\n",
        "\n",
        "    Returns:\n",
        "        dict or None: Returns a JSON object if the objective is found within the top 3 pages; otherwise, returns None.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        # Get top 2 links from the map result\n",
        "        top_links = map_website[:2]\n",
        "        logger.info(f\"{Colors.CYAN}Analyzing the {len(top_links)} top links: {top_links}{Colors.RESET}\")\n",
        "\n",
        "        batch_scrape_result = app.batch_scrape_urls(top_links, {'formats': ['markdown']})\n",
        "        logger.info(f\"{Colors.GREEN}Batch scraping completed.{Colors.RESET}\")\n",
        "\n",
        "        for scrape_result in batch_scrape_result['data']:\n",
        "            check_prompt = f\"\"\"\n",
        "            Given the following scraped content and objective, determine if the objective is met.\n",
        "            If it is, extract the relevant information in a simple and concise JSON format. Use only the necessary fields and avoid nested structures if possible.\n",
        "            If the objective is not met with confidence, respond with 'Objective not met'.\n",
        "\n",
        "            Objective: {objective}\n",
        "            Scraped content: {scrape_result['markdown']}\n",
        "\n",
        "            Remember:\n",
        "            1. Only return JSON if you are confident the objective is fully met.\n",
        "            2. Keep the JSON structure as simple and flat as possible.\n",
        "            3. Do not include any explanations or markdown formatting in your response.\n",
        "            \"\"\"\n",
        "\n",
        "            completion = client.messages.create(\n",
        "                model=\"claude-3-5-sonnet-20241022\",\n",
        "                max_tokens=1000,\n",
        "                temperature=0,\n",
        "                system=\"You are an expert web crawler. Respond with the relevant information in JSON format.\",\n",
        "                messages=[\n",
        "                    {\n",
        "                        \"role\": \"user\",\n",
        "                        \"content\": [\n",
        "                            {\n",
        "                                \"type\": \"text\",\n",
        "                                \"text\": check_prompt\n",
        "                            }\n",
        "                        ]\n",
        "                    }\n",
        "                ]\n",
        "            )\n",
        "\n",
        "            result = completion.content[0].text\n",
        "            if result and result != 'Objective not met':\n",
        "                try:\n",
        "                    return json.loads(result)\n",
        "                except json.JSONDecodeError as e:\n",
        "                    logger.info(f\"{Colors.RED}JSON parsing error: {e}. Raw result: {result}{Colors.RESET}\")\n",
        "                    continue  # Skip to the next result if parsing fails\n",
        "\n",
        "        logger.info(f\"{Colors.RED}Objective not met in examined content.{Colors.RESET}\")\n",
        "        return None\n",
        "    except Exception as e:\n",
        "        logger.info(f\"{Colors.RED}Error during analysis: {str(e)}{Colors.RESET}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "l3WzoekQEFCq",
      "metadata": {
        "id": "l3WzoekQEFCq"
      },
      "source": [
        "### Step 3: Find and Extract Information\n",
        "\n",
        "This function aims to find and extract information related to a given `objective` from the top-ranked pages of a website.\n",
        "\n",
        "**Functionality:**\n",
        "\n",
        "1. **Selects Top Links:** It selects the top two URLs from the `map_website` list, assuming they are the most relevant to the objective.\n",
        "2. **Scrapes Content:** It uses the `app.batch_scrape_urls` function to scrape content from these selected URLs in Markdown format.\n",
        "3. **Analyzes Content:**  For each scraped page, it constructs a prompt for the Anthropic Claude model. This prompt asks the model to determine if the scraped content fulfills the `objective`. If it does, the model is asked to extract the relevant information and format it as JSON.\n",
        "4. **Extracts JSON:** The function uses a regular expression to identify JSON-like blocks within the Anth"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "Kc31wkA3EEtl",
      "metadata": {
        "id": "Kc31wkA3EEtl"
      },
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "02DvK01wELut",
      "metadata": {
        "id": "02DvK01wELut"
      },
      "outputs": [],
      "source": [
        "def find_objective_in_top_pages(map_website, objective, app, client):\n",
        "    \"\"\"\n",
        "    Scrapes the top 3 pages of a website to determine if the specified objective is present.\n",
        "\n",
        "    Args:\n",
        "        map_website (str): The website map or URL structure to guide the scraping.\n",
        "        objective (str): The objective or target content to search for on the pages.\n",
        "        app (object): The application instance used for executing the scraping process.\n",
        "        client (object): The client responsible for requesting the content of the pages.\n",
        "\n",
        "    Returns:\n",
        "        dict or None: Returns a JSON object containing the found objective details if located on one of the top 3 pages; otherwise, returns None.\n",
        "    \"\"\"\n",
        "    try:\n",
        "        top_links = map_website[:2]\n",
        "        logger.info(f\"{Colors.CYAN}Analyzing top links: {top_links}{Colors.RESET}\")\n",
        "\n",
        "        batch_scrape_result = app.batch_scrape_urls(top_links, {'formats': ['markdown']})\n",
        "        logger.info(f\"{Colors.GREEN}Batch scraping completed.{Colors.RESET}\")\n",
        "\n",
        "        # Regex pattern to match JSON-like blocks in the response\n",
        "        json_pattern = r\"\\{(?:[^{}]|(?:\\{[^{}]*\\}))*\\}\"\n",
        "\n",
        "        for scrape_result in batch_scrape_result['data']:\n",
        "            check_prompt = f\"\"\"\n",
        "            Given scraped content and objective, determine if the objective is met.\n",
        "            Extract relevant information in simple JSON if met.\n",
        "            Objective: {objective}\n",
        "            Scraped content: {scrape_result['markdown']}\n",
        "            \"\"\"\n",
        "\n",
        "            completion = client.messages.create(\n",
        "                model='claude-3-5-sonnet-20241022',\n",
        "                max_tokens=1000,\n",
        "                temperature=0,\n",
        "                system=\"Expert web crawler\",\n",
        "                messages=[{'role': 'user', 'content': check_prompt}]\n",
        "            )\n",
        "\n",
        "            result = completion.content[0].text\n",
        "            # Search for JSON-like block in the result text\n",
        "            json_match = re.search(json_pattern, result, re.DOTALL)\n",
        "            if json_match:\n",
        "                try:\n",
        "                    return json.loads(json_match.group(0))\n",
        "                except json.JSONDecodeError as e:\n",
        "                    logger.info(f\"{Colors.RED}JSON parsing error: {e}. Raw result: {json_match.group(0)}{Colors.RESET}\")\n",
        "                    continue  # Skip to the next result if parsing fails\n",
        "            else:\n",
        "                logger.info(f\"{Colors.YELLOW}No JSON found in the response. Raw result: {result}{Colors.RESET}\")\n",
        "\n",
        "        logger.info(f\"{Colors.RED}Objective not met in examined content.{Colors.RESET}\")\n",
        "        return None\n",
        "    except Exception as e:\n",
        "        logger.info(f\"{Colors.RED}Error during analysis: {str(e)}{Colors.RESET}\")\n",
        "        return None"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9fc4cef6",
      "metadata": {
        "id": "9fc4cef6"
      },
      "source": [
        "### Step 4: Executing the Main Function\n",
        "The main function prompts for user input (website URL and objective), calls the `find_relevant_page_via_map` and `find_objective_in_top_pages` functions, and displays results accordingly.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "e3721623",
      "metadata": {
        "id": "e3721623"
      },
      "outputs": [],
      "source": [
        "# Main function to execute the process\n",
        "def main():\n",
        "    url = input(f\"{Colors.BLUE}Enter website URL:{Colors.RESET}\") or \"https://www.firecrawl.dev/\"\n",
        "    objective = input(f\"{Colors.BLUE}Enter objective:{Colors.RESET}\") or \"find pricing plans\"\n",
        "\n",
        "    map_website = find_relevant_page_via_map(objective, url, app, client)\n",
        "\n",
        "    if map_website:\n",
        "        result = find_objective_in_top_pages(map_website, objective, app, client)\n",
        "        if result:\n",
        "            logger.info(f\"{Colors.GREEN}Objective met. Extracted info:{Colors.RESET}\")\n",
        "            logger.info(f\"{Colors.MAGENTA}{json.dumps(result, indent=2)}{Colors.RESET}\")\n",
        "        else:\n",
        "            logger.info(f\"{Colors.RED}Objective not fulfilled with available content.{Colors.RESET}\")\n",
        "    else:\n",
        "        logger.info(f\"{Colors.RED}No relevant pages identified.{Colors.RESET}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "0cr6H3nlBSMG",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0cr6H3nlBSMG",
        "outputId": "452e3ae2-fd24-44a9-d0e1-8086f1204ef2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[94mEnter website URL:\u001b[0mhttps://www.firecrawl.dev/\n",
            "\u001b[94mEnter objective:\u001b[0myes or no: is firecrawl backed by y combinator?\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:root:\u001b[96mObjective: yes or no: is firecrawl backed by y combinator?\u001b[0m\n",
            "INFO:root:\u001b[96mInitiating search on the website: https://www.firecrawl.dev/\u001b[0m\n",
            "INFO:httpx:HTTP Request: POST https://api.anthropic.com/v1/messages \"HTTP/1.1 200 OK\"\n",
            "INFO:root:\u001b[92mMapping completed. Links found: 42\u001b[0m\n",
            "INFO:root:\u001b[96mAnalyzing top links: ['https://www.firecrawl.dev/blog/your-ip-has-been-temporarily-blocked-or-banned', 'https://www.firecrawl.dev/blog/how-to-quickly-install-beautifulsoup-with-python']\u001b[0m\n",
            "INFO:root:\u001b[92mBatch scraping completed.\u001b[0m\n",
            "INFO:httpx:HTTP Request: POST https://api.anthropic.com/v1/messages \"HTTP/1.1 200 OK\"\n",
            "INFO:root:\u001b[92mObjective met. Extracted info:\u001b[0m\n",
            "INFO:root:\u001b[95m{\n",
            "  \"can_determine\": false,\n",
            "  \"reason\": \"No mention of Y Combinator backing in the scraped content\"\n",
            "}\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "main()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}