ryan_regex_lesson_work.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regex\n",
    "\n",
    "- What is a regular expression?\n",
    "- When are regular expressions useful?\n",
    "\n",
    "## Where we're going\"\n",
    "\n",
    "\n",
    "```\n",
    "76.185.131.226 - - [11/May/2020:14:25:53 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:25:46 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:25:58 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\"\n",
    "76.185.131.226 - - [11/May/2020:16:25:58 +0000] \"GET /favicon.ico HTTP/1.1\" 200 162 \"https://python.zach.lol/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\"\n",
    "104.5.217.57 - - [11/May/2020:16:26:27 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:26:46 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:26:54 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "104.5.217.57 - - [11/May/2020:16:27:04 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:27:05 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:27:10 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_file_lines = '''\n",
    "76.185.131.226 - - [11/May/2020:14:25:53 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:25:46 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:25:58 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\"\n",
    "76.185.131.226 - - [11/May/2020:16:25:58 +0000] \"GET /favicon.ico HTTP/1.1\" 200 162 \"https://python.zach.lol/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36\"\n",
    "104.5.217.57 - - [11/May/2020:16:26:27 +0000] \"GET / HTTP/1.1\" 200 42 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:26:46 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:26:54 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "104.5.217.57 - - [11/May/2020:16:27:04 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:27:05 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "76.185.131.226 - - [11/May/2020:16:27:10 +0000] \"GET /documentation HTTP/1.1\" 200 348 \"-\" \"python-requests/2.23.0\"\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re # part of the python stdlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- search: shows a single match for a regex\n",
    "- findall: shows *all* the matches for a regex in a subject"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Literals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = r'a'\n",
    "subject = 'abc'\n",
    "\n",
    "re.search(regexp, subject)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: rgba(0, 100, 200, .1); padding: 1em 3em; border-radius: 5px; border: 1px solid black\">\n",
    "    <div style=\"font-weight: bold; font-size: 1.2em; border-bottom: 1px dashed black; padding-bottom: .5em;\">\n",
    "        Mini Exercise\n",
    "    </div>\n",
    "    <p>Use the cell above to start experimenting with regular expressions.</p>\n",
    "    <ol>\n",
    "        <li>Change your regular expression to match the literal character \"b\". What do you notice?</li>\n",
    "        <li>Change your regular expression to match the literal string \"ab\". What do you notice?</li>\n",
    "        <li>Change your regular expression to match the literal \"d\". What do you notice?</li>\n",
    "        <li>Use <code>re.findall</code> instead of <code>re.search</code>. How do the results differ?</li>\n",
    "        <li>Change your regular expression to just the \".\" character. What are the results?</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(1, 2), match='b'>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'b', \"abc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 2), match='ab'>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'ab', \"abc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'abc', \"abc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.search(r'z', \"abc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'a', \"abba\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.search(r'a', \"ABBA\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='A'>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'a', \"ABBA\", re.IGNORECASE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'.', 'abc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'a']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'a', \"abba\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'zappa', \"abba\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metacharacters\n",
    "\n",
    "- `.`\n",
    "- `\\w`\n",
    "- `\\s`\n",
    "- `\\d`\n",
    "- Captial variants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = r'\\w'\n",
    "subject = 'abc 123 !@#'\n",
    "\n",
    "re.search(regexp, subject)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: rgba(0, 100, 200, .1); padding: 1em 3em; border-radius: 5px; border: 1px solid black\">\n",
    "    <div style=\"font-weight: bold; font-size: 1.2em; border-bottom: 1px dashed black; padding-bottom: .5em;\">\n",
    "        Mini Exercise\n",
    "    </div>\n",
    "    <p>Continue to use the same subject variable from above.</p>\n",
    "    <ol>\n",
    "        <li>Use all of the above metacharacters with <code>re.findall</code>. What do you notice?</li>\n",
    "        <li>What does the regular expression <code>\\w\\w</code> match?</li>\n",
    "        <li>Use only metacharacters to write a regular expression to match \"c 1\".</li>\n",
    "        <li>Use a combination of metacharacters to match 3 digits in a row.</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c', '1', '2', '3']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'\\w', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c', ' ', '1', '2', '3', ' ', '!', '@', '#']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'.', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\w', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(4, 5), match='1'>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\d', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(4, 7), match='123'>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\d\\d\\d', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(3, 4), match=' '>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# \\s matches whitespace\n",
    "re.search(f'\\s', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ', ' ']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# findall finds the two spaces in this string\n",
    "re.findall(f'\\s', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(2, 5), match='c 1'>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use only metacharacters to write a regular expression to match \"c 1\".\n",
    "re.search(r'\\w\\s\\d', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(2, 7), match='c 123'>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use only metacharacters to write a regular expression to match \"c 1\".\n",
    "re.search(r'c....', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(3, 4), match=' '>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Capital Variants mean NOT\n",
    "re.search(r'\\W', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c', '1', '2', '3', '!', '@', '#']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find everything that's not a space character\n",
    "re.findall(r'\\S', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c', ' ', ' ', '!', '@', '#']"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find all non-digits\n",
    "re.findall(r'\\D', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c', ' ', ' ', '!', '@', '#']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find all non words, non digits, non spaces\n",
    "re.findall(r'\\D', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' ', ' ', '!', '@', '#']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'\\W', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'...', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeating\n",
    "\n",
    "- `{}` specific number of repetitions\n",
    "- `*` is zero or more repetitions of whatever came before this character\n",
    "- `+` is one or more repetitions (refers to the expression before it)\n",
    "- `?` is optional, referring to the expression before it\n",
    "\n",
    "\n",
    "- `\\w+` matches one or more of `\\w`\n",
    "- `\\d+` matches one or more of `\\d`\n",
    "\n",
    "\n",
    "- greedy + non-greedy\n",
    "- non-greedy means to match as little as possible for the match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = r'\\w+'\n",
    "subject = 'abc 123'\n",
    "\n",
    "re.search(regexp, subject)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(4, 7), match='123'>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\d+', \"abc 123 !@#\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(3, 4), match=' '>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\W+', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 9), match='      abc'>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\s*\\w+', \"      abc 123 !@#\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(4, 6), match='12'>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match exactly 2 repetitions of \\d\n",
    "re.search(r'\\d{2}', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match 3 non digits\n",
    "re.search(r'\\D{3}', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(4, 7), match='123'>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match 2 or more \\d\n",
    "re.search(r'\\d{2,}', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(4, 6), match='12'>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\d{1,2}', 'abc 123 !@# 4567')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find a and \n",
    "re.search(r'az?', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 4), match='zapp'>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'zappa?', 'zapp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 5), match='zappa'>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'zappa?', 'zappa')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\w{3}', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(7, 10), match=' !@'>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'\\W{3}', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# question mark can mean something different in a different context\n",
    "# the question mark here makes the repetition non-greedy\n",
    "re.search(r'\\w+', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abc', '123']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'\\w+', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='acb 123'>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match as much as you can up to and including a digit\n",
    "re.search(r'.+\\d', 'acb 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 10), match='acb 123 !@'>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match as much as you can up to and including a @ character\n",
    "re.search(r'.+@', 'acb 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 5), match='acb 1'>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match as little as we can up to and including a digit\n",
    "# Non-greedy\n",
    "re.search(r'.+?\\d', 'acb 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='acb 123'>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Greedy\n",
    "# Match as little as we can up to and including a digit\n",
    "re.search(r'.+\\d', 'acb 123 !@#')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: rgba(0, 100, 200, .1); padding: 1em 3em; border-radius: 5px; border: 1px solid black\">\n",
    "    <div style=\"font-weight: bold; font-size: 1.2em; border-bottom: 1px dashed black; padding-bottom: .5em;\">\n",
    "        Mini Exercise\n",
    "    </div>\n",
    "    <p>Use the string below as your subject for this exercise.</p>\n",
    "    <pre><code>Codeup, founded in 2014, is located at 600 Navarro St. Suite 350, San Antonio, TX 78230. You can find us online at http://codeup.com and our alumni portal is located at https://alumni.codeup.com.</code></pre>\n",
    "    <ol>\n",
    "        <li>Write a regular expression that matches all the separate groups of numbers.</li>\n",
    "        <li>Write a regular expression that matches a 5 digit number, but not a number with fewer digits.</li>\n",
    "        <li>Write a regular expression to find all the numbers that have 4 or more digits\n",
    "        <li>Write a regular expression that matches any urls in the subject.</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "string = \"Codeup, founded in 2014, is located at 600 Navarro St. Suite 350, San Antonio, TX 78230. You can find us online at http://codeup.com and our alumni portal is located at https://alumni.codeup.com.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2', '0', '1', '4', '6', '0', '0', '3', '5', '0', '7', '8', '2', '3', '0']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This finds all single number characters \n",
    "re.findall(r'\\d', string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2014', '600', '350', '78230']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This finds all the groups of numbers where we have one or more digit until some other character\n",
    "re.findall(r'\\d+', string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(82, 87), match='78230'>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches a 5 digit number, but not a number with fewer digits.\n",
    "re.search(r'\\d{5}', string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2014', '78230']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression to find all the numbers that have 4 or more digits\n",
    "re.findall(r'\\d{4,}', string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2014', '600', '350', '7823', '0']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression to find all number groups up to 4 characters\n",
    "re.findall(r'\\d{1,4}', string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' 2014,', ' 600 ', ' 350,']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['2014', '600', '350']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression to find all numberal strings up to 4 characters \n",
    "# This should start with a non-digit, get 1 up to and including 4, and ends with a non-digit\n",
    "# but it looks like we need to process this to get rid of the first and last character\n",
    "# This seems like a good candidate for capture groups rather than hacking my results\n",
    "# Needing extra string/list manipulation AFTER regex seems like a hack or that I need more regex :)\n",
    "matches = re.findall(r'\\D\\d{1,4}\\D', string)\n",
    "print(matches)\n",
    "matches = [s[1:-1] for s in matches]\n",
    "matches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Any/None Of"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='abc'>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = r'[a1][b2][c3]'\n",
    "subject = 'abc 123'\n",
    "\n",
    "re.search(r'[a1][b2][c3]', 'abc 123')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 3), match='123'>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'[a1][b2][c3]', '123 abc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'[a-z]', 'abc 123 !@# ABC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', 'c']"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'[a-z]', 'abc 123 !@# ABC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(5, 6), match='2'>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'[2-6]', 'abc 123 6789')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2', '3', '6']"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(r'[2-6]', 'abc 123 6789')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(1, 2), match='b'>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find anything except a vowel\n",
    "re.search(r'[^aeiou]', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['a', 'b', ' ', '1', '2', ' ', '!', '@', '#']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find anything not the c or 3 character\n",
    "re.findall(r'[^c3]', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_vowel(x):\n",
    "    return bool(re.search(r'[aeiou]', x, re.IGNORECASE))\n",
    "\n",
    "assert is_vowel(\"E\") == True\n",
    "assert is_vowel(\"y\") == False\n",
    "assert is_vowel(\"b\") == False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: rgba(0, 100, 200, .1); padding: 1em 3em; border-radius: 5px; border: 1px solid black\">\n",
    "    <div style=\"font-weight: bold; font-size: 1.2em; border-bottom: 1px dashed black; padding-bottom: .5em;\">\n",
    "        Mini Exercise\n",
    "    </div>\n",
    "    <p>For this exercise you should make up various subjects and test them with your regular expressions.</p>\n",
    "    <ol>\n",
    "        <li>Write a regular expression that matches even numbers.</li>\n",
    "        <li>Write a regular expression that matches 2 or more odd numbers in a row.</li>\n",
    "        <li>Write a regular expression that any word with a vowel in it.</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1', '2', '4', '6', '8', '1', '0']"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches even numbers.\n",
    "re.findall(r'[246810]', 'abc 123455678910')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(5, 6), match='2'>"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches even numbers.\n",
    "# But this is kinda begging for anchors\n",
    "re.search(r'[02468]+', 'abc 123455678910')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['11', '33']"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches 2 or more odd numbers in a row.\n",
    "re.findall(r'[13579]{2,}', 'abc 112233')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Anchors\n",
    "\n",
    "- `^` starts with\n",
    "- `$` ends with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(1, 2), match='b'>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = r'b'\n",
    "subject = 'abc 123'\n",
    "\n",
    "re.search(regexp, subject)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# does this start with a b\n",
    "re.search(r'^b', 'abc 123')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# does this start with a word character\n",
    "re.search(r'^\\w', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='1235b67'>"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Does this start with and end with a digit? (and have zero or more of anything else inside)\n",
    "re.search(r'^\\d.*\\d$', '1235b67')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 23), match='1234 434 BAERGGJ #$$@ 1'>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Does this start with and end with a digit? (and have zero or more of anything else inside)\n",
    "re.search(r'^\\d.*\\d$', '1234 434 BAERGGJ #$$@ 1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 0), match=''>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Does this start and end with nothing?\n",
    "re.search(r'^$', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Does this sstart with a word?\n",
    "re.search(r'^\\w', 'abc 123 !@#$')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(10, 11), match='#'>"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# does this end with not a word?\n",
    "re.search(r'\\W$', 'abc 123 !@#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: rgba(0, 100, 200, .1); padding: 1em 3em; border-radius: 5px; border: 1px solid black\">\n",
    "    <div style=\"font-weight: bold; font-size: 1.2em; border-bottom: 1px dashed black; padding-bottom: .5em;\">\n",
    "        Mini Exercise\n",
    "    </div>\n",
    "    <p>For this exercise you should make up various subjects and test them with your regular expressions.</p>\n",
    "    <ol>\n",
    "        <li>Write a regular expression that matches if a word starts with a vowel.</li>\n",
    "        <li>Write a regular expression that matches if a word starts with a capital letter.</li>\n",
    "        <li>Write a regular expression that matches if a word ends with a capital letter.</li>        \n",
    "        <li>Write a regular expression that matches if a word starts <b>and</b> ends with a capital letter.</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='a'>"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches if a word starts with a vowel.\n",
    "re.search(r'^[aeiou]', 'abc 123 !@#', re.IGNORECASE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 1), match='A'>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches if a word starts with a capital letter.\n",
    "re.search(r'^[A-Z]', 'Abc 123 !@@!#')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(10, 11), match='C'>"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches if a word ends with a capital letter.\n",
    "re.search(r'[A-Z]$', 'abc 123 ABC')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 11), match='ABC 123 DEF'>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Write a regular expression that matches if a word starts and ends with a capital letter.\n",
    "# (and has anything in between)\n",
    "re.search(r'^[A-Z].*[A-Z]$', 'ABC 123 DEF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 6), match='ABCDEF'>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match a regex that matches if a word is only capitals\n",
    "re.search(r'^[A-Z]+[A-Z]$', 'ABCDEF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find something that starts with anything other than\n",
    "re.search(r'^[^a-c]', 'abc 123')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 7), match='def 123'>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find something that starts with anything other than\n",
    "# Math the entire rest of the string\n",
    "re.search(r'^[^a-c].*', 'def 123')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 4), match='!@#$'>"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Match anything and the entire string that ends with a dollar sign\n",
    "re.search(r'.*\\$$', '!@#$')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 4), match='!@#$'>"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.search(r'.*\\$$', '!@#$')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Capture Groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('abc', '123')"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What if I want to pull the numbers and digits out separately\n",
    "match = re.search(r'([a-z]+)\\s(\\d+)', 'abc 123')\n",
    "match.groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What if I want to pull the numbers and digits out separately\n",
    "# empty because order matters\n",
    "match = re.search(r'([a-z]+)\\s(\\d+)', '123 abc')\n",
    "match"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>123</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     0\n",
       "0  NaN\n",
       "1  123\n",
       "2  123"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = '.*?(\\d+)'\n",
    "s = pd.Series(['abc', 'abc123', '123'])\n",
    "s.str.extract(regexp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `re.sub`\n",
    "\n",
    "- removing\n",
    "- substitution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'abc'"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regexp = r'\\d+'\n",
    "subject = 'abc123'\n",
    "\n",
    "re.sub(regexp, '', subject)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"background-color: rgba(0, 100, 200, .1); padding: 1em 3em; border-radius: 5px; border: 1px solid black\">\n",
    "    <div style=\"font-weight: bold; font-size: 1.2em; border-bottom: 1px dashed black; padding-bottom: .5em;\">\n",
    "        Mini Exercise\n",
    "    </div>\n",
    "    <p>Use the code below to get started on this exercise.</p>\n",
    "    <pre><code>dates = pd.Series(['2020-11-12', '2020-07-13', '2021-01-12'])</code></pre>\n",
    "    <p>Use regular expression substitution to reformat the dates in the format common in the US: m/d/y.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Misc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas Usage\n",
    "\n",
    "- `.str`\n",
    "    - `.extract`\n",
    "    - `.count`\n",
    "    - `.contains`\n",
    "    - `.replace`\n",
    "- extract + concat\n",
    "- named groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>You should go check out https://regex101.com, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>My favorite search engine is https://duckduckg...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>If you have a question, you can get it answere...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text\n",
       "0  You should go check out https://regex101.com, ...\n",
       "1  My favorite search engine is https://duckduckg...\n",
       "2  If you have a question, you can get it answere..."
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame()\n",
    "df['text'] = pd.Series([\n",
    "    'You should go check out https://regex101.com, it is a great website!',\n",
    "    'My favorite search engine is https://duckduckgo.com',\n",
    "    'If you have a question, you can get it answered through http://askjeeves.com, it is great!',\n",
    "])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https</td>\n",
       "      <td>regex101</td>\n",
       "      <td>com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https</td>\n",
       "      <td>duckduckgo</td>\n",
       "      <td>com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>askjeeves</td>\n",
       "      <td>com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       0           1    2\n",
       "0  https    regex101  com\n",
       "1  https  duckduckgo  com\n",
       "2   http   askjeeves  com"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text.str.extract(r'(https?)://(\\w+)\\.(\\w+)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interactive Regex Tool\n",
    "\n",
    "To install the `hlre` tool:\n",
    "\n",
    "```\n",
    "python -m pip install hlre\n",
    "```\n",
    "\n",
    "[For more documentation and the source](https://github.com/zgulde/hlre)\n",
    "\n",
    "See also [regex101](https://regex101.com) (make sure to select the Python flavor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named capture groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'protocol': 'https', 'base_domain': 'regex101', 'tld': 'com'}"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = 'You should go check out https://regex101.com, it is a great website!'\n",
    "\n",
    "match = re.search(r'(?P<protocol>https?)://(?P<base_domain>\\w+)\\.(?P<tld>\\w+)', text)\n",
    "match.groupdict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>protocol</th>\n",
       "      <th>base_domain</th>\n",
       "      <th>tld</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https</td>\n",
       "      <td>regex101</td>\n",
       "      <td>com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https</td>\n",
       "      <td>duckduckgo</td>\n",
       "      <td>com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http</td>\n",
       "      <td>askjeeves</td>\n",
       "      <td>com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  protocol base_domain  tld\n",
       "0    https    regex101  com\n",
       "1    https  duckduckgo  com\n",
       "2     http   askjeeves  com"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.text.str.extract(r'(?P<protocol>https?)://(?P<base_domain>\\w+)\\.(?P<tld>\\w+)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verbose regular expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `re.VERBOSE`\n",
    "- `(?# this is a comment)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'protocol': 'https', 'base_domain': 'regex101', 'tld': 'com'}"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = 'You should go check out https://regex101.com, it is a great website!'\n",
    "\n",
    "regexp = r'''\n",
    "(?P<protocol>https?)\n",
    ":// (?# ignore the :// that seperates protocol from domain)\n",
    "(?P<base_domain>\\w+)\n",
    "\\.\n",
    "(?P<tld>\\w+)\n",
    "'''\n",
    "match = re.search(regexp, text, re.VERBOSE) # whitespace in the regex is ignored\n",
    "match.groupdict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}