2_data_wrangling.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "b4441388-25dc-4e46-98fe-66b620cb121c",
      "metadata": {},
      "source": [
        "# Part 2: Wrangle your data\n",
        "\n",
        "## Python **packages** let you use code written by experts around the world\n",
        "\n",
        "Because Python is open source, lots of different people and\n",
        "organizations can contribute (including you!). Many contributions are in\n",
        "the form of **packages** which do not come with a standard Python\n",
        "download.\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-titled callout-read\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div><div class=\"callout-title-container flex-fill\">Read More: Packages need to be installed and imported.</div></div><div class=\"callout-body-container callout-body\"><p>Learn more about using Python packages. How do you find and use\n",
        "packages? What is the difference between installing and importing\n",
        "packages? When do you need to do each one? <a\n",
        "href=\"https://www.earthdatascience.org/courses/intro-to-earth-data-science/python-code-fundamentals/use-python-packages/\">This\n",
        "article on Python packages</a> will walk you through the basics.</p></div></div>\n",
        "\n",
        "In the cell below, someone was trying to import the **pandas package**,\n",
        "which helps us to work with [**tabular data** such as comma-separated\n",
        "value or csv\n",
        "files](https://www.earthdatascience.org/courses/intro-to-earth-data-science/file-formats/use-text-files/).\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-titled callout-task\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div><div class=\"callout-title-container flex-fill\">Try It: Import a package</div></div><div class=\"callout-body-container callout-body\"><ol type=\"1\">\n",
        "<li>Correct the typo below to properly import the pandas package under\n",
        "its <strong>alias</strong> pd.</li>\n",
        "<li>Run the cell to import pandas</li>\n",
        "</ol></div></div>\n",
        "\n",
        "> **Warning**\n",
        ">\n",
        "> Make sure to run your code in the right **environment** to avoid\n",
        "> import errors!\n",
        ">\n",
        "> We’ve created a coding **environment** for you to use that already has\n",
        "> all the software and packages you will need! When you try to run some\n",
        "> code, you may be prompted to select a **kernel**. The **kernel**\n",
        "> refers to the version of Python you are using. You should use the\n",
        "> **base** kernel, which should be the default option for you."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "01452646",
      "metadata": {
        "template": "student"
      },
      "outputs": [],
      "source": [
        "# Import pandas\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f941643e-a138-4d54-ab4b-5354461e95cd",
      "metadata": {},
      "source": [
        "## Download the practice data\n",
        "\n",
        "Next, lets download some climate data from\n",
        "**Boulder, CO** to practice with. We keep our\n",
        "practice data on GitHub, so that we can check that it still works and\n",
        "make sure it looks just like the data you would download from the\n",
        "original source.\n",
        "\n",
        "The cell below contains the URL for the data you will use in this part\n",
        "of the notebook. There are two things to notice about the URL code:\n",
        "\n",
        "1.  It is surrounded by quotes – that means Python will interpret it as\n",
        "    a `string`, or text, type, which makes sense for a URL.\n",
        "2.  The URL is too long to display as one line on most screens. We’ve\n",
        "    put parentheses around it so that we can easily split it into\n",
        "    multiple lines by writing two strings – one on each line.\n",
        "\n",
        "However, we still have a problem - we can’t get the URL back later on\n",
        "because it isn’t saved in a **variable**. In other words, we need to\n",
        "give the url a **name** so that we can request in from Python later\n",
        "(sadly, Python has no ‘hey what was that thingy I typed yesterday?’\n",
        "function).\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-titled callout-read\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div><div class=\"callout-title-container flex-fill\">Read More: Names/variables in Python</div></div><div class=\"callout-body-container callout-body\"><p>One of the most common challenges for new programmers is making sure\n",
        "that your results are stored so you can use them again. In Python, this\n",
        "is called <strong>naming</strong>, or saving a\n",
        "<strong>variable</strong>. Learn more in this <a\n",
        "href=\"https://www.earthdatascience.org/courses/intro-to-earth-data-science/python-code-fundamentals/get-started-using-python/variables/\">hands-on\n",
        "activity on using variables</a> from our learning portal.</p></div></div>\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-titled callout-task\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div><div class=\"callout-title-container flex-fill\">Try It: Save the URL for later</div></div><div class=\"callout-body-container callout-body\"><ol type=\"1\">\n",
        "<li>Pick an expressive variable name for the URL.</li>\n",
        "<li>Click on the <code>Jupyter</code> tab in the console panel at the\n",
        "bottom of VSCode to see all your variables. Your new url variable will\n",
        "not be there until you define it and run the code.</li>\n",
        "<li>At the end of the cell where you define your url variable,\n",
        "<strong>call your variable (type out its name)</strong> so you can take a look at it.</li>\n",
        "</ol></div></div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "f8eeb561",
      "metadata": {
        "template": "student"
      },
      "outputs": [],
      "source": [
        "data_url=(\n",
        "    'https://github.com/cu-esiil-edu/esiil-learning-portal'\n",
        "    '/releases/download/data-release/climate-foundations-data.csv'\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "07497c16-8a02-45af-986c-efa027b281dc",
      "metadata": {},
      "source": [
        "The `pandas` library you imported can download data from the internet\n",
        "directly into a type of Python **object** called a `DataFrame`. In the\n",
        "code cell below, you can see an attempt to do just this. But there are\n",
        "some problems…\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-titled callout-task\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div><div class=\"callout-title-container flex-fill\">Try It: Fix some code!</div></div><div class=\"callout-body-container callout-body\"><ol type=\"1\">\n",
        "<li><p>Leave a space between the <code>#</code> and text in the comment\n",
        "and try making the comment more informative</p></li>\n",
        "<li><p>Make any changes needed to get this code to run. HINT: The\n",
        "<code>my_url</code> variable doesn’t exist - you need to replace it with\n",
        "the variable name <strong>you</strong> chose.</p></li>\n",
        "<li><p>Modify the <code>.read_csv()</code> statement to include the\n",
        "following parameters:</p>\n",
        "<ul>\n",
        "<li><code>index_col='DATE'</code> – this sets the <code>DATE</code>\n",
        "column as the index. Needed for subsetting and resampling later on</li>\n",
        "<li><code>parse_dates=True</code> – this lets <code>python</code> know\n",
        "that you are working with time-series data, and values in the indexed\n",
        "column are <strong>date time objects</strong></li>\n",
        "<li><code>na_values=['NaN']</code> – this lets <code>python</code> know\n",
        "how to handle missing values</li>\n",
        "</ul></li>\n",
        "<li><p>Clean up the code by using <strong>expressive variable\n",
        "names</strong>, <strong>expressive column names</strong>, <strong>PEP-8\n",
        "compliant code</strong>, and <strong>descriptive\n",
        "comments</strong></p></li>\n",
        "</ol></div></div>\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-warning\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div></div><div class=\"callout-body-container callout-body\"><p><strong>Make sure to call your <code>DataFrame</code> by typing it’s\n",
        "name as the last line of your code cell</strong> Then, you will be able\n",
        "to run the test cell below and find out if your answer is correct.</p></div></div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "c31ae9ff",
      "metadata": {
        "template": "student"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>STATION</th>\n",
              "      <th>PRCP</th>\n",
              "      <th>TOBS</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>DATE</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1893-10-01</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.94</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-02</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-03</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-04</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.04</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-05</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-26</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>74.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-27</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>69.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-28</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>73.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-29</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>66.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-30</th>\n",
              "      <td>USC00050848</td>\n",
              "      <td>0.00</td>\n",
              "      <td>78.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>45971 rows × 3 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                STATION  PRCP  TOBS\n",
              "DATE                               \n",
              "1893-10-01  USC00050848  0.94   NaN\n",
              "1893-10-02  USC00050848  0.00   NaN\n",
              "1893-10-03  USC00050848  0.00   NaN\n",
              "1893-10-04  USC00050848  0.04   NaN\n",
              "1893-10-05  USC00050848  0.00   NaN\n",
              "...                 ...   ...   ...\n",
              "2023-09-26  USC00050848  0.00  74.0\n",
              "2023-09-27  USC00050848  0.00  69.0\n",
              "2023-09-28  USC00050848  0.00  73.0\n",
              "2023-09-29  USC00050848  0.00  66.0\n",
              "2023-09-30  USC00050848  0.00  78.0\n",
              "\n",
              "[45971 rows x 3 columns]"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "climate_dataframe = pd.read_csv(\n",
        "    data_url,\n",
        "    index_col='DATE',parse_dates=True,na_values=['NaN'])\n",
        "climate_dataframe"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4b9eeb50-b4b3-4fce-9ec7-fa2de5969f00",
      "metadata": {},
      "source": [
        "> HINT: Check out the `type()` function below - you can use it to check\n",
        "> that your data is now in `DataFrame` type object"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "e194ab72",
      "metadata": {
        "template": "student"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "pandas.core.frame.DataFrame"
            ]
          },
          "execution_count": 4,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Check that the data was imported into a pandas DataFrame\n",
        "type(climate_dataframe)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f5b2f77c-85bf-48cf-acbf-f392fe31fa0f",
      "metadata": {},
      "source": [
        "## Clean up your `DataFrame`\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-titled callout-task\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div><div class=\"callout-title-container flex-fill\">Try It: Get rid of unwanted columns</div></div><div class=\"callout-body-container callout-body\"><p>You can use <strong>double brackets</strong> (<code>[[</code> and\n",
        "<code>]]</code>) to select only the columns that you want from your\n",
        "<code>DataFrame</code>:</p>\n",
        "<ol type=\"1\">\n",
        "<li>Change <code>some_column_name</code> to the Precipitation column\n",
        "name and <code>another_column_name</code> to the Observed Temperature\n",
        "column name.</li>\n",
        "</ol>\n",
        "<div data-__quarto_custom=\"true\" data-__quarto_custom_type=\"Callout\"\n",
        "data-__quarto_custom_context=\"Block\" data-__quarto_custom_id=\"9\">\n",
        "<div data-__quarto_custom_scaffold=\"true\">\n",
        "\n",
        "</div>\n",
        "<div data-__quarto_custom_scaffold=\"true\">\n",
        "<p>Column names are text values, not variable names, so you need to put\n",
        "them in quotes!</p>\n",
        "</div>\n",
        "</div></div></div>\n",
        "\n",
        "<link rel=\"stylesheet\" type=\"text/css\" href=\"./assets/styles.css\"><div class=\"callout callout-style-default callout-warning\"><div class=\"callout-header\"><div class=\"callout-icon-container\"><i class=\"callout-icon\"></i></div></div><div class=\"callout-body-container callout-body\"><p><strong>Make sure to call your <code>DataFrame</code> by typing it’s\n",
        "name as the last line of your code cell</strong> Then, you will be able\n",
        "to run the test cell below and find out if your answer is correct.</p></div></div>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "f581f291",
      "metadata": {
        "template": "student"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>PRCP</th>\n",
              "      <th>TOBS</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>DATE</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1893-10-01</th>\n",
              "      <td>0.94</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-02</th>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-03</th>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-04</th>\n",
              "      <td>0.04</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1893-10-05</th>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-26</th>\n",
              "      <td>0.00</td>\n",
              "      <td>74.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-27</th>\n",
              "      <td>0.00</td>\n",
              "      <td>69.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-28</th>\n",
              "      <td>0.00</td>\n",
              "      <td>73.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-29</th>\n",
              "      <td>0.00</td>\n",
              "      <td>66.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023-09-30</th>\n",
              "      <td>0.00</td>\n",
              "      <td>78.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>45971 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            PRCP  TOBS\n",
              "DATE                  \n",
              "1893-10-01  0.94   NaN\n",
              "1893-10-02  0.00   NaN\n",
              "1893-10-03  0.00   NaN\n",
              "1893-10-04  0.04   NaN\n",
              "1893-10-05  0.00   NaN\n",
              "...          ...   ...\n",
              "2023-09-26  0.00  74.0\n",
              "2023-09-27  0.00  69.0\n",
              "2023-09-28  0.00  73.0\n",
              "2023-09-29  0.00  66.0\n",
              "2023-09-30  0.00  78.0\n",
              "\n",
              "[45971 rows x 2 columns]"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "climate_dataframe = climate_dataframe[['PRCP', 'TOBS']]\n",
        "climate_dataframe"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4550ada7-7e73-40c8-9e3a-e3e26a9891e4",
      "metadata": {},
      "source": [
        "Great work! You finished the data cleaning section of this coding\n",
        "challenge. You can go on to the next notebook – but first, make sure to\n",
        "store the climate `DataFrame` you made so that you can use it in the\n",
        "next notebooks. We’ll do this using the [ipython `store` **cell\n",
        "magic**](https://ipython.readthedocs.io/en/stable/config/extensions/storemagic.html).\n",
        "[Cell magic\n",
        "commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html)\n",
        "aren’t part of your regular code; they’re there to help you with coding\n",
        "in Jupyter Notebooks. You can tell it’s a magic command because it\n",
        "starts with `%`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "a9bb0ec5",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Stored 'climate_dataframe' (DataFrame)\n"
          ]
        }
      ],
      "source": [
        "%store climate_dataframe"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}