2-Get_better-at-dask-dataframes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d3f8ffbf-9dc9-42d4-9482-c7fb9e4333a9",
   "metadata": {},
   "source": [
    "<img src=\"images/coiled-logo.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"5%\"\n",
    "     alt=\"Coiled logo\\\">\n",
    "\n",
    "### Sign up for the next live session https://www.coiled.io/tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "036478b0-c89d-4051-a513-f4436f48968f",
   "metadata": {},
   "source": [
    "<img src=\"http://dask.readthedocs.io/en/latest/_images/dask_horizontal.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"30%\"\n",
    "     alt=\"Dask logo\\\">\n",
    "\n",
    "# Get better at Dask Dataframes\n",
    "\n",
    "In this lesson, you will learn the advantages of working with the parquet data format and best practices when working with big data. You will learn how to manipulate inconvenient file sizes and datatypes, as well as how to make your data easier to manipulate. You will be exploring the Uber/Lyft dataset and learning some key practices of feature engineering with Dask Dataframes.\n",
    "\n",
    "## Dask Dataframes \n",
    "\n",
    "<img src=\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"30%\"\n",
    "     alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
    "\n",
    "At its core, the `dask.dataframe` module implements a \"blocked parallel\" `DataFrame` object that looks and feels like the `pandas` API, but for parallel and distributed workflows. One Dask `DataFrame` is comprised of many in-memory pandas `DataFrame`s separated along the index. One operation on a Dask `DataFrame` triggers many pandas operations on the constituent pandas `DataFrame`s in a way that is mindful of potential parallelism and memory constraints.\n",
    "\n",
    "Dask dataframes are very useful, but getting the most out of them can be tricky.  Where your data is stored, the format your data was saved in, the size of each file and the data types, are some examples of things you need to care when it comes to working with dataframes. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4307fa56-f051-467b-bf1f-e3acae08e3a8",
   "metadata": {},
   "source": [
    "### Work close to your data\n",
    "\n",
    "To get started when you are working with data that is in the cloud it's always better to work close to your data to minimize the impact of IO networking. \n",
    "\n",
    "In this lesson, we will use Coiled Clusters that will be created on the same region that our datasets are stored. (the region is `\"us-east-2\"`)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a52c65b-f0bc-4341-9841-4de07f710dce",
   "metadata": {},
   "source": [
    "## Parquet vs CSV\n",
    "\n",
    "Most people are familiarized with **csv** files, but when it comes to working with data, working with **parquet** can make a big difference. \n",
    "\n",
    "### Parquet is where it's at!!\n",
    "\n",
    "The Parquet file format is column-oriented and it is designed to efficiently store and retrieve data. Columnar formats provide better compression and improved performance, and enable you to query data column by column. Consequently, aggregation queries are faster compared to row-oriented storage.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/coiled/dask-tutorial/main/images/storage-files.png\"\n",
    "     align=\"right\"\n",
    "     width=\"50%\"\n",
    "     alt=\"Dask DataFrame is composed of pandas DataFrames\"/>\n",
    "     \n",
    "     \n",
    "- **Column pruning:** Parquet lets you read specific columns from a dataset without reading the entire file.\n",
    "- **Better compression:**  Because in each column the data types are fairly similar, the compression of each column is quite straightforward. (saves on storage)\n",
    "- **Schema:** Parquet stores the file schema in the file metadata.\n",
    "- **Column metadata:** Parquet stores metadata statistics for each column, which can make certain types of queries a lot more efficient.\n",
    "\n",
    "    \n",
    "### Small motivation example: \n",
    "\n",
    "Let's see an example where we compare reading the same data but in one case it is stored as `csv` files, while the other as `parquet` files. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d336f266-3094-45e0-a1f7-f71dfb5f8962",
   "metadata": {},
   "source": [
    "**Note - Windows Users**\n",
    "\n",
    "Unless you are using WSL, you will need to go to a command prompt or PowerShell window within an environment that includes coiled and run the following command from there.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e205e7c7-1d13-4f16-9035-ade902f43fad",
   "metadata": {},
   "outputs": [],
   "source": [
    "### coiled login\n",
    "#!coiled login --token ### --account dask-tutorials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea1905f-09aa-4865-844e-d752639e29c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import coiled\n",
    "import dask\n",
    "import dask.dataframe as dd\n",
    "from dask.distributed import Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1162a7b-969a-46c7-a6b2-bfef92cade59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use this to avoid re-using clusters on a team\n",
    "import uuid\n",
    "\n",
    "id_cluster = uuid.uuid4().hex[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0ac86a4-5336-4c28-98e9-64323e450863",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "cluster = coiled.Cluster(\n",
    "    n_workers=10,\n",
    "    name=f\"nyc-uber-lyft-{id_cluster}\",\n",
    "    account=\"dask-tutorials\",\n",
    "    worker_vm_types=[\"r6i.2xlarge\"],\n",
    "    backend_options={\"region_name\": \"us-east-2\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "254390a1-e055-446c-ab44-c8d76121928d",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(cluster)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c665a6e-650a-4668-9a0c-8ae53c9a795f",
   "metadata": {},
   "outputs": [],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6cb22fca-88e1-4019-9b26-d1f398b176bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# data dictionary\n",
    "data = {\n",
    "    \"5GB-csv\": \"s3://coiled-datasets/h2o-benchmark/N_1e8_K_1e2/*.csv\",\n",
    "    \"5GB-pq\": \"s3://coiled-datasets/h2o-benchmark/N_1e8_K_1e2_parquet/*.parquet\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6829081-0fe8-432b-a126-254ba8139f82",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_csv = dd.read_csv(data[\"5GB-csv\"], storage_options={\"anon\": True})\n",
    "ddf_pq = dd.read_parquet(data[\"5GB-pq\"], storage_options={\"anon\": True})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78de69de-fe56-46e4-b8b7-63936c7de7f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1a9742b-a915-4e7f-8d81-59a127bf9e9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_pq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f938ab3c-30da-416c-87a4-58128c7973b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ddf_csv.groupby(\"id1\").agg({\"v1\": \"sum\"}).compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84a6f2fb-b1ba-4b8d-ade5-be1c2c6f4a93",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ddf_pq.groupby(\"id1\").agg({\"v1\": \"sum\"}).compute()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0530b707-cad2-4b43-8921-1eb821616822",
   "metadata": {},
   "source": [
    "### Memory usage \n",
    "\n",
    "Notice that the `parquet` version without doing much it is already ~5X faster. Let's take a look at the memory usage as well as the `dtypes` in both cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4be1cc4f-4656-4565-bd83-dfec45e613fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "## memory usage for 1 partition\n",
    "ddf_csv.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "011266f8-b9d7-46c4-b88d-574582f5528b",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf_pq.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e323f48-9bbb-46bd-af08-7d6ded1ccead",
   "metadata": {},
   "source": [
    "## Uber/Lyft data transformation\n",
    "\n",
    "In the example above we saw that the format in which the data is stored, already makes a big difference. \n",
    "\n",
    "**Working with parquet** \n",
    "\n",
    "Let's use the Uber/Lyft dataset, as an example of a `parquet` dataset to learn how to troubleshoot the nuances of working with real data. The data comes from [High-Volume For-Hire Services](https://www.nyc.gov/site/tlc/businesses/high-volume-for-hire-services.page)\n",
    "\n",
    "_Data dictionary:_\n",
    "\n",
    "https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_hvfhs.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fda99d96-4b61-4bd8-882b-d7b457279276",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect data\n",
    "import s3fs\n",
    "\n",
    "s3 = s3fs.S3FileSystem()\n",
    "files = s3.glob(\"nyc-tlc/trip data/fhvhv_tripdata_*.parquet\")\n",
    "files[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3af054b4-894c-4ff0-b5f2-e1dd9d117066",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b48f56f-2937-4b6d-8477-7c07721981c9",
   "metadata": {},
   "source": [
    "**Inspect the data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "882652d4-cc6b-454a-be90-d6b8111186ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = dd.read_parquet(\n",
    "    \"s3://nyc-tlc/trip data/fhvhv_tripdata_*.parquet\",\n",
    ")\n",
    "ddf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c371785-d8c3-4882-86fc-0c6c9fd8ce43",
   "metadata": {},
   "source": [
    "## Note:\n",
    "If you are having problems reading the data, it is because even though the data is public, it requires local AWS credentials. We will provide a set of credentials live for this tutorial\"\n",
    "Edit the code above as:\n",
    "\n",
    "```python\n",
    "s3_storage_options = {\"key\": \"***\", \"secret\": \"***\"}\n",
    "\n",
    "ddf = dd.read_parquet(\n",
    "    \"s3://nyc-tlc/trip data/fhvhv_tripdata_*.parquet\",\n",
    "    storage_options=s3_storage_options,\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3727d3f4-3544-4889-8499-d4ea867bbf85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect dtypes\n",
    "ddf.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70ea4e9a-d812-4d50-9f73-c34054b34007",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# inspect memory usage of 1 partition\n",
    "ddf.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0547a9bf-436a-4a49-8cdd-7d79f4c281f0",
   "metadata": {},
   "source": [
    "### Challenges:\n",
    "\n",
    "- Big partitions\n",
    "- Inefficient data types\n",
    "\n",
    "### Recommendations and best practices:\n",
    "\n",
    "**Partition size**\n",
    "\n",
    "In general we recommend starting with partitions that are in the order of ~100MB (in memory). However, the choice of the partition size can vary depending on the worker memory that you have available. \n",
    "\n",
    "For documentation on partition sizes visit the [repartition docs](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.repartition.html) as well as the repartition section in the Dask Dataframe [best practices](https://docs.dask.org/en/stable/dataframe-best-practices.html#repartition-to-reduce-overhead)\n",
    "\n",
    "**Data Types**\n",
    "\n",
    "- Avoid object types for strings: use `\"string[pyarrow]\"`\n",
    "- Reduce int/float representation if possible\n",
    "- Use categorical dtypes when possible (avoid high cardinality).\n",
    "- Consider using Nullable dtypes (very new/experimental)\n",
    "\n",
    "### Create conversions dictionary\n",
    "\n",
    "Based on these recommendations, let's work on better `dtypes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac85d0af-b961-4fff-be71-b14b60cd52bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b76f77fb-67bd-4c3e-a6aa-3698365a0dd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.config.set({\"dataframe.convert-string\": True})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b60dc7af-5906-42c3-8371-c20153e00394",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = dd.read_parquet(\n",
    "    \"s3://nyc-tlc/trip data/fhvhv_tripdata_*.parquet\",\n",
    ")\n",
    "ddf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b606cb75-32f6-450b-9126-120faad85dff",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9bccadb-6f7a-4dba-a4fd-113bd96fbbc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "conversions = {}\n",
    "for column, dtype in ddf.dtypes.items():\n",
    "    if dtype == \"float64\":\n",
    "        conversions[column] = \"float32\"\n",
    "    if dtype == \"int64\":\n",
    "        conversions[column] = \"int32\"\n",
    "    if \"flag\" in column:\n",
    "        conversions[column] = pd.CategoricalDtype(categories=[\"Y\", \"N\"])\n",
    "    if column == \"airport_fee\":\n",
    "        conversions[\n",
    "            column\n",
    "        ] = \"float32\"  # noticed that this has floats and the <NA> is making it an object\n",
    "conversions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53645d0d-8cd9-4fe8-892a-b4c11f2b787d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use new dtypes this takes a bit of time\n",
    "ddf = ddf.astype(conversions)\n",
    "ddf = ddf.persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eed8a8ed-abb8-4fa4-8686-3a7a68bd28df",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.partitions[0].memory_usage(deep=True).compute().apply(dask.utils.format_bytes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5970a21-6c8c-40dd-869e-07d998e7c2e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.utils.format_bytes(ddf.partitions[0].memory_usage(deep=True).compute().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec42ac57-72fd-4ff3-8416-dcdb287e25b7",
   "metadata": {},
   "source": [
    "### Repartition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4815f73c-0ae4-4b8b-84f2-1750cbaa969a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf = ddf.repartition(partition_size=\"128MB\").persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6c13761-4906-47a6-b138-1ec5875e1242",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.utils.format_bytes(ddf.memory_usage(deep=True).compute().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "329c807b-1dfe-4173-9d34-73bc8fc38695",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.npartitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "589ca6bf-5bba-4444-8ca2-671f45dec7c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dask.utils.format_bytes(ddf.partitions[0].memory_usage(deep=True).compute().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c3fa314-86a2-4343-9a9a-c5b368e6a67a",
   "metadata": {},
   "source": [
    "### Other repartition options \n",
    "\n",
    "Sometimes, a repartition by size is not convenient for your use case. You can also repartition on a period of time if you have a timeseries with a datetime index. For example: if you where to need your data partition every `1d` you can do:\n",
    "\n",
    "```python\n",
    "ddf = ddf.set_index(\"request_datetime\")\n",
    "ddf = ddf.repartition(freq=\"1d\")\n",
    "```\n",
    "\n",
    "**Note:**\n",
    "Read more about repartition in the [dask documentation on this feature](https://docs.dask.org/en/stable/generated/dask.dataframe.DataFrame.repartition.html#dask-dataframe-dataframe-repartition)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9603233c-7e0f-41bf-aa70-84c4d55a7f12",
   "metadata": {},
   "source": [
    "### Save data to and S3 bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56a3cf0e-edde-4979-b892-5149d0dbccf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creds to be provided in live.\n",
    "s3_storage_options = {\"key\": \"***\", \"secret\": \"***\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ca7ac61-9346-4ff9-82bc-4d8024e22ea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "usr_id = \"your_name\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0d899c-2155-4872-ade7-d80a19f5190e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ddf.to_parquet(\n",
    "    f\"s3://dask-tutorials-datasets/{usr_id}/\",\n",
    "    storage_options=s3_storage_options,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c4291e8-0dce-467b-95ce-badd2513b10c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.shutdown()\n",
    "client.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beeff184-6c3d-44bb-86f3-f071c1ab5781",
   "metadata": {},
   "source": [
    "## Let's do some data analysis\n",
    "\n",
    "Now we are at a stage that our whole dataset is ~80GB in memory. When it comes to exploring data we do not necessarily need the whole data set, we can work with a sample, as well as only select a subset of columns. One of the beauties of the parquet file format is **column pruning**\n",
    "\n",
    "Note: Keep in mind, that if you will do feature engineering, your data size will increase and having extra memory can help."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "287e76d2-04ce-4f5c-a620-39017f8b05a0",
   "metadata": {},
   "source": [
    "### Read data back\n",
    "\n",
    "After you save your data, you will want to read it back to do some data analysis or train a model. When reading data back, there are some caveats regarding the `dtypes`.\n",
    "\n",
    "- **Roundtriping for string pyarrow dtype** is not yet supported in pandas/dask. Hence when you read your data you need to tell pandas/dask to cast those columns as \"string[pyarrow]\" otherwise they'll be \"string[python]\". \n",
    "- **Nullable dtypes:** Using nullable dtypes is a fairly new feature and still under development, consider this experimental. Available in `dask >= 2022.12.0`\n",
    "\n",
    "**What are nullable dtypes?**\n",
    "\n",
    "Pandas (hence Dask) primarily uses NaN to represent missing data. Because NaN is a float, this forces an array of integers with any missing values to become floating point. In some cases, this may not matter much. But if your integer column is, say, an identifier, casting to float can be problematic. \n",
    "\n",
    "Nullable dtypes, allow you to work around this issue. \n",
    "\n",
    "If you want to read more about nullable dtypes, check the pandas [missing data docs](\n",
    "https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#missing-data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00a71e42-785c-4d44-9ece-da7f138ff64c",
   "metadata": {},
   "source": [
    "NOTE: \n",
    "1. If you are in a live session you will be able to read the parquet files we stored, providing the credentials that we share with you live. \n",
    "2. If you are following this tutorial on your own the credentials will not work, but you can read a copy of the dataset we wrote, from `\"s3://coiled-datasets/uber-lyft-tlc/\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3476887b-5643-4e4a-9cdb-d2488692cdf7",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "cluster = coiled.Cluster(\n",
    "    name=f\"uber-lyft-{id_cluster}\",\n",
    "    n_workers=15,\n",
    "    account=\"dask-tutorials\",\n",
    "    worker_vm_types=[\"m6i.xlarge\"],\n",
    "    backend_options={\"region_name\": \"us-east-2\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "304ba45b-5604-4bdd-8a33-7433cd9df26d",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Client(cluster)\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d5f536a-2f94-406a-8716-91d43ebf1e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use bucket where you wrote above if you are following from public session\n",
    "# or public data uri (\"s3://coiled-datasets/uber-lyft-tlc/\") otherwise\n",
    "file_to_read = (\n",
    "    f\"s3://dask-tutorials-datasets/{usr_id}/\"  # replace for public uri if needed\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15530dc1-703e-4491-9eec-c391cd663a34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# if reading form public uri and in binder use storage_options=storage_options={'anon': True}\n",
    "df = dd.read_parquet(\n",
    "   file_to_read,    # replace for \"s3://coiled-datasets/uber-lyft-tlc/\" if needed\n",
    "   storage_options = s3_storage_options,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e2900ba-cbdf-46fc-9ee4-9fbc69ea881c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80eb7dc4-e84e-40b9-9842-dad6670dad3b",
   "metadata": {},
   "source": [
    "## Memory usage \n",
    "\n",
    "```python\n",
    "dask.utils.format_bytes(\n",
    "    df.memory_usage(deep=True).sum().compute()\n",
    ")\n",
    "```\n",
    "'82.81 GiB'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "834816b0-5899-43a8-853e-ffc91a0a7da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dff7b58b-97c5-4e06-9fdf-f2fe4fc979a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d52ca6d7-4030-4103-b128-4263f95044c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# len(df)\n",
    "## 783_431_901"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1262fb9-adaa-4e23-9b5f-7e50e62128ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "##let's count to see NaN\n",
    "df.count().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb55c7c8-bb9b-4fe5-9729-6b91f263a3ac",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a column tip > 0 = True\n",
    "df[\"tip_flag\"] = df.tips > 0\n",
    "\n",
    "df_small = df[\n",
    "    [\n",
    "        \"hvfhs_license_num\",\n",
    "        \"tips\",\n",
    "        \"base_passenger_fare\",\n",
    "        \"driver_pay\",\n",
    "        \"trip_miles\",\n",
    "        \"trip_time\",\n",
    "        \"shared_request_flag\",\n",
    "        \"tip_flag\",\n",
    "    ]\n",
    "].persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54707176-6ddf-4c3c-9870-7cdfd03f8335",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb809b7b-9cdf-4d54-b191-305a3c89eb97",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.base_passenger_fare.sum().compute() / 1e9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "286fc4f5-003b-47a9-8a56-fc227141e670",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.driver_pay.sum().compute() / 1e9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ca23fa1-ab0c-4426-8165-7742bcae7891",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.tips.sum().compute() / 1e6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fcaad19-5ad8-4ce8-a5e6-669a802c37e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc2110ff-d2fe-4a57-8556-f8381cd12d48",
   "metadata": {},
   "source": [
    "### Are New Yorkers tippers? \n",
    "\n",
    "Let's see how many trips have tip by provider "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac3192f3-7913-41e6-9b7b-ed01bb8a96c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_counts = df_small.groupby([\"hvfhs_license_num\"]).tip_flag.value_counts().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13062d36-4168-41b7-b2c5-d6c8e62de44e",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fac3870d-0b82-47e9-856b-e15e77cf3425",
   "metadata": {
    "tags": []
   },
   "source": [
    "**From the data dictionary we know:**\n",
    "\n",
    "As of September 2019, the HVFHS licenses are the following:\n",
    "\n",
    "- HV0002: Juno  \n",
    "- HV0003: Uber  \n",
    "- HV0004: Via  \n",
    "- HV0005: Lyft  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28c54f0e-0526-447e-8f5c-19fc6e6b1962",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(tip_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd1d5b68-5e2e-487d-9db7-b1bc0c5b8e2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "## this is a pandas\n",
    "tip_counts = tip_counts.unstack(level=\"tip_flag\")\n",
    "tip_counts / 1e6"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e01315d8-0a69-47d1-a443-64bfa5d84aa7",
   "metadata": {},
   "source": [
    "### Percentage of total rides that tip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "554de773-2424-44c5-b711-d1d4e4a6bad6",
   "metadata": {},
   "outputs": [],
   "source": [
    "tip_counts[True] * 100 / (tip_counts[True] + tip_counts[False])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c42ee9b-b7db-4fde-8125-84c5aca7483c",
   "metadata": {},
   "source": [
    "### sum and mean of tips by provider "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b10f437-ba9d-41d3-bc43-0d4e96e9dcdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_total = (\n",
    "    df_small.loc[lambda x: x.tip_flag]\n",
    "    .groupby(\"hvfhs_license_num\")\n",
    "    .tips.agg([\"sum\", \"mean\"])\n",
    "    .compute()\n",
    ")\n",
    "tips_total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef8c80f8-0d20-4bf4-a4ee-fa4a5325ce26",
   "metadata": {},
   "outputs": [],
   "source": [
    "provider = {\"HV0002\": \"Juno\", \"HV0005\": \"Lyft\", \"HV0003\": \"Uber\", \"HV0004\": \"Via\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "776e2689-133d-4cfb-bd08-0ca34ee05f31",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_total = tips_total.assign(provider=lambda df: df.index.map(provider)).set_index(\n",
    "    \"provider\"\n",
    ")\n",
    "tips_total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63f9aad-c1e0-49c6-9de2-8d99e4bd44e2",
   "metadata": {},
   "source": [
    "### What percentage of the passenger fare is the tip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d089e27-053a-4f8c-820a-0fedb4d232ea",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "- Create a new column named \"tip_percentage\" that represents the what fraction of the passenger fare is the tip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cec9faaf-f329-4ebb-8e60-d1fe95d22f2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "tip_percentage = df_small.tips / df_small.base_passenger_fare\n",
    "df_small[\"tip_percentage\"] = tip_percentage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b14e09d-7a4b-454f-ab39-12cbcd5d7ef4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small = df_small.persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c5aaa54-f344-464e-90af-4270c1792528",
   "metadata": {},
   "source": [
    "## Tip percentage mean of trip with tip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ab6fa5d-1b7b-4cd6-aa02-f5986784cd49",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_perc_mean = (\n",
    "    df_small.loc[lambda x: x.tip_flag]\n",
    "    .groupby(\"hvfhs_license_num\")\n",
    "    .tip_percentage.mean()\n",
    "    .compute()\n",
    ")\n",
    "tips_perc_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c16d4260-46a4-4a5a-842d-b80f3e0cc76b",
   "metadata": {},
   "outputs": [],
   "source": [
    "(tips_perc_mean.to_frame().set_index(tips_perc_mean.index.map(provider)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42ba5d51-9537-47ad-86db-3aa7a98a885a",
   "metadata": {},
   "source": [
    "### Base pay per mile per - by provider\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70eaed48-4b26-48ce-8036-fc26f4e13bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "dollars_per_mile = df_small.base_passenger_fare / df_small.trip_miles\n",
    "df_small[\"dollars_per_mile\"] = dollars_per_mile\n",
    "df_small = df_small.persist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb6f2544-c41f-45dc-8f8a-f3863f369044",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    df_small.groupby(\"hvfhs_license_num\")\n",
    "    .dollars_per_mile.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    "    .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "910754f2-478c-4eeb-99a5-8ba4b2cf5d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter: check only trips with tip\n",
    "(\n",
    "    df_small.loc[lambda x: x.tip_flag]\n",
    "    .groupby(\"hvfhs_license_num\")\n",
    "    .dollars_per_mile.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    "    .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5508b41a-e186-4af6-8a35-2a659c3a6975",
   "metadata": {},
   "source": [
    "### Get insight on the data\n",
    "\n",
    "We are seeing weird numbers, let's try to take a deeper look and remove some outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15df0aab-84fb-45bc-af6d-d2f9892b38be",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(\n",
    "    df_small[[\"trip_miles\", \"base_passenger_fare\", \"tips\", \"tip_flag\"]]\n",
    "    .loc[lambda x: x.tip_flag]\n",
    "    .describe()\n",
    "    .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e970925-c96d-40e5-a906-176a6f69a44c",
   "metadata": {},
   "source": [
    "### Getting to know the data\n",
    "\n",
    "- How would you get more insights on the data?\n",
    "- Can you visualize it?\n",
    "\n",
    "**Hint:** Get a small sample, like 0.1% of the data to plot ~700_000 rows (go smaller if needed depending on your machine), compute it and work with that pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21a73002-ab6b-4627-b5af-7ecdeedd0d78",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# needed to avoid plots from breaking\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1abd1c5-8d60-4772-b1a9-4a180790c0d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Take a sample\n",
    "df_tiny = (\n",
    "    df_small.loc[lambda x: x.tip_flag][[\"trip_miles\", \"base_passenger_fare\", \"tips\"]]\n",
    "    .sample(frac=0.001)\n",
    "    .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a46b9993-3009-4e5d-a0e4-2c8368ec0613",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# box plot\n",
    "df_tiny.boxplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "500530b7-0f24-4566-b234-1f1a5acc2cc3",
   "metadata": {},
   "source": [
    "### Cleaning up outliers\n",
    "\n",
    "- Play with the pandas dataframe `df_tiny` to get insights on good filters for the bigger dataframe. \n",
    "\n",
    "Hint: think about pandas dataframe quantiles [docs here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.quantile.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d674a25-104a-4506-8602-25e214023954",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_tiny.tips.quantile([0.25, 0.75])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d552cdcf-2ea9-4443-a9a5-b55523e33e85",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "- Calculate the first and third quantiles for `base_passenger_fare` and `trip_miles`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "905ed15c-106a-42eb-bfee-59f6d12b84d6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# solution\n",
    "df_tiny.base_passenger_fare.quantile([0.25, 0.75])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f08228ab-5f07-460a-b268-d176e77aef94",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# solution\n",
    "df_tiny.trip_miles.quantile([0.25, 0.75])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0ee5c4c-3423-443c-8ce9-d0c597aa0758",
   "metadata": {},
   "source": [
    "### Conditions to filter the dataset\n",
    "\n",
    "We can use the information of Q1 and Q3 to create contions to filter the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fcd0e94-2c7a-423d-a410-e43eedecb3f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_filter_vals = df_tiny.tips.quantile([0.25, 0.75]).values\n",
    "tips_condition = df_tiny.tips.between(*tips_filter_vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1133eddf-b478-4f61-9889-3f41e9050689",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "tips_condition"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da2dfb9b-af06-4183-8ea5-a30dc02bdd5f",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "- Create filter conditions for the `base_passenger_fare` and `trip_miles`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89d5bdc4-1685-471d-898e-9ed239ad281b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Solution\n",
    "fare_filter_vals = df_tiny.base_passenger_fare.quantile([0.25, 0.75]).values\n",
    "fares_condition = df_tiny.base_passenger_fare.between(*fare_filter_vals)\n",
    "\n",
    "miles_filter_vals = df_tiny.trip_miles.quantile([0.25, 0.75]).values\n",
    "miles_condition = df_tiny.trip_miles.between(*miles_filter_vals)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dacba7f-bb18-4027-99e2-43a7c91ebd21",
   "metadata": {},
   "source": [
    "### Filter dataframe and plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a82012b-ee3a-4df9-b24b-877f114a207a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# solution\n",
    "df_tiny.loc[(tips_condition & fares_condition) & miles_condition].boxplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1337177-5c12-4f73-bc68-f9156c1bbc6c",
   "metadata": {},
   "source": [
    "## Filtering our big dataset based on the insights\n",
    "\n",
    "Based on these numbers let's go back to our `df_small` dataset and try to filter it.\n",
    "\n",
    "**Note:**\n",
    "\n",
    "Sometimes when you are trying to filter and you have been doing feature engineering, you might get a divisions not known error.\n",
    "If that's the case you can do \n",
    "\n",
    "```python\n",
    "df_small = df_small.reset_index()\n",
    "df_small = (df_small\n",
    "            .set_index(\"column_to_be_the_index\")\n",
    "            .persist()\n",
    "           )\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75a5a9be-5e13-4f0d-8c76-13ab7caea857",
   "metadata": {},
   "outputs": [],
   "source": [
    "tips_condition = df_small.tips.between(*tips_filter_vals)\n",
    "miles_condition = df_small.trip_miles.between(*miles_filter_vals)\n",
    "fares_condition = df_small.base_passenger_fare.between(*fare_filter_vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "879314c1-b9de-483f-af5d-59347db07f61",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_small = df_small.loc[(tips_condition & fares_condition) & miles_condition].persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39de8d6c-b94f-4b26-9f5e-5758f37efe1c",
   "metadata": {},
   "source": [
    "### Stats on `dollars_per_mile`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e33c8db5-e8f1-4559-bc02-deb716e19fcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    df_small.groupby(\"hvfhs_license_num\")\n",
    "    .dollars_per_mile.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    "    .compute()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4933836-e833-4787-b561-f2d0bb5f76ee",
   "metadata": {},
   "source": [
    "### Let's look at the `tip_percentage` again"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aded0737-d5bf-4181-b9aa-2c5c72435d06",
   "metadata": {},
   "source": [
    "### Exercise \n",
    "- Compute the `tip_percentage` mean by provider "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e745175-5725-4b92-a58d-4f870a3d3ace",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Solution\n",
    "tips_perc_avg = df_small.groupby(\"hvfhs_license_num\").tip_percentage.mean().compute()\n",
    "tips_perc_avg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46be3c7a-8128-4ab1-b8f1-3b398faa4238",
   "metadata": {},
   "outputs": [],
   "source": [
    "(tips_perc_avg.to_frame().set_index(tips_perc_avg.index.map(provider)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1538a243-8de1-4dde-82f0-dd5434b21f11",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3665fc2-8b8e-4e0e-a656-e47cefbafb86",
   "metadata": {},
   "source": [
    "### Average trip time by provider"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "176f6aa4-8eec-4d4b-9959-678fa61d50ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trips_time_avg = (\n",
    "    df_small.groupby(\"hvfhs_license_num\")\n",
    "    .trip_time.agg([\"min\", \"max\", \"mean\", \"std\"])\n",
    "    .compute()\n",
    ")\n",
    "trips_time_avg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50ade8e4-2fd4-4cf4-b533-033c694ae04d",
   "metadata": {},
   "source": [
    "### In minutes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3715c5a-7520-42c3-a41c-c1295a19d5ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trips_time_avg.set_index(trips_time_avg.index.map(provider)) / 60"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf7fdf77-320b-4d87-802c-1f260867a141",
   "metadata": {},
   "source": [
    "## What we've learned\n",
    "- Most New Yorkers do not tip\n",
    "- But it looks like of those who tip, it is common to tip around 20% regardless of the provider. Unless it's Via, they tend to tip slightly less.\n",
    "- The trip_time column needs some cleaning of outliers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "451a0ec9-fc40-4f23-bd8a-69803f0a6fe2",
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster.shutdown()\n",
    "client.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c592b6f-6386-4e82-9cb2-b74637846a22",
   "metadata": {},
   "source": [
    "### Useful links\n",
    "\n",
    "- https://tutorial.dask.org/01_dataframe.html\n",
    "\n",
    "**Useful links**\n",
    "\n",
    "* [DataFrames documentation](https://docs.dask.org/en/stable/dataframe.html)\n",
    "* [Dataframes and parquet](https://docs.dask.org/en/stable/dataframe-parquet.html)\n",
    "* [Dataframes examples](https://examples.dask.org/dataframe.html)\n",
    "\n",
    "### Other lesson\n",
    "\n",
    "Register [here](https://www.coiled.io/tutorials) for reminders. \n",
    "\n",
    "We have another lesson, where we’ll parallelize a custom Python workflow that scrapes, parses, and cleans data from Stack Overflow. We’ll get to: ‍\n",
    "\n",
    "- Learn how to do arbitrary task scheduling using the Dask Futures API\n",
    "- Utilize blocking and non-blocking distributed calculations\n",
    "\n",
    "By the end, we’ll see how much faster this workflow is using Dask and how the Dask Futures API is particularly well-suited for this type of fine-grained execution.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}