Hands_On_Data_Processing_Visualization_2023.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gustavmarco/teaching/blob/main/Hands_On_Data_Processing_Visualization_2023.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8rr3y-u3UpCT"
      },
      "source": [
        "![analysis.png](https://cdn-icons-png.flaticon.com/512/2318/2318736.png)\n",
        "\n",
        "#**Hands On Session**\n",
        "## **Data preprocessing, visualization**\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Introduction to cBioPortal**"
      ],
      "metadata": {
        "id": "rXlCgzfWBqUN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "![cbioportal](https://frontend.cbioportal.org/reactapp/images/369b022222badf37b2b0c284f4ae2284.png)\n",
        "\n",
        "The cBioPortal is an exploratory analysis tool for exploring large-scale cancer genomic data sets that hosts data from large consortium efforts (like TCGA), as well as publications from individual labs. You can find and explore the website here: https://www.cbioportal.org/\n"
      ],
      "metadata": {
        "id": "d-4hL6j3GHH8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 1\n",
        "Explore the cBioPortal database with it's tools and possibilities. When finished, download the dataset that we will work with in the following.\n",
        "\n"
      ],
      "metadata": {
        "id": "7VIMGKAPELWl"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GJBs_flRovLc"
      },
      "source": [
        "## **Working with tabular data**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "![pandas](https://miro.medium.com/max/962/1*n_ms1q5YoHAQXXUIfeADKQ.png)\n",
        "\n",
        "When working with tabular data, a very convenient tool in Python is the Pandas library. We use Pandas on a daily basis to get comprehensive information about our dataset. You can do various operations and analyses on your tabular data and dive deep into the properties and statistics of your data. You can extract different subsets from your dataset, store them, plot them and analyze them. We will work with the data you downloaded before "
      ],
      "metadata": {
        "id": "-xcA6VYbI2Tl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import all libraries we will need for this notebook\n",
        "import io\n",
        "import pandas as pd\n",
        "from google.colab import files"
      ],
      "metadata": {
        "id": "0iCR7lEWUHsN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 2\n",
        "Import the file you downloaded from cBioPortal (as .csv or .tsv). The file formats are very similar but the difference is that the values in the tables are: **Comma**-Separated Values for .**c**sv and **Tab**-Separated Values for .**t**sv. When importing the data the seperator (sep) in the pd.read_csv function has to be set accordingly (',' or '\\t')."
      ],
      "metadata": {
        "id": "-x0F0TrLHbvN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Upload file to colab so that we can work with them - choose file from menu\n",
        "uploaded = files.upload()\n",
        "# Store file content to dataframe called 'dataframe_orig' via pd.read_csv\n",
        "dataframe_orig = pd.read_csv(io.BytesIO(uploaded['coadread_tcga_pan_can_atlas_2018_clinical_data.tsv']), sep='\\t')"
      ],
      "metadata": {
        "id": "890Xyr04Ty3a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 3\n",
        "In this part we will explore the different options to gather information about our dataframe. You will learn how to use functions like:\n",
        "- .info()\n",
        "- .describe()\n",
        "- .head()\n",
        "- .tail()\n",
        "- .value_counts()\n",
        "- .unique()\n",
        "\n",
        "In Pandas, indexing a DataFrame returns a reference to the original DataFrame. Thus, changing the subset will change the initial DataFrame. Thus, you'd want to use the copy if you want to make sure the initial DataFrame shouldn't change.To keep the original DataFrame, we use the .copy() method on the original DataFrame so that we don not manipulate the original DataFrame."
      ],
      "metadata": {
        "id": "0yFoJmj5Ikmv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display dataframe\n",
        "dataframe = dataframe_orig.copy()\n",
        "dataframe"
      ],
      "metadata": {
        "id": "rvZ-zeuBR7Kh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display general information about the dataframe using .info(), including non-null counts\n",
        "dataframe.info()"
      ],
      "metadata": {
        "id": "hQb2GFLGU4Vf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display statistical information about the DataFrame using .describe(), only for numerical values\n",
        "dataframe.describe()"
      ],
      "metadata": {
        "id": "QeYrEwCJWHoQ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display statistical information about the DataFrame using .describe(), for all values\n",
        "dataframe.describe(include='all')"
      ],
      "metadata": {
        "id": "bQtzaXTBWZYC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the first 10 lines as an overview using head(10)\n",
        "dataframe.head(10)"
      ],
      "metadata": {
        "id": "BcqiuAaibk0J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the last 5 lines as an overview using head(10)\n",
        "dataframe.tail(5)"
      ],
      "metadata": {
        "id": "EmMdTYH2RaNn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display a column of choice, in this case 'Cancer Type Detailed'\n",
        "dataframe['Cancer Type Detailed']"
      ],
      "metadata": {
        "id": "mKCClIe-dKUv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display frequency of value occurrence for column of choice\n",
        "dataframe['Overall Survival Status'].value_counts()"
      ],
      "metadata": {
        "id": "AupQKEFZI1sq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate unique elements in a column of choice: in this case in the column named 'Cancer Type Detailed'\n",
        "unique_elements = dataframe['Cancer Type Detailed'].unique()\n",
        "\n",
        "# Display the unique elements in this column\n",
        "print(f'Unique class names in column \"Cancer Type Detailed\": \\n', unique_elements, '\\n')\n",
        "\n",
        "# Display the unique elements in this column, sorted alphabetically\n",
        "print(f'Unique class names in column \"Cancer Type Detailed\": \\n', sorted(unique_elements), '\\n')\n",
        "\n",
        "# Display how many unique elements are in this column\n",
        "print(f'Unique classes in column \"Cancer Type Detailed\": ', len(unique_elements))"
      ],
      "metadata": {
        "id": "vkT_S1nZdSzk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 4\n",
        "In this part we will explore sorting functions of the dataframe."
      ],
      "metadata": {
        "id": "jUVCIVNhNjM-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display dataframe\n",
        "dataframe"
      ],
      "metadata": {
        "id": "_4aYxq4QaCvm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the dataframe sorted for the column 'Mutation Count', with the highest value on top\n",
        "dataframe.sort_values(by='Mutation Count', ascending=False)"
      ],
      "metadata": {
        "id": "A6NhHi_KdXsz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Check if the sorting of the dataframe is saved to the dataframe\n",
        "dataframe"
      ],
      "metadata": {
        "id": "m8lkJh-qNs_P"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Sort dataframe for multiple columns\n",
        "dataframe.sort_values(['Overall Survival Status', 'Diagnosis Age'], ascending = [True, False])"
      ],
      "metadata": {
        "id": "EgeOtWYAaRf8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Sort dataframe for multiple columns AND save changes to dataframe by setting 'inplace' argument to 'True'\n",
        "dataframe_sorted = dataframe.sort_values(['Overall Survival Status', 'Diagnosis Age'], ascending = [True, False])"
      ],
      "metadata": {
        "id": "wFOq3WuTbroA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Check if the sorting of the dataframe is saved to the new dataframe\n",
        "dataframe_sorted"
      ],
      "metadata": {
        "id": "q_w_LQKKbgvi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# You can also apply the sorting to your basic dataframe by using the 'inplace' argument\n",
        "dataframe.sort_values(['Overall Survival Status', 'Diagnosis Age'], ascending = [True, False], inplace=True)"
      ],
      "metadata": {
        "id": "vTSEAF7qN8ck"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Check if the sorting of the dataframe is saved to the dataframe\n",
        "dataframe"
      ],
      "metadata": {
        "id": "b4_q_dFhOH1J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 5\n",
        "In this part we will explore basic visualization functionalities for DataFrames. First we are interested in visualizing the distribution of values/classes in each column with a histogram. Second we want to visualize how many values are missing in each column. We will need this information for further investigation and creation of a new dataset."
      ],
      "metadata": {
        "id": "QcKmMdjnOKUj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display dataframe\n",
        "dataframe"
      ],
      "metadata": {
        "id": "OKNZYfgDSwc_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import plotting library\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# To avoid an error regarding the value's format we set the type of entries in column 'Overall Survival Status' to string\n",
        "dataframe['Overall Survival Status'] = dataframe['Overall Survival Status'].astype(str)\n",
        "\n",
        "# Create a histogram of the data distribution for each column in the dataset\n",
        "# Loop through all columns in dataframe\n",
        "for col in dataframe.columns:\n",
        "  # Print some characteristics of dataframe and compare the min and max values to the plots (does it look correct?)\n",
        "  print(col, type(dataframe[col][0]), dataframe[col].min(), dataframe[col].max())\n",
        "  # create a separate figure for each plot\n",
        "  plt.figure()\n",
        "  # The label below the x axis shall contain the column name\n",
        "  plt.xlabel(str(col))\n",
        "  # Plot the histogram for each column; horizontal orientation leads to better readability; bins are the maximum amount of bars into which the data is summarized \n",
        "  plt.hist(dataframe[col], orientation='horizontal', bins=100)"
      ],
      "metadata": {
        "id": "Ht7-SZjgV56p"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display dataframe\n",
        "dataframe"
      ],
      "metadata": {
        "id": "bCbLIRitLk3f"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a dataframe containing information on missing values (Nan's) -> will be shown as 'True' in resulting dataframe\n",
        "dataframe_nans = dataframe.isna()\n",
        "display(dataframe_nans)\n",
        "\n",
        "# Sum 'True's column wise (axis=0) to get amount of NaN's per column\n",
        "dataframe_nans = dataframe_nans.sum(axis=0)\n",
        "print('\\n Frequency of occurrence of NaN per column:')\n",
        "display(dataframe_nans)\n",
        "\n",
        "# Sort data beginning with highest frequence of NaN's \n",
        "dataframe_nans = dataframe_nans.sort_values(ascending=False)\n",
        "print('\\n Sorted frequency of occurrence of NaN per column:')\n",
        "display(dataframe_nans)"
      ],
      "metadata": {
        "id": "L4xs5DRcZSvV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a bar diagram of the frequency of NaN's for each column in the dataset\n",
        "# Create a separate figure for each plot\n",
        "plt.figure()\n",
        "# The label below the x axis\n",
        "plt.xlabel('NaN Frequency of Occurence per Column')\n",
        "# Plot the horizontal bar chart (bar'h'); horizontal orientation leads to better readability; bins are the maximum amount of bars into which the data is summarized \n",
        "plt.barh(dataframe_nans.index, dataframe_nans.values)"
      ],
      "metadata": {
        "id": "PiweQy3IPjR9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 5\n",
        "Extract a sub-dataframe from the original based on predefined conditions and save as a separate file.\n",
        "\n",
        "Conditions:\n",
        "1.   delete columns that contain more than 20 missing values (NaN's)\n",
        "2.   only include rows with 'Overall Survival Status' = '0:LIVING'\n",
        "3.   only include rows with 'Diagnosis Age' > 50"
      ],
      "metadata": {
        "id": "rnvM7SGi9oEW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Display your input dataframe\n",
        "dataframe"
      ],
      "metadata": {
        "id": "rFJhzYyqdZu3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 1.\n",
        "# Count the number of NaN values in each column\n",
        "sum_nans = dataframe.isna().sum(axis=0)\n",
        "display(sum_nans)\n",
        "\n",
        "# Filter out columns with more than 20 NaN values\n",
        "display(sum_nans > 20)\n",
        "cols_to_drop = sum_nans[sum_nans > 20].index\n",
        "print(f'The following columns will be dropped: {cols_to_drop.values}')\n",
        "# Drop the columns\n",
        "dataframe_nans = dataframe.drop(columns=cols_to_drop)\n",
        "display(dataframe_nans)"
      ],
      "metadata": {
        "id": "NI5dWG5aAx5k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 2. & 3.\n",
        "# Only include rows with 'Overall Survival Status' = '0:LIVING' AND 'Diagnosis Age' > 50\n",
        "dataframe_filtered = dataframe_nans[(dataframe_nans['Overall Survival Status'] == '0:LIVING') & (dataframe_nans['Diagnosis Age'] > 50)]\n",
        "display(dataframe_filtered)\n",
        "# test if it worked\n",
        "print(f'Test for unique classes in \"Overall Survival Status\": {dataframe_filtered[\"Overall Survival Status\"].unique()}')\n",
        "print(f'Test for minimum age in \"Diagnosis Age\": {dataframe_filtered[\"Diagnosis Age\"].min()}')"
      ],
      "metadata": {
        "id": "uPLwbyYDDiRW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 6\n",
        "Compare the unprocessed vs. the processed DataFrame regarding their dimensions."
      ],
      "metadata": {
        "id": "W8cRHiwiQrgx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Read out the number of columns and rows of the original data set using 'shape'\n",
        "num_rows, num_cols = dataframe.shape\n",
        "print(f'Dimensions before removing columns: {num_rows} rows, {num_cols} columns')\n",
        "# Read out the number of columns and rows of the processed data set using 'shape'\n",
        "num_rows, num_cols = dataframe_filtered.shape\n",
        "print(f'Dimensions after removing columns: {num_rows} rows, {num_cols} columns')"
      ],
      "metadata": {
        "id": "opcTTKdZFxZx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###⌨ Task 7\n",
        "Save your newly created DataFrame 'dataframe_filtered' to a separate file."
      ],
      "metadata": {
        "id": "q3YeMkvvQ3hH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Save preprocessed dataset as a separate file on your local drive\n",
        "dataframe_filtered.to_csv('dataframe_filtered.csv', sep=',', index=False) \n",
        "files.download('dataframe_filtered.csv')\n",
        "print('Preprocessed dataset saved as .csv to download folder.')"
      ],
      "metadata": {
        "id": "VhwNGpwvYgjE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "dcRtxymoKWZQ"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    },
    "colab": {
      "provenance": [],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}