01_howto_prepare_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "77129f2a-fae7-4c4e-b99f-23c48e5fd723",
   "metadata": {},
   "source": [
    "# 01. Preparing Weather Station Data for PyMica\n",
    "\n",
    "In this tutorial, we'll cover the preparation of weather station data for use in PyMica.\n",
    "\n",
    "The data format for weather station data used by PyMica is a list containing a dictionary for each weather station, including at least the following variables:\n",
    "\n",
    "- `id`: Identification code.\n",
    "- `lon`: Longitude coordinate.\n",
    "- `lat`: Latitude coordinate.\n",
    "- `value`: Observation value.\n",
    "\n",
    "It can also contain other keys referring to the variables used in interpolation, such as altitude or distance to the coast. Altitude must be named 'altitude'; the names of other explanatory variables do not need to be specific in PyMica.\n",
    "\n",
    "An element of the list containing these variables is organized as follows for each weather station:\n",
    "```\n",
    "{\n",
    "    \"id\": \"id_code\",\n",
    "    \"lon\": \"longitude coordinate value\",\n",
    "    \"lat\": \"latitude coordinate value\",\n",
    "    \"value\": \"value\",\n",
    "    \"altitude\": \"altitude value\"\n",
    "}\n",
    "```\n",
    "\n",
    "The weather station data is supplied to `pymica.pymica.PyMica.interpolate()` as a list of dictionaries, one for each station.\n",
    "\n",
    "As an example, we'll work with data from the Automatic Weather Station Network (XEMA) of the Meteorological Service of Catalonia ([XEMA](https://www.meteo.cat/observacions/xema)). However, you can also provide your own data to PyMica.\n",
    "\n",
    "First, let's import the required library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "34c8afd1-4362-40ac-b8e5-6234a0c2a0fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee8a6016-53d7-4f97-8fcb-301e57732509",
   "metadata": {},
   "source": [
    "Now, let's suppose that our data is in a .csv format. In the `sample-data/data` directory, we'll find data from the XEMA network for 2017/02/21 12:00 UTC and its corresponding metadata.\n",
    "\n",
    "We'll open both .csv files, `XEMA_20170221_1200.csv` and `XEMA_metadata.csv`, using the pandas library and present the head of data file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0034e3e6-5490-4117-91e7-3a546eb27bb8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>altitude</th>\n",
       "      <th>dist</th>\n",
       "      <th>hr</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>temp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C6</td>\n",
       "      <td>264.0</td>\n",
       "      <td>0.858731</td>\n",
       "      <td>80.0</td>\n",
       "      <td>41.65660</td>\n",
       "      <td>0.95172</td>\n",
       "      <td>8.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C7</td>\n",
       "      <td>427.0</td>\n",
       "      <td>0.839116</td>\n",
       "      <td>86.0</td>\n",
       "      <td>41.66695</td>\n",
       "      <td>1.16234</td>\n",
       "      <td>7.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C8</td>\n",
       "      <td>554.0</td>\n",
       "      <td>0.825381</td>\n",
       "      <td>76.0</td>\n",
       "      <td>41.67555</td>\n",
       "      <td>1.29609</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C9</td>\n",
       "      <td>240.0</td>\n",
       "      <td>0.448604</td>\n",
       "      <td>47.0</td>\n",
       "      <td>40.71825</td>\n",
       "      <td>0.39988</td>\n",
       "      <td>15.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CC</td>\n",
       "      <td>626.0</td>\n",
       "      <td>0.849968</td>\n",
       "      <td>47.0</td>\n",
       "      <td>42.07398</td>\n",
       "      <td>2.20862</td>\n",
       "      <td>15.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key  altitude      dist    hr       lat      lon  temp\n",
       "0  C6     264.0  0.858731  80.0  41.65660  0.95172   8.8\n",
       "1  C7     427.0  0.839116  86.0  41.66695  1.16234   7.1\n",
       "2  C8     554.0  0.825381  76.0  41.67555  1.29609   9.3\n",
       "3  C9     240.0  0.448604  47.0  40.71825  0.39988  15.7\n",
       "4  CC     626.0  0.849968  47.0  42.07398  2.20862  15.2"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_data = 'sample-data/data/XEMA_20170221_1200.csv'\n",
    "file_metadata = 'sample-data/data/XEMA_metadata.csv'\n",
    "\n",
    "station_data = pd.read_csv(file_data)\n",
    "metadata = pd.read_csv(file_metadata)\n",
    "\n",
    "station_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c9110a3-04dd-499c-adf0-b8af7e9847ed",
   "metadata": {},
   "source": [
    "And we also present the head of metedata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5e8c1669-b9c4-4e64-9491-c3009032dd24",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>key</th>\n",
       "      <th>altitude</th>\n",
       "      <th>dist</th>\n",
       "      <th>lat</th>\n",
       "      <th>lon</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C6</td>\n",
       "      <td>264.0</td>\n",
       "      <td>0.858731</td>\n",
       "      <td>41.65660</td>\n",
       "      <td>0.95172</td>\n",
       "      <td>Castellnou de Seana</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C7</td>\n",
       "      <td>427.0</td>\n",
       "      <td>0.839116</td>\n",
       "      <td>41.66695</td>\n",
       "      <td>1.16234</td>\n",
       "      <td>Tàrrega</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C8</td>\n",
       "      <td>554.0</td>\n",
       "      <td>0.825381</td>\n",
       "      <td>41.67555</td>\n",
       "      <td>1.29609</td>\n",
       "      <td>Cervera</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C9</td>\n",
       "      <td>240.0</td>\n",
       "      <td>0.448604</td>\n",
       "      <td>40.71825</td>\n",
       "      <td>0.39988</td>\n",
       "      <td>Mas de Barberans</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CC</td>\n",
       "      <td>626.0</td>\n",
       "      <td>0.849968</td>\n",
       "      <td>42.07398</td>\n",
       "      <td>2.20862</td>\n",
       "      <td>Orís</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  key  altitude      dist       lat      lon                 name\n",
       "0  C6     264.0  0.858731  41.65660  0.95172  Castellnou de Seana\n",
       "1  C7     427.0  0.839116  41.66695  1.16234              Tàrrega\n",
       "2  C8     554.0  0.825381  41.67555  1.29609              Cervera\n",
       "3  C9     240.0  0.448604  40.71825  0.39988     Mas de Barberans\n",
       "4  CC     626.0  0.849968  42.07398  2.20862                 Orís"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b3630db-0d24-4a25-8244-20687b9c9b05",
   "metadata": {},
   "source": [
    "Now, let's prepare the data in the format required by PyMICA, selecting the air temperature variable (`temp`) and using `altitude` and `dist` as predictor variables. The variable `dist` refers to the distance from a station to the coastline to account for proximity to sea influence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3304c70-8f61-49c1-ae19-1626d3f5620f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data = []\n",
    "for key in station_data['key']:\n",
    "    df_data = station_data[station_data['key'] == key]\n",
    "    df_meta = metadata[metadata['key'] == key]\n",
    "    data.append(\n",
    "        {\n",
    "            'id': key, \n",
    "            'lon': float(df_meta['lon'].iloc[0]),\n",
    "            'lat': float(df_meta['lat'].iloc[0]),\n",
    "            'value': float(df_data['temp'].iloc[0]),\n",
    "            'altitude': float(df_meta['altitude'].iloc[0]),\n",
    "            'dist': float(df_meta['dist'].iloc[0])\n",
    "        }\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0a64493-208c-4888-938f-35ec26a14547",
   "metadata": {},
   "source": [
    "If we print the first element of `data`, we can see all the required variables for a station, which include identification code, longitude, latitude, temperature value, altitude, and distance to the coastline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed42ecd3-4a1d-4491-bdff-24daf74fa8d3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample data:  {'id': 'C6', 'lon': 0.95172, 'lat': 41.6566, 'value': 8.8, 'altitude': 264.0, 'dist': 0.8587308027349195}\n",
      "Number of points:  180\n"
     ]
    }
   ],
   "source": [
    "print('Sample data: ', data[0])\n",
    "print('Number of points: ', len(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84fbd261",
   "metadata": {},
   "source": [
    "We have now completed this tutorial on how to prepare raw observation station data to be ready to feed the PyMICA class."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}