B_03_mobility_metrics_on_user_dicts.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "01066def",
   "metadata": {},
   "outputs": [],
   "source": [
    "import skmob\n",
    "import statistics\n",
    "import pandas as pd\n",
    "import geopandas as gpd\n",
    "import backend_codes.load_subsets as ls\n",
    "import skmob.measures.individual as sk_id\n",
    "from skmob.measures.collective import random_location_entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bfb3bfc",
   "metadata": {},
   "source": [
    "# Calculating user dict based mobility indictors\n",
    "User dicts are derived via the skmob package.\n",
    "\n",
    "For distance based statistics, we use the EPSG:20823 projection!\n",
    "\n",
    "Different mobility metrics are calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "479598e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rolling Window Parameters\n",
    "start = \"20200406\"\n",
    "end = \"20200409\"\n",
    "denom = \"3days\"\n",
    "allow_even_subsets = False\n",
    "sam = False\n",
    "del_one_tweeters = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "65562573",
   "metadata": {},
   "outputs": [],
   "source": [
    "ref = \"statistics_notebookdemo/\" + denom + \"_overlap.csv\"\n",
    "if type(sam) == int:\n",
    "    ref = ref.split('.csv')[0] + \"_\" +str(sam) + \".csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "04b5e00a",
   "metadata": {},
   "outputs": [],
   "source": [
    "path = f\"data/{ref}\"\n",
    "file = pd.read_csv(f'data/{ref}', index_col='start_date')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "535d2351",
   "metadata": {},
   "source": [
    "### Load in the neighborhood shapefiles\n",
    "The neighborhoods of Rio are loaded and reprojected to the EPSG:20823.\n",
    "\n",
    "Also the representative point (centroid, but always inside the respective polygon) is calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "42864dbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "barrios_path = 'data/shps/neighborhoods.shp'\n",
    "barrios_20823 = gpd.read_file(barrios_path).to_crs(20823)\n",
    "\n",
    "barrios_20823.CODBAIRRO = barrios_20823.CODBAIRRO.astype(int)\n",
    "barrios_20823['geometry'] = barrios_20823.geometry.representative_point()# oder representative_point?\n",
    "barrios_20823['lat'] = barrios_20823['geometry'].y\n",
    "barrios_20823['lon'] = barrios_20823['geometry'].x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d591e80",
   "metadata": {},
   "source": [
    "# Handle tweet geometries\n",
    "First, load the tweets, reproject to EPSG:20823 and extract the lat and lon coordinates for the respective tweeets.\n",
    "\n",
    "Also save the coordinates of the neighborhoods' representative point for each tweet (these are used later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f407e5c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df = ls.load_and_subset(start, end, del_one_tweeters=del_one_tweeters, samp_size=sam, tweets_path=\"data/tweets/preprocessed_tweets_with_poi_location.csv\")\n",
    "tweets_df['wkt'] = gpd.GeoSeries.from_wkt(tweets_df.wkt).set_crs(4326).to_crs(20823)\n",
    "tweets_df['lat'] = gpd.GeoSeries(tweets_df['wkt']).y\n",
    "tweets_df['lon'] = gpd.GeoSeries(tweets_df['wkt']).x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3855d0c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df['lat_cod'] = tweets_df.cod.apply(lambda x: barrios_20823.loc[barrios_20823.CODBAIRRO == x].lat.values[0])\n",
    "tweets_df['lon_cod'] = tweets_df.cod.apply(lambda x: barrios_20823.loc[barrios_20823.CODBAIRRO == x].lon.values[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "82379829",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1d009a0",
   "metadata": {},
   "source": [
    "Define a TrajectoryDataFrame with the skmob package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "235c4841",
   "metadata": {},
   "outputs": [],
   "source": [
    "tdf = skmob.TrajDataFrame(tweets_df, latitude='lat', longitude='lon', datetime='Timestamp', user_id='User_ID')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b4a022a",
   "metadata": {},
   "source": [
    "# Stats based on POI user dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2775d4b6",
   "metadata": {},
   "source": [
    "### 0. Number of Trips\n",
    "This also inculdes 'trips' from one district in the same. This is the whole number of tweets minus the number of users, since each users has the number of tweets - 1 trips."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "851c058a",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats['poi_number_of_total_trips'] = len(tweets_df) - tweets_df.User_ID.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "470023b9",
   "metadata": {},
   "source": [
    "### 1. Radius of gyration\n",
    "To calculate the Radius of Gyration we use the poi attached to the tweet as well as its standart deviation. A user can have a radius of gyration of 0, if all tweets were sent from the same location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "43be404e",
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(tdf) == 0:\n",
    "    raise ValueError(\"Encountered empty subset!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "864d6003",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 1495.51it/s]\n"
     ]
    }
   ],
   "source": [
    "rg_df = sk_id.radius_of_gyration(tdf)\n",
    "stats[\"mean_rog\"] = rg_df.radius_of_gyration.mean()\n",
    "stats['std_rog'] = rg_df.radius_of_gyration.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d69cb8c0",
   "metadata": {},
   "source": [
    "### 2. Jump lengths\n",
    "We calculate three metrics. The mean distance of the mean travel distance between tweets for each user. The same, but only for users that actually moved. And the mean distance over all movements, without weighting it by users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "593fe537",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 1043.22it/s]\n"
     ]
    }
   ],
   "source": [
    "jl = sk_id.jump_lengths(tdf).jump_lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "93e1a8c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Caluclated the mean for each user, save in the meaned variable\n",
    "def mean2(x):\n",
    "    if len(x) == 0:\n",
    "        return 0\n",
    "    else:\n",
    "        return statistics.mean(x)\n",
    "\n",
    "meaned = jl.apply(mean2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4b73cfff",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats[\"jl_simple_means_over_user_means\"] = meaned.mean()\n",
    "stats[\"jl_std_over_user_means\"] = meaned.std()\n",
    "stats[\"jl_simple_means_only_with_movement_user_means\"] = meaned[meaned > 0].mean()\n",
    "stats[\"jl_std_only_with_movement_user_means\"] = meaned[meaned > 0].std()\n",
    "\n",
    "all_moves = []\n",
    "jl.apply(lambda x: all_moves.extend(x))\n",
    "all_moves = pd.Series(all_moves)\n",
    "\n",
    "stats['mean_total_distance'] = all_moves.sum() / stats['poi_number_of_total_trips']\n",
    "stats['std_total_distance'] = all_moves.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3283343",
   "metadata": {},
   "source": [
    "### 3. Avg. inter-event time (time between tweets)\n",
    "We calculate this once based on all tweets and once based on the mean of all user means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "78cc5cf4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 1496.77it/s]\n"
     ]
    }
   ],
   "source": [
    "wt = sk_id.waiting_times(tdf).waiting_times\n",
    "stats['avg_avg_time_between_tweets_per_user'] = wt.apply(mean2).mean()\n",
    "stats['avg_time_between_tweets_total'] = wt.apply(sum).sum() / stats['poi_number_of_total_trips']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b34a248d",
   "metadata": {},
   "source": [
    "### 4. Average amount of trips per user\n",
    "Including trips to the same location."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "1691d242",
   "metadata": {},
   "outputs": [],
   "source": [
    "stats['avg_number_of_trips_per_user'] = stats['poi_number_of_total_trips'] / tdf.uid.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff27ad99",
   "metadata": {},
   "source": [
    "### 5. Maximum distance\n",
    "The mean over all users maximum distance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9fc7160e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 1186.55it/s]\n"
     ]
    }
   ],
   "source": [
    "max_dist = sk_id.maximum_distance(tdf).maximum_distance\n",
    "stats['mean_max_distance'] = max_dist.mean()\n",
    "stats['std_max_distance'] = max_dist.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8a1beb6",
   "metadata": {},
   "source": [
    "### 7. Number of locations\n",
    "The mean over all users total visited locations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "70de4102",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 822.10it/s]\n"
     ]
    }
   ],
   "source": [
    "locs = sk_id.number_of_locations(tdf).number_of_locations\n",
    "stats['mean_number_of_locations'] = locs.mean()\n",
    "stats['std_number_of_locations'] = locs.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb125ec5",
   "metadata": {},
   "source": [
    "### 8. Maximum distance from home"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b5c21f29",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 211.83it/s]\n"
     ]
    }
   ],
   "source": [
    "home_dist = sk_id.max_distance_from_home(tdf).max_distance_from_home\n",
    "stats['mean_max_dist_from_home'] = home_dist.mean()\n",
    "stats['std_max_dist_from_home'] = home_dist.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1709874",
   "metadata": {},
   "source": [
    "### 9. Mean random location entropy\n",
    "For the loaction entropy we need the neighborhoods, not the poi-coordinates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8fd03261",
   "metadata": {},
   "outputs": [],
   "source": [
    "#############################################\n",
    "tweets_df['lat'] = tweets_df['lat_cod']\n",
    "tweets_df['lon'] = tweets_df['lon_cod']\n",
    "#############################################"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "98414e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "tdf_barr = skmob.TrajDataFrame(tweets_df, latitude='lat', longitude='lon', datetime='Timestamp', user_id='User_ID')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ba376355",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 65/65 [00:00<00:00, 2035.71it/s]\n"
     ]
    }
   ],
   "source": [
    "stats['mean_random_location_entropy_barrios'] = random_location_entropy(tdf_barr).mean()[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fdfda35",
   "metadata": {},
   "source": [
    "### Write into .csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "1a75386f",
   "metadata": {},
   "outputs": [],
   "source": [
    "start = int(start)\n",
    "for name, val in stats.items():\n",
    "    file.loc[start, name] = val\n",
    "    \n",
    "file.to_csv(f'data/{ref}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "16aeb2a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "poi_number_of_total_trips : 240\n",
      "mean_rog : 4488.164655904539\n",
      "std_rog : 5913.697306015925\n",
      "jl_simple_means_over_user_means : 3283.5975167123306\n",
      "jl_std_over_user_means : 4850.386303826439\n",
      "jl_simple_means_only_with_movement_user_means : 8052.632005270718\n",
      "jl_std_only_with_movement_user_means : 4380.498564011526\n",
      "mean_total_distance : 2786.8369518013983\n",
      "std_total_distance : 5086.792843633841\n",
      "avg_avg_time_between_tweets_per_user : 43655.641412196914\n",
      "avg_time_between_tweets_total : 30240.0\n",
      "avg_number_of_trips_per_user : 2.3300970873786406\n",
      "mean_max_distance : 4062.376615746517\n",
      "std_max_distance : 5765.259167799827\n",
      "mean_number_of_locations : 1.5339805825242718\n",
      "std_number_of_locations : 0.8142786291520704\n",
      "mean_max_dist_from_home : 4003.6544237807652\n",
      "std_max_dist_from_home : 5661.241728569151\n",
      "mean_random_location_entropy_barrios : 0.686327208847591\n"
     ]
    }
   ],
   "source": [
    "for key, val in stats.items():\n",
    "    print(key, \":\", val)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}