example_visualization_SFBA_first_failed_experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates the result of the first round of data collection, collected in the San Francisco Bay Area by @shankari. The round had several shortcomings, some of which were addressed during the data collection and some of which were fixed before starting the second round of data collection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import all the dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# for reading and validating data\n",
    "import emeval.input.spec_details as eisd\n",
    "import emeval.input.phone_view as eipv\n",
    "import emeval.input.eval_view as eiev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Visualization helpers\n",
    "import emeval.viz.phone_view as ezpv\n",
    "import emeval.viz.eval_view as ezev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# For plots\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# For maps\n",
    "import branca.element as bre"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and validate data\n",
    "\n",
    "The first issue to note is that we actually have two specs here. The first spec is the checked in `evaluation.spec.sample`, which defines calibration for both stationary and moving instances, and some evaluation trips. However, while starting with the calibration, we noticed some inconsistencies between the power curves. So in order to be more consistent, I defined a second, calibration-only spec `examples/calibration.only.json`, which essentially repeats the calibration experiments multiple times.\n",
    "\n",
    "After that, I returned to the first set of experiments for the moving calibration and the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "DATASTORE_URL = \"http://cardshark.cs.berkeley.edu\"\n",
    "AUTHOR_EMAIL = \"shankari@eecs.berkeley.edu\"\n",
    "sdt3 = eisd.SpecDetails(DATASTORE_URL, AUTHOR_EMAIL, \"sfba_trial_3\")\n",
    "sd_ca_only = eisd.SpecDetails(DATASTORE_URL, AUTHOR_EMAIL, \"sfba_calibration_only_1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pvt3 = eipv.PhoneView(sdt3)\n",
    "pv_ca_only = eipv.PhoneView(sd_ca_only)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Issue #1: Identical transition timestamps\n",
    "\n",
    "While exploring the data after the collection was done, there were many inconsistencies with the way in which the transitions and configurations were pushed to the server. In particular, because I save the timestamps as integer unix timestamps (using arrow.get().unix()), it is possible for elements stored in quick succession to have identical write timestamps and to not be retrieved correctly. And sometimes, due to races, the transitions were not even stored correctly (https://github.com/e-mission/e-mission-docs/issues/415) I resolved these manually for the most part so that we could get preliminary results but I did not resolve this since it is only for validation. The validation check fails because there were no modified sensor configs detected during the medium accuracy calibration on android.\n",
    "\n",
    "```\n",
    "About to retrieve messages using {'user': 'ucb-sdb-android-1', 'key_list': ['config/sensor_config'], 'start_time': 1561132633, 'end_time': 1561135735}\n",
    "response = <Response [200]>\n",
    "Found 0 entries\n",
    "medium_accuracy_train_AO -> []\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Commented out because this fails\n",
    "# pvt3.validate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pv_ca_only.validate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evt3 = eiev.EvaluationView()\n",
    "evt3.from_view_single_run(pvt3, \"\")\n",
    "evt3.from_view_eval_trips(pvt3, \"\", \"\")\n",
    "ev_ca_only = eiev.EvaluationView()\n",
    "ev_ca_only.from_view_multiple_runs(pv_ca_only, \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now for the results (calibration, phone view)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Battery drain over time (stationary)\n",
    "\n",
    "#### First experiment (single run)\n",
    "\n",
    "The figures below show the battery drain over time for both the stationary and moving calibrations\n",
    "The first set of figures are the initial stationary data collected with the first spec. As we can see, the android curves are almost identical, but the iOS curves show a clear difference between two pairs of phones. Phones (1,4) and phones (2,3) are almost identical with each other but noticeably different from the other pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, [android_ax, ios_ax]) = plt.subplots(ncols=2, nrows=1, figsize=(25,6))\n",
    "\n",
    "ezpv.plot_all_power_drain(ios_ax, pvt3.map()[\"ios\"], \"calibration\", \"stationary\")\n",
    "# ios_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))\n",
    "ezpv.plot_all_power_drain(android_ax, pvt3.map()[\"android\"], \"calibration\", \"stationary\")\n",
    "# android_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Second experiment (multiple runs)\n",
    "\n",
    "Since this was surprising, I decided to run the experiments multiple times to try and avoid noise. The results are shown below. There is clearly a greater variation in the iOS case than the android case; I am not sure if it can be controlled any better. We may just need to work with higher tolerances on iOS. This also indicates several issues that need to be addressed in the next round.\n",
    "\n",
    "##### Issue #2: Medium accuracy on iOS\n",
    "\n",
    "The iOS accuracy levels are defined (as CLLocationAccuracy constants) [https://developer.apple.com/documentation/corelocation/cllocationaccuracy?language=objc]\n",
    "Based on the list, I picked high accuracy = `kCLLocationAccuracyBest` and medium accuracy = `kCLLocationAccuracyNearestTenMeters`. However, at least in our testing, there was no significant difference in power drain between the two options. We will see later that there doesn't appear to be a significant difference in accuracy either. The option which really separated from the curve was `kCLLocationAccuracyHundredMeters` which I had mapped to low accuracy. In the next round, I need to switch medium accuracy = `kCLLocationAccuracyHundredMeters` and low accuracy = `kCLLocationAccuracyKilometer`?\n",
    "\n",
    "##### Issue #3: Built-in duty cycling on android\n",
    "\n",
    "It appears that android has some form of built-in duty cycling in high accuracy mode, where the power drain slope abruptly changes around 2 hours. We will see some additional evidence of this later. After 2.5 hours, the slope appears to be more similar to medium accuracy. There does not appear to be such a knee during medium accuracy collection.\n",
    "\n",
    "##### Issue #4: Unexpected and unexplained move out of duty cycling on android\n",
    "\n",
    "This only happened once, but it looks like one phone moved back into the active state during one run causing a second clear increase in slope at around 12.5 hours. We will see additional evidence for this later as well. It is not clear what caused this to happen, and it is also not clear why the others did not follow suit. Such idiosyncracies could complicate efforts to observe power drain during evaluation.\n",
    "\n",
    "##### Issue #5: Representing multiple runs\n",
    "\n",
    "This is more of a UI issue, but the current version of the UI did not allow for more than one full screen of calibration options. This meant that we could only see one low accuracy option, which is why we have limited low accuracy data. We need to figure out how best to represent this - allow the UI to display more options? separate the run from the calibration option? both?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, [android_ax, ios_ax]) = plt.subplots(ncols=1, nrows=2, figsize=(10,10))\n",
    "\n",
    "ezpv.plot_all_power_drain(ios_ax, pv_ca_only.map()[\"ios\"], \"calibration\", \"stationary\")\n",
    "ios_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5), ncol=2)\n",
    "ezpv.plot_all_power_drain(android_ax, pv_ca_only.map()[\"android\"], \"calibration\", \"stationary\")\n",
    "android_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5), ncol=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Recap of the issues with another view\n",
    "\n",
    "This other view displays the plots for each phone over multiple runs. This highlights the previous issues again:\n",
    "- medium accuracy and high accuracy on iOS are almost identical, low accuracy is significantly different\n",
    "- the duty cycling for `high-accuracy-stationary-4` on `ucb-sdb-android-3` is very clear and is different from the others\n",
    "- for `high-accuracy-stationary-0` on `ucb-sdb-android-1`, there are two discontinuities - the second one, around 12.5 hours sharply increases the power drain\n",
    "- the `high-accuracy-stationary-0` run on `ucb-sdb-ios-3`, the `medium-accuracy-stationary-0` run on `ucb-sdb-ios-4` are significantly different from the others. The first is an outlier even in the aggregate (see above), the second is only an outlier for this phone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, ax) = plt.subplots(figsize=(12,3), nrows=0, ncols=0)\n",
    "ezpv.plot_separate_power_drain(ifig, pv_ca_only.map()[\"ios\"], 4, \"calibration\", \"stationary\")\n",
    "(ifig, ax) = plt.subplots(figsize=(12,3), nrows=0, ncols=0)\n",
    "ezpv.plot_separate_power_drain(ifig, pv_ca_only.map()[\"android\"], 4, \"calibration\", \"stationary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Battery drain over time (moving calibration)\n",
    "\n",
    "The moving calibration runs were not very useful in terms of battery drain, since there were too few points to be useful. Part of this is inherent in the definition of moving calibration, since it is unlikely that we will move for 10-15 hours at a time to collect the kind of data we have in the stationary case. And if our trip lasts for an hour, but we only read the battery level once an hour, we will end up with close to no data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, [android_ax, ios_ax]) = plt.subplots(ncols=1, nrows=2, figsize=(10,10))\n",
    "\n",
    "ezpv.plot_all_power_drain(ios_ax, pvt3.map()[\"ios\"], \"calibration\", \"AO\")\n",
    "ios_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))\n",
    "ezpv.plot_all_power_drain(android_ax, pvt3.map()[\"android\"], \"calibration\", \"AO\")\n",
    "android_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking counts (stationary)\n",
    "\n",
    "We now check the number of data points collected during calibration and their distribution in an effort to validate the duty cycling. Observations from this are:\n",
    "\n",
    "##### on android: more points = more power drain\n",
    "\n",
    "As we would expect, the number of points across the various phones and the various runs is almost identical. In the cases where it is significantly different (e.g. `high-accuracy-stationary-0` on `ucb-sdb-android-1` and `high-accuracy-stationary-3` on `ucb-sdb-android-3`), we have see significant differences in the power drain as well. However, we do not understand why these two runs behave differently from the other runs.\n",
    "\n",
    "##### on iOS: almost no points\n",
    "\n",
    "Since iOS has a distance filter, and not a time filter, and this calibration was stationary, almost no points are generated for high accuracy sensing. However, with low accuracy sensing (which is actually medium accuracy), we do get a significant number of points (an order of magnitude more), although nowhere near the number of entries on android.\n",
    "\n",
    "##### on android: medium accuracy = almost no points\n",
    "\n",
    "On android, medium accuracy sensing generates two orders of magnitude fewer points than high accuracy. So the additional power drain on android probably reflects not just the sensing cost but also the processing cost. This also indicates that the medium accuracy sensing, which relies on WiFi and cellular signal strengths, is likely to be suspended when the phone is in doze mode, and is consistent with prior observed behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "count_df = ezpv.get_count_df(pv_ca_only); count_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, ax) = plt.subplots(nrows=1, ncols=3, figsize=(16,8))\n",
    "count_df.filter(like=\"high_accuracy\").filter(like=\"android\", axis=0).plot(ax=ax[0],kind=\"bar\")\n",
    "count_df.filter(like=\"ios\", axis=0).plot(ax=ax[1],kind=\"bar\")\n",
    "count_df.filter(like=\"medium_accuracy\").filter(like=\"android\", axis=0).plot(ax=ax[2],kind=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking counts (moving)\n",
    "\n",
    "Although the battery drain is not significant while moving, the counts are likely to be much more relevant, specially in the iOS case, with the distance filter.\n",
    "\n",
    "##### on iOS: significant number of points\n",
    "\n",
    "Since iOS has a distance filter, we finally have a reasonable set of location points for both platforms. The number of points on iOS is still consistently lower than the corresponding count on android\n",
    "\n",
    "##### on iOS: medium accuracy is consistently lower than high accuracy\n",
    "\n",
    "Recall that the \"medium\" accuracy here is `kCLLocationAccuracyNearestTenMeters` which did not have a significantly different power drain than `kCLLocationAccuracyBest`. However, the number of points is much lower when this medium accuracy is selected.\n",
    "\n",
    "##### on android: medium accuracy = significant number of points, but lower\n",
    "\n",
    "On android, medium accuracy sensing now generates ~ 0.5 * the number of points with high accuracy, but the medium accuracy numbers are consistently lower than the high accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "count_df = ezpv.get_count_df(pvt3); count_df.filter(like=\"AO\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, ax) = plt.subplots(nrows=1, ncols=2, figsize=(16,8), sharey=True)\n",
    "count_df.filter(like=\"AO\").filter(like=\"android\", axis=0).plot(ax=ax[0],kind=\"bar\")\n",
    "count_df.filter(like=\"AO\").filter(like=\"ios\", axis=0).plot(ax=ax[1],kind=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking densities (stationary)\n",
    "\n",
    "Density checks don't make as much sense on iOS, since there are so few entries, so we will focus mainly on android.\n",
    "\n",
    "##### on android: duty cycling = density variation\n",
    "\n",
    "In general, most of the android points are sensed right after the calibration starts, at around zero. There are also a couple of minor bumps around hours 2, 6 and 15. This seems consistent with the explanation of doze mode, in which the phone goes into a lower power state when not in use and wakes up at increasing intervals. The exceptions are `high-accuracy-stationary-1` on `ucb-sdb-android-1`, which corresponds to the abrupt increase in power drain seen in the power curves. There is also a somewhat unusual bump related to `low-accuracy-stationary-4` on `ucb-sdb-android-4` but probably because the accuracy is already low, and the bump is small, we do not see a visible difference in slope for that curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "android_density_df = ezpv.get_location_density_df(pv_ca_only.map()[\"android\"])\n",
    "nRows = ezpv.get_row_count(len(android_density_df), 2)\n",
    "print(nRows)\n",
    "android_ax = android_density_df.plot(kind='density', subplots=False, layout=(nRows, 2), figsize=(10,10), sharex=True, sharey=True)\n",
    "android_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))\n",
    "\n",
    "(ifig, ax) = plt.subplots(figsize=(16,4), nrows=0, ncols=0)\n",
    "ezpv.plot_separate_density_curves(ifig, pv_ca_only.map()[\"android\"], 4, \"calibration\", \"stationary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking densities (moving)\n",
    "\n",
    "As expected, when moving, while the densities do vary, they do not show the kind of spiky behavior that we see while stationary. Instead, we get points pretty much throughout the travel time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "android_density_df = ezpv.get_location_density_df(pvt3.map()[\"android\"])\n",
    "nRows = ezpv.get_row_count(len(android_density_df), 2)\n",
    "print(nRows)\n",
    "android_ax = android_density_df.filter(like=\"AO\").plot(kind='density', subplots=False, layout=(nRows, 2), figsize=(10,10), sharex=True, sharey=True)\n",
    "android_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))\n",
    "\n",
    "ios_density_df = ezpv.get_location_density_df(pvt3.map()[\"ios\"])\n",
    "nRows = ezpv.get_row_count(len(ios_density_df), 2)\n",
    "print(nRows)\n",
    "ios_ax = ios_density_df.filter(like=\"AO\").plot(kind='density', subplots=False, layout=(nRows, 2), figsize=(10,10), sharex=True, sharey=True)\n",
    "ios_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking trajectories (stationary)\n",
    "\n",
    "While checking the counts and densities, we looked at the location sensing data **over time**. We can also look at it **over space**, by displaying it on a map. At this point, stationary data is less interesting because we basically expect it to be concenrated around a single location. However, on visualizing it, we can see some unexpected behavior.\n",
    "\n",
    "##### on all phones: there are unexpected jumps\n",
    "\n",
    "Even in the case of high accuracy sensing, on both android and iOS, we see jumps from the stationary location. These jumps are particularly pronounced in `ucb-sdb-android-2_medium_accuracy_stationary_2`, `ucb-sdb-ios-1_low_accuracy_stationary_4`, where they cover 5-6 blocks, but we can see at least one block displacements in a bunch of other maps (e.g. `ucb-sdb-ios-2_high_accuracy_stationary_1`)\n",
    "\n",
    "##### on android: low accuracy really sucks\n",
    "\n",
    "The low accuracy option on android jumps all over the map in a very distinctive zig-zag pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ha_map_list = ezpv.get_map_list(pv_ca_only, \"calibration\", \"\")\n",
    "ha_map_list.extend(ezpv.get_map_list(pvt3, \"calibration\", \"stationary\"))\n",
    "rows = ezpv.get_row_count(len(ha_map_list), 8)\n",
    "evaluation_maps = bre.Figure(ratio=\"{}%\".format((rows/4) * 100))\n",
    "for i, curr_map in enumerate(ha_map_list):\n",
    "    evaluation_maps.add_subplot(rows, 8, i+1).add_child(curr_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking trajectories (moving)\n",
    "\n",
    "As expected, these are more interesting than the stationary trajectories. Some observations:\n",
    "- the high accuracy trajectories look reasonably good, but the medium accuracy trajectories on android have significant zig zags\n",
    "\n",
    "- the iOS medium accuracy trajectories look really good in comparison, but note that in this run, \"medium accuracy\" seems to incur a power drain close to high accuracy. We need to retry with the medium accuracy set to low accuracy (issue already identified)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ha_map_list = ezpv.get_map_list(pvt3, \"calibration\", \"AO\")\n",
    "rows = ezpv.get_row_count(len(ha_map_list), 8)\n",
    "evaluation_maps = bre.Figure(ratio=\"{}%\".format((rows/4) * 100))\n",
    "for i, curr_map in enumerate(ha_map_list):\n",
    "    evaluation_maps.add_subplot(rows, 8, i+1).add_child(curr_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now for the results (calibration, evaluation view)!\n",
    "\n",
    "### Trajectory matching\n",
    "\n",
    "In the phone view, we were able to compare phone results against each other (e.g. `ucb-sdb-android-1` v/s `ucb-sdb-android-2` for the same run) by plotting them on the same graph. We need something similar for trajectories, so that we can get a better direct comparison against various configurations. To make this easier, we want to switch the view so that the calibration ranges are first grouped by the settings and then by the phone. Once we do that, we can compare trajectories from different phones for the same experiment in the same map.\n",
    "\n",
    "##### Issue #1: No matching with ground truth\n",
    "\n",
    "Zooming into the maps, we can see that even in the high accuracy case, there are mismatches between the trajectories. For example, the iOS high accuracy maps between South San Francisco and San Francisco, android medium accuracy maps between SF and the Easy Bay. Even if the trajectories match, they don't necessarily match with the ground truth, for example, the android high accuracy maps between 22nd street and 4th and King, iOS medium accuracy right after reaching Oakland. We should extend the spec to support this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map_list = ezev.get_map_list_single_run(evt3, \"calibration\", \"AO\")\n",
    "rows = ezpv.get_row_count(len(map_list), 2)\n",
    "evaluation_maps = bre.Figure(ratio=\"{}%\".format((rows/4) * 100))\n",
    "for i, curr_map in enumerate(map_list):\n",
    "    evaluation_maps.add_subplot(rows, 2, i+1).add_child(curr_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Battery drain (stationary)\n",
    "\n",
    "This is less important since the plots with all curves do allow for direct comparisons between the battery drain curves across multiple phones. But just for the record, let us generate subplots that are grouped by run instead of by phone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "(ifig, ax) = plt.subplots(figsize=(16,6),nrows=0,ncols=0)\n",
    "ezev.plot_separate_power_drain_multiple_runs(ifig, 3, ev_ca_only.map(\"calibration\")[\"android\"], \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now for the results (evaluation, evaluation view)!\n",
    "\n",
    "### Trajectory matching\n",
    "\n",
    "Finally, we get to the evaluation, in which we run different regimes across the different phones. We also have pre-determined ground truth for the trips. Since our entire goal is to compare the trips against each other, we will go directly to the evaluation view.\n",
    "\n",
    "\n",
    "##### Issue #1: Tracking not turned off for the power control\n",
    "\n",
    "We can see that the power control also has location entries. This is because, even in the case of the power control, although we were setting the accuracy to the lowest possible and also sampling at a very low rate, we were not turning tracking off. We need to fix this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evt3.map(\"evaluation\")[\"android\"][\"HAHFDC v/s HAMFDC\"][\"short_walk_suburb\"][\"power_control\"][\"location_df\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other observations include:\n",
    "- The trajectory lines all match up pretty well, but that is not surprising since this was a high accuracy v/s high accuracy comparison, with only the filter being different\n",
    "- The android evaluation phones ran out of battery before the second set of trips, so we only have the accuracy control for the `short_car_suburb` and `short_car_suburb_freeway`\n",
    "- There is a clear zigzag in the android `short_bike_suburb` case\n",
    "- The gap between the actual start of the trip and the detected start of the trip is much larger on iOS (~ 3-4 blocks) than android (~ 1-2 blocks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map_list = ezev.get_map_list_eval_trips(evt3, \"evaluation\", \"AO\")\n",
    "rows = ezpv.get_row_count(len(map_list), 2)\n",
    "evaluation_maps = bre.Figure(ratio=\"{}%\".format((rows/4) * 100))\n",
    "for i, curr_map in enumerate(map_list):\n",
    "    evaluation_maps.add_subplot(rows, 2, i+1).add_child(curr_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation_maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigating android evaluation power drain\n",
    "\n",
    "The android evaluation power drain is surprising. We would expect power drain of the evaluation (which duty cycles the sensing) to be much lower than the accuracy control, which senses continuously. However, both the evaluation phones ran out of battery before the second trip, and the accuracy control did not. Let's verify this from the battery drain.\n",
    "\n",
    "We know that individual trip power drains will not tell us much because of the short durations. But the range-specific tracking should have some values...\n",
    "\n",
    "Aha! We can see that the duty cycling works as expected on iOS. The power drain of both regimes is almost identical to the power control, although we would expect the power control to get even lower when we actually stop tracking. However, on android, the evaluation regimes are in fact almost identical and with a much higher drain than the accuracy control."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ifig, [android_ax, ios_ax]) = plt.subplots(ncols=1, nrows=2, figsize=(10,10))\n",
    "\n",
    "ezpv.plot_all_power_drain(ios_ax, pvt3.map()[\"ios\"], \"evaluation\", \"\")\n",
    "ios_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5), ncol=1)\n",
    "ezpv.plot_all_power_drain(android_ax, pvt3.map()[\"android\"], \"evaluation\", \"\")\n",
    "android_ax.legend(loc=\"center left\", bbox_to_anchor=(1, 0.5), ncol=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Issue #1: Milliseconds?\n",
    "\n",
    "Investigating this further by viewing the logs on the phone, we realize that we configure the android filter_time with the value of the filter directly. e.g. if filter = 1, then filter_time = 1. However, the API expects the time in milliseconds, so we are effectively setting this to 1 millisecond, not one second. Since the minimum filter_time is 1 second, this ensures that:\n",
    "- we get data every second in all the regimes (see plots below)\n",
    "- the HAHFDC and HAMFDC are effectively the same\n",
    "- the number of entries that we need to find before we detect the trip end is in the tens of thousands. We don't actually achieve this number, so we keep sensing anyway, so we never actually duty cycle (see transition list below)\n",
    "\n",
    "But why are they **worse** than the accuracy control? Given that the sensing is largely identical, this **must be** due to the additional processing of iterating over all the entries to determine whether the trip has ended. So there really does appear to be a tradeoff between lower sensing and more local computation in the duty cycling case, specially for CPU-hungry phones. We might want to experiment further with this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_eval_range = evt3.map(\"evaluation\")[\"android\"][\"HAHFDC v/s HAMFDC\"]\n",
    "(ifig, ax_list) = plt.subplots(ncols=3, nrows=1, figsize=(12,4))\n",
    "\n",
    "for i, (regime, regime_map) in enumerate(test_eval_range[\"short_walk_suburb\"].items()):\n",
    "    if i == 3:\n",
    "        continue\n",
    "    regime_map[\"location_df\"].ts.diff().hist(ax=ax_list[i], label=regime)\n",
    "\n",
    "    ax_list[i].set_title(regime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import arrow\n",
    "\n",
    "test_transition_phone = \"ucb-sdb-android-2\"\n",
    "test_eval_range = pvt3.map()[\"android\"][test_transition_phone][\"evaluation_ranges\"][0]\n",
    "transition_entries = pvt3.spec_details.retrieve_data_from_server(test_transition_phone, [\"statemachine/transition\"], test_eval_range[\"start_ts\"], test_eval_range[\"end_ts\"])\n",
    "print(\"\\n\".join([str((t[\"data\"][\"transition\"], t[\"data\"][\"ts\"], arrow.get(t[\"data\"][\"ts\"]).to(pvt3.spec_details.eval_tz))) for t in transition_entries]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the motion activity\n",
    "\n",
    "In addition to location data, we also read the motion_activity data from the closed source phone APIs. Let's quickly check how accurate the raw motion activity is.\n",
    "\n",
    "The medium accuracy runs seem to be much more noisy wrt motion activity transitions. We should really not get a lot of transitions since we essentially took various trains for the entire route. The high accuracy sensing seems to be largely stable, except for one extraneous transition in the middle of the `ucb-sdb-ios-1` run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(ifig, ax) = plt.subplots(nrows=2, ncols=2, figsize=(12,8), sharex=True)\n",
    "ezpv.display_unprocessed_android_activity_transitions(pvt3, ax[0][0], \"calibration\", \"medium_accuracy_train_AO\")\n",
    "ezpv.display_unprocessed_android_activity_transitions(pvt3, ax[0][1], \"calibration\", \"high_accuracy_train_AO\")\n",
    "ezpv.display_unprocessed_ios_activity_transitions(pvt3, ax[1][0], \"calibration\", \"medium_accuracy_train_AO\")\n",
    "ezpv.display_unprocessed_ios_activity_transitions(pvt3, ax[1][1], \"calibration\", \"high_accuracy_train_AO\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}