radar_trajectories_starting_kit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Paris Saclay Center for Data Science](http://www.datascience-paris-saclay.fr)\n",
    "\n",
    "## [Radar trajectories RAMP](http://www.ramp.studio/problems/radar_trajectories): classifying flying objects using radar trajectories\n",
    "\n",
    "_Balázs Kégl (LAL/CNRS), Silvère Bonnabel (Mines ParisTech), Sami Jouaber (Mines ParisTech)_\n",
    "\n",
    "## Introduction\n",
    "\n",
    "The first goal of a traffic control center is to yield a clear vision of the state of the air traffic. This vision is based on various information about the flying aircrafts, namely radar measurements and other information provided by the planes themselves about their position, as well as some of their characteristics, through the Automatic Dependence Surveillance Broadcast (ADSB) system. To provide air traffic controllers with efficient decision making tools, one should be able to automatically detect a number of features regarding an aircraft, solely based on the radar measurements (that do not depend on whether the aircraft is cooperating or not). In particular, it is desirable to establish whether the aircraft is civilian or military, if its behavior is aggressive or not, what kind of maneuvers it is capable of performing, and which category of airplane it is (helicopter, fighter, liner, drone, etc.). Such information may prove useful for instance to adapt the tracking\n",
    "algorithm of the radars to the type of aircraft (highly maneuvering aircrafts may require more radar\n",
    "resource to be tracked), and more generally for decision making supporting tools.\n",
    "\n",
    "## Type of data used and prediction task\n",
    "\n",
    "Radar measurements are classified since they contain information about French military operations and trips. Hence, we have collected aircraft trajectories (i.e., position over time) using the publicly available ADSB data. Those trajectories serve as a publicly available alternative to the radar position measurements. The goal of the classification task is to recognize the type of flying objects. There are 19 types, labeled by 4-digit strings: '1111', '1112', '1121', '1122', '1132', '1222', '1224', '1231', '1232', '1233', '1234', '1324', '1332', '1333', '1334', '4111', '4121', '4122', '4222'. Each digit denotes an attribute\n",
    "1. Species: Kind of aircraft\n",
    "     - 1 = Airplane - Liner\n",
    "     - 4 = Helicopter\n",
    "2. WTC: Wake Turbulence Category\n",
    "     - 1 = Light\n",
    "     - 2 = Middle\n",
    "     - 3 = Heavy\n",
    "3. EngType: Kind of engine\n",
    "     - 1 = Piston\n",
    "     - 2 = Turboprop\n",
    "     - 3 = Jet\n",
    "4. Engines: Number of engines\n",
    "\n",
    "The models will be evaluated by cross entropy a.k.a negative log likelihood (NLL)\n",
    "$$\n",
    "- \\frac{1}{19} \\sum_{i=1}^{19} \\log \\hat{p}_i\n",
    "$$\n",
    "where $\\hat{p}_i$ is the predicted probability of the $i$th class given the input (radar trajectories, see below). Besides this official score, we will also display the accuracy (number of correctly classified types divided by the size of the test set) which is more human readable. The NLL score has the advantage that it incentivizes you to come up with unbiased probability estimates which can then be aggregated across aircraft types if we change the task (e.g., to predict the number of engines). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements\n",
    "\n",
    "* numpy>=1.10.0  \n",
    "* matplotlib>=1.5.0 \n",
    "* pandas>=0.19.0  \n",
    "* scikit-learn>=0.18\n",
    "* ramp-workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory data analysis\n",
    "\n",
    "Let's first import the `problem.py` script which has data reader routines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import imp\n",
    "module_name = str(int(1000000000 * np.random.rand()))\n",
    "problem = imp.load_source('', 'problem.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the training data and labels from `data/train.pkl` into a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>data</th>\n",
       "      <th>trajectory_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[[142.3, 13065.2072311, -37.3482725781, 7.6629...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[[178.1, 12293.3296794, -41.8313170148, -2.782...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[[528.1, -2603.7653628, -173.591179302, 4.0816...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[[204.0, 11468.8976453, -75.9557784458, -3.710...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[[335.0, 6668.3636331, -107.36545794, 0.927818...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[[38.9, 17597.3725596, -47.7188174498, 8.40393...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[[264.2, 9455.57780146, -76.1110502916, -2.119...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[[238.6, 9664.36696588, 29.6750335695, 8.19662...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[[363.7, 5592.22368649, -125.457647118, -0.083...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[[49.2, 17759.6881089, -5.62268972554, -5.7678...</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>[[472.1, -42.1417064816, 7.0713644971, 1.87018...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>[[291.0, 2298.736692, -0.0323523516254, 0.4857...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>[[675.4, -4260.21887764, -13.2879657703, 0.545...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>[[556.4, -1825.59914711, -8.14625975095, 0.900...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>[[330.8, 2426.90138763, 3.87205271099, -0.1364...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>[[685.9, -4395.5911207, -22.1951759656, -1.736...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>[[14.9, 4911.55669731, 12.1206841511, 0.370553...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>[[357.8, 2686.60159488, 21.9511101768, -0.1748...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>[[132.2, 4480.04507975, -16.5359879127, -1.786...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>[[659.1, -3976.53311388, -3.88350480198, 0.680...</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>[[106.3, 1146.38431717, -8.18094992124, 2.1219...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>[[106.8, 1142.22162909, -7.19163728452, 2.7712...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>[[461.8, 187.279251201, -21.6533741143, 0.8091...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>[[373.2, 198.704394999, -15.093281687, 2.11122...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>[[55.3, -940.90046099, 26.9039372166, -2.71493...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>[[176.9, 2250.71107565, 14.3212124757, -1.0962...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>[[314.5, -52.1320242234, -12.4979824227, 2.874...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>[[6.4, -3425.34755694, 21.9551620197, -5.30540...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>[[117.9, 1669.27922476, 86.3650682046, -3.1277...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>[[378.3, 98.9739328194, -3.80143135981, 6.1554...</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4530</th>\n",
       "      <td>[[148.6, 2783.40869038, -46.6481307732, -2.041...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4531</th>\n",
       "      <td>[[237.4, -525.434142389, -47.5379049754, 0.496...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4532</th>\n",
       "      <td>[[192.1, 945.589811744, -29.4108783189, 1.0554...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4533</th>\n",
       "      <td>[[287.9, -2341.72297706, -35.260872666, -0.594...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4534</th>\n",
       "      <td>[[53.8, 3533.5143775, -12.6013107342, 1.739942...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4535</th>\n",
       "      <td>[[209.9, 510.906686786, -24.5684797162, -1.280...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4536</th>\n",
       "      <td>[[76.3, 3601.35436204, 4.34863452767, 0.206743...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4537</th>\n",
       "      <td>[[454.6, -3171.62984681, 5.00043957379, -0.212...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4538</th>\n",
       "      <td>[[307.5, -3004.67800843, -26.8507987693, 0.030...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4539</th>\n",
       "      <td>[[240.3, -654.899098708, -43.4519449136, 1.235...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4540</th>\n",
       "      <td>[[386.5, -1387.77040902, -31.8813172193, 2.091...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4541</th>\n",
       "      <td>[[354.8, 400.292189388, -68.3890927356, -1.006...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4542</th>\n",
       "      <td>[[248.6, 1062.99894428, 14.3608892973, -1.2425...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4543</th>\n",
       "      <td>[[11.8, 2582.63360695, -8.71923928381, -2.7321...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4544</th>\n",
       "      <td>[[470.3, -2022.66027478, 27.1378827479, 2.3485...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4545</th>\n",
       "      <td>[[396.4, -1587.48940415, -12.248539134, 1.1476...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4546</th>\n",
       "      <td>[[322.5, 1671.31871288, -5.84098864689, -1.815...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4547</th>\n",
       "      <td>[[488.8, -1166.00487404, 56.2008798405, 0.3148...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4548</th>\n",
       "      <td>[[378.2, -1048.35005812, -48.8377903704, 1.522...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4549</th>\n",
       "      <td>[[372.4, -746.457891409, -53.1925133702, 1.095...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4550</th>\n",
       "      <td>[[98.1, 4955.26590254, -4.73412492723, 0.04181...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4551</th>\n",
       "      <td>[[56.4, 5127.94332756, -4.54195074342, 0.04850...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4552</th>\n",
       "      <td>[[229.4, 3424.07912631, -16.2357814001, -0.672...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4553</th>\n",
       "      <td>[[161.6, 4300.92124025, -14.1158033421, 0.0712...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4554</th>\n",
       "      <td>[[62.6, 5103.2111739, -3.59358707414, 0.022763...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4555</th>\n",
       "      <td>[[71.5, 5069.3993689, -4.30435198872, -0.02487...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4556</th>\n",
       "      <td>[[131.3, 4751.98564098, -13.6974594605, -0.527...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4557</th>\n",
       "      <td>[[142.3, 4572.96214542, -16.6859366302, 0.1635...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4558</th>\n",
       "      <td>[[239.3, 3219.52936855, -24.8032633276, -0.753...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4559</th>\n",
       "      <td>[[104.4, 4927.94947637, -4.02758534422, 0.0481...</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4560 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    data trajectory_id\n",
       "index                                                                 \n",
       "0      [[142.3, 13065.2072311, -37.3482725781, 7.6629...            10\n",
       "1      [[178.1, 12293.3296794, -41.8313170148, -2.782...            10\n",
       "2      [[528.1, -2603.7653628, -173.591179302, 4.0816...            10\n",
       "3      [[204.0, 11468.8976453, -75.9557784458, -3.710...            10\n",
       "4      [[335.0, 6668.3636331, -107.36545794, 0.927818...            10\n",
       "5      [[38.9, 17597.3725596, -47.7188174498, 8.40393...            10\n",
       "6      [[264.2, 9455.57780146, -76.1110502916, -2.119...            10\n",
       "7      [[238.6, 9664.36696588, 29.6750335695, 8.19662...            10\n",
       "8      [[363.7, 5592.22368649, -125.457647118, -0.083...            10\n",
       "9      [[49.2, 17759.6881089, -5.62268972554, -5.7678...            10\n",
       "10     [[472.1, -42.1417064816, 7.0713644971, 1.87018...            12\n",
       "11     [[291.0, 2298.736692, -0.0323523516254, 0.4857...            12\n",
       "12     [[675.4, -4260.21887764, -13.2879657703, 0.545...            12\n",
       "13     [[556.4, -1825.59914711, -8.14625975095, 0.900...            12\n",
       "14     [[330.8, 2426.90138763, 3.87205271099, -0.1364...            12\n",
       "15     [[685.9, -4395.5911207, -22.1951759656, -1.736...            12\n",
       "16     [[14.9, 4911.55669731, 12.1206841511, 0.370553...            12\n",
       "17     [[357.8, 2686.60159488, 21.9511101768, -0.1748...            12\n",
       "18     [[132.2, 4480.04507975, -16.5359879127, -1.786...            12\n",
       "19     [[659.1, -3976.53311388, -3.88350480198, 0.680...            12\n",
       "20     [[106.3, 1146.38431717, -8.18094992124, 2.1219...            14\n",
       "21     [[106.8, 1142.22162909, -7.19163728452, 2.7712...            14\n",
       "22     [[461.8, 187.279251201, -21.6533741143, 0.8091...            14\n",
       "23     [[373.2, 198.704394999, -15.093281687, 2.11122...            14\n",
       "24     [[55.3, -940.90046099, 26.9039372166, -2.71493...            14\n",
       "25     [[176.9, 2250.71107565, 14.3212124757, -1.0962...            14\n",
       "26     [[314.5, -52.1320242234, -12.4979824227, 2.874...            14\n",
       "27     [[6.4, -3425.34755694, 21.9551620197, -5.30540...            14\n",
       "28     [[117.9, 1669.27922476, 86.3650682046, -3.1277...            14\n",
       "29     [[378.3, 98.9739328194, -3.80143135981, 6.1554...            14\n",
       "...                                                  ...           ...\n",
       "4530   [[148.6, 2783.40869038, -46.6481307732, -2.041...            47\n",
       "4531   [[237.4, -525.434142389, -47.5379049754, 0.496...            47\n",
       "4532   [[192.1, 945.589811744, -29.4108783189, 1.0554...            47\n",
       "4533   [[287.9, -2341.72297706, -35.260872666, -0.594...            47\n",
       "4534   [[53.8, 3533.5143775, -12.6013107342, 1.739942...            47\n",
       "4535   [[209.9, 510.906686786, -24.5684797162, -1.280...            47\n",
       "4536   [[76.3, 3601.35436204, 4.34863452767, 0.206743...            47\n",
       "4537   [[454.6, -3171.62984681, 5.00043957379, -0.212...            47\n",
       "4538   [[307.5, -3004.67800843, -26.8507987693, 0.030...            47\n",
       "4539   [[240.3, -654.899098708, -43.4519449136, 1.235...            47\n",
       "4540   [[386.5, -1387.77040902, -31.8813172193, 2.091...             6\n",
       "4541   [[354.8, 400.292189388, -68.3890927356, -1.006...             6\n",
       "4542   [[248.6, 1062.99894428, 14.3608892973, -1.2425...             6\n",
       "4543   [[11.8, 2582.63360695, -8.71923928381, -2.7321...             6\n",
       "4544   [[470.3, -2022.66027478, 27.1378827479, 2.3485...             6\n",
       "4545   [[396.4, -1587.48940415, -12.248539134, 1.1476...             6\n",
       "4546   [[322.5, 1671.31871288, -5.84098864689, -1.815...             6\n",
       "4547   [[488.8, -1166.00487404, 56.2008798405, 0.3148...             6\n",
       "4548   [[378.2, -1048.35005812, -48.8377903704, 1.522...             6\n",
       "4549   [[372.4, -746.457891409, -53.1925133702, 1.095...             6\n",
       "4550   [[98.1, 4955.26590254, -4.73412492723, 0.04181...             9\n",
       "4551   [[56.4, 5127.94332756, -4.54195074342, 0.04850...             9\n",
       "4552   [[229.4, 3424.07912631, -16.2357814001, -0.672...             9\n",
       "4553   [[161.6, 4300.92124025, -14.1158033421, 0.0712...             9\n",
       "4554   [[62.6, 5103.2111739, -3.59358707414, 0.022763...             9\n",
       "4555   [[71.5, 5069.3993689, -4.30435198872, -0.02487...             9\n",
       "4556   [[131.3, 4751.98564098, -13.6974594605, -0.527...             9\n",
       "4557   [[142.3, 4572.96214542, -16.6859366302, 0.1635...             9\n",
       "4558   [[239.3, 3219.52936855, -24.8032633276, -0.753...             9\n",
       "4559   [[104.4, 4927.94947637, -4.02758534422, 0.0481...             9\n",
       "\n",
       "[4560 rows x 2 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_df, y_array = problem.get_train_data()\n",
    "X_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['1111', '1111', '1111', ..., '4222', '4222', '4222'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training data has two columns. `trajectory_id` is the id of the long trajectory from which we sampled 1000 consecutive trajectory points at random starting times. To have enough training data, we took 10 (possibly overlapping) samples from each long trajectory. The trajectory itself is found in the `data` column. Each instance is represented by a 2D numpy array where each column is a trajectory feature and each row is a time point. Let us convert a single instance into a pandas table to visualize it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>T</th>\n",
       "      <th>X</th>\n",
       "      <th>Vx</th>\n",
       "      <th>Ax</th>\n",
       "      <th>Jx</th>\n",
       "      <th>Y</th>\n",
       "      <th>Vy</th>\n",
       "      <th>Ay</th>\n",
       "      <th>Jy</th>\n",
       "      <th>Z</th>\n",
       "      <th>Vz</th>\n",
       "      <th>Az</th>\n",
       "      <th>Jz</th>\n",
       "      <th>U2</th>\n",
       "      <th>C2</th>\n",
       "      <th>U3</th>\n",
       "      <th>C3</th>\n",
       "      <th>T3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>142.3</td>\n",
       "      <td>13065.207231</td>\n",
       "      <td>-37.348273</td>\n",
       "      <td>7.662910</td>\n",
       "      <td>0.558759</td>\n",
       "      <td>10648.565303</td>\n",
       "      <td>-81.879076</td>\n",
       "      <td>13.799308</td>\n",
       "      <td>1.480220</td>\n",
       "      <td>278.248441</td>\n",
       "      <td>0.275643</td>\n",
       "      <td>0.042042</td>\n",
       "      <td>-0.014298</td>\n",
       "      <td>89.994869</td>\n",
       "      <td>1.708234e-06</td>\n",
       "      <td>89.995291</td>\n",
       "      <td>0.000154</td>\n",
       "      <td>-0.000016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>142.4</td>\n",
       "      <td>13061.548323</td>\n",
       "      <td>-35.959268</td>\n",
       "      <td>7.698060</td>\n",
       "      <td>0.532016</td>\n",
       "      <td>10640.499794</td>\n",
       "      <td>-79.613038</td>\n",
       "      <td>13.917953</td>\n",
       "      <td>1.442313</td>\n",
       "      <td>278.276444</td>\n",
       "      <td>0.283573</td>\n",
       "      <td>0.040485</td>\n",
       "      <td>-0.014462</td>\n",
       "      <td>87.357340</td>\n",
       "      <td>1.929820e-06</td>\n",
       "      <td>87.357800</td>\n",
       "      <td>0.000169</td>\n",
       "      <td>-0.000015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>142.5</td>\n",
       "      <td>13058.059018</td>\n",
       "      <td>-34.561718</td>\n",
       "      <td>7.729593</td>\n",
       "      <td>0.505136</td>\n",
       "      <td>10632.705141</td>\n",
       "      <td>-77.324428</td>\n",
       "      <td>14.031347</td>\n",
       "      <td>1.404058</td>\n",
       "      <td>278.305394</td>\n",
       "      <td>0.291147</td>\n",
       "      <td>0.038914</td>\n",
       "      <td>-0.014616</td>\n",
       "      <td>84.696987</td>\n",
       "      <td>2.190796e-06</td>\n",
       "      <td>84.697487</td>\n",
       "      <td>0.000186</td>\n",
       "      <td>-0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>142.6</td>\n",
       "      <td>13054.741148</td>\n",
       "      <td>-33.156487</td>\n",
       "      <td>7.757482</td>\n",
       "      <td>0.478135</td>\n",
       "      <td>10625.185351</td>\n",
       "      <td>-75.014413</td>\n",
       "      <td>14.139425</td>\n",
       "      <td>1.365475</td>\n",
       "      <td>278.335239</td>\n",
       "      <td>0.298362</td>\n",
       "      <td>0.037330</td>\n",
       "      <td>-0.014762</td>\n",
       "      <td>82.015333</td>\n",
       "      <td>2.499874e-06</td>\n",
       "      <td>82.015876</td>\n",
       "      <td>0.000206</td>\n",
       "      <td>-0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>142.7</td>\n",
       "      <td>13051.596381</td>\n",
       "      <td>-31.744444</td>\n",
       "      <td>7.781706</td>\n",
       "      <td>0.451030</td>\n",
       "      <td>10617.944223</td>\n",
       "      <td>-72.684172</td>\n",
       "      <td>14.242129</td>\n",
       "      <td>1.326588</td>\n",
       "      <td>278.365926</td>\n",
       "      <td>0.305219</td>\n",
       "      <td>0.035735</td>\n",
       "      <td>-0.014899</td>\n",
       "      <td>79.313924</td>\n",
       "      <td>2.868085e-06</td>\n",
       "      <td>79.314511</td>\n",
       "      <td>0.000228</td>\n",
       "      <td>-0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>142.8</td>\n",
       "      <td>13048.626221</td>\n",
       "      <td>-30.326460</td>\n",
       "      <td>7.802246</td>\n",
       "      <td>0.423839</td>\n",
       "      <td>10610.985345</td>\n",
       "      <td>-70.334893</td>\n",
       "      <td>14.339404</td>\n",
       "      <td>1.287418</td>\n",
       "      <td>278.397402</td>\n",
       "      <td>0.311716</td>\n",
       "      <td>0.034130</td>\n",
       "      <td>-0.015027</td>\n",
       "      <td>76.594330</td>\n",
       "      <td>3.309504e-06</td>\n",
       "      <td>76.594965</td>\n",
       "      <td>0.000254</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>142.9</td>\n",
       "      <td>13045.832000</td>\n",
       "      <td>-28.903414</td>\n",
       "      <td>7.819086</td>\n",
       "      <td>0.396578</td>\n",
       "      <td>10604.312091</td>\n",
       "      <td>-67.967776</td>\n",
       "      <td>14.431199</td>\n",
       "      <td>1.247987</td>\n",
       "      <td>278.429614</td>\n",
       "      <td>0.317856</td>\n",
       "      <td>0.032515</td>\n",
       "      <td>-0.015146</td>\n",
       "      <td>73.858147</td>\n",
       "      <td>3.842247e-06</td>\n",
       "      <td>73.858831</td>\n",
       "      <td>0.000284</td>\n",
       "      <td>-0.000012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>143.0</td>\n",
       "      <td>13043.214882</td>\n",
       "      <td>-27.476185</td>\n",
       "      <td>7.832214</td>\n",
       "      <td>0.369263</td>\n",
       "      <td>10597.927615</td>\n",
       "      <td>-65.584024</td>\n",
       "      <td>14.517467</td>\n",
       "      <td>1.208315</td>\n",
       "      <td>278.462509</td>\n",
       "      <td>0.323637</td>\n",
       "      <td>0.030893</td>\n",
       "      <td>-0.015256</td>\n",
       "      <td>71.106997</td>\n",
       "      <td>4.489832e-06</td>\n",
       "      <td>71.107733</td>\n",
       "      <td>0.000320</td>\n",
       "      <td>-0.000011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>143.1</td>\n",
       "      <td>13040.775853</td>\n",
       "      <td>-26.045654</td>\n",
       "      <td>7.841621</td>\n",
       "      <td>0.341912</td>\n",
       "      <td>10591.834848</td>\n",
       "      <td>-63.184851</td>\n",
       "      <td>14.598164</td>\n",
       "      <td>1.168423</td>\n",
       "      <td>278.496031</td>\n",
       "      <td>0.329061</td>\n",
       "      <td>0.029265</td>\n",
       "      <td>-0.015359</td>\n",
       "      <td>68.342531</td>\n",
       "      <td>5.283088e-06</td>\n",
       "      <td>68.343323</td>\n",
       "      <td>0.000362</td>\n",
       "      <td>-0.000010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>143.2</td>\n",
       "      <td>13038.515727</td>\n",
       "      <td>-24.612703</td>\n",
       "      <td>7.847303</td>\n",
       "      <td>0.314539</td>\n",
       "      <td>10586.036493</td>\n",
       "      <td>-60.771474</td>\n",
       "      <td>14.673252</td>\n",
       "      <td>1.128332</td>\n",
       "      <td>278.530128</td>\n",
       "      <td>0.334128</td>\n",
       "      <td>0.027631</td>\n",
       "      <td>-0.015452</td>\n",
       "      <td>65.566434</td>\n",
       "      <td>6.262841e-06</td>\n",
       "      <td>65.567285</td>\n",
       "      <td>0.000411</td>\n",
       "      <td>-0.000010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>143.3</td>\n",
       "      <td>13036.435139</td>\n",
       "      <td>-23.178215</td>\n",
       "      <td>7.849256</td>\n",
       "      <td>0.287160</td>\n",
       "      <td>10580.535029</td>\n",
       "      <td>-58.345115</td>\n",
       "      <td>14.742694</td>\n",
       "      <td>1.088062</td>\n",
       "      <td>278.564745</td>\n",
       "      <td>0.338842</td>\n",
       "      <td>0.025994</td>\n",
       "      <td>-0.015538</td>\n",
       "      <td>62.780428</td>\n",
       "      <td>7.483758e-06</td>\n",
       "      <td>62.781343</td>\n",
       "      <td>0.000471</td>\n",
       "      <td>-0.000009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>143.4</td>\n",
       "      <td>13034.534547</td>\n",
       "      <td>-21.743073</td>\n",
       "      <td>7.847482</td>\n",
       "      <td>0.259792</td>\n",
       "      <td>10575.332698</td>\n",
       "      <td>-55.906999</td>\n",
       "      <td>14.806461</td>\n",
       "      <td>1.047633</td>\n",
       "      <td>278.599828</td>\n",
       "      <td>0.343202</td>\n",
       "      <td>0.024353</td>\n",
       "      <td>-0.015616</td>\n",
       "      <td>59.986280</td>\n",
       "      <td>9.019915e-06</td>\n",
       "      <td>59.987261</td>\n",
       "      <td>0.000542</td>\n",
       "      <td>-0.000009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>143.5</td>\n",
       "      <td>13032.814229</td>\n",
       "      <td>-20.308156</td>\n",
       "      <td>7.841987</td>\n",
       "      <td>0.232449</td>\n",
       "      <td>10570.431514</td>\n",
       "      <td>-53.458353</td>\n",
       "      <td>14.864525</td>\n",
       "      <td>1.007063</td>\n",
       "      <td>278.635325</td>\n",
       "      <td>0.347213</td>\n",
       "      <td>0.022712</td>\n",
       "      <td>-0.015685</td>\n",
       "      <td>57.185808</td>\n",
       "      <td>1.097299e-05</td>\n",
       "      <td>57.186862</td>\n",
       "      <td>0.000629</td>\n",
       "      <td>-0.000008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>143.6</td>\n",
       "      <td>13031.274284</td>\n",
       "      <td>-18.874343</td>\n",
       "      <td>7.832777</td>\n",
       "      <td>0.205147</td>\n",
       "      <td>10565.833255</td>\n",
       "      <td>-51.000405</td>\n",
       "      <td>14.916862</td>\n",
       "      <td>0.966371</td>\n",
       "      <td>278.671181</td>\n",
       "      <td>0.350875</td>\n",
       "      <td>0.021069</td>\n",
       "      <td>-0.015747</td>\n",
       "      <td>54.380899</td>\n",
       "      <td>1.348449e-05</td>\n",
       "      <td>54.382031</td>\n",
       "      <td>0.000735</td>\n",
       "      <td>-0.000008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>143.7</td>\n",
       "      <td>13029.914631</td>\n",
       "      <td>-17.442509</td>\n",
       "      <td>7.819866</td>\n",
       "      <td>0.177900</td>\n",
       "      <td>10561.539464</td>\n",
       "      <td>-48.534384</td>\n",
       "      <td>14.963453</td>\n",
       "      <td>0.925577</td>\n",
       "      <td>278.707345</td>\n",
       "      <td>0.354192</td>\n",
       "      <td>0.019428</td>\n",
       "      <td>-0.015801</td>\n",
       "      <td>51.573516</td>\n",
       "      <td>1.675442e-05</td>\n",
       "      <td>51.574733</td>\n",
       "      <td>0.000866</td>\n",
       "      <td>-0.000008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>143.8</td>\n",
       "      <td>13028.735006</td>\n",
       "      <td>-16.013525</td>\n",
       "      <td>7.803267</td>\n",
       "      <td>0.150724</td>\n",
       "      <td>10557.551445</td>\n",
       "      <td>-46.061519</td>\n",
       "      <td>15.004284</td>\n",
       "      <td>0.884699</td>\n",
       "      <td>278.743764</td>\n",
       "      <td>0.357168</td>\n",
       "      <td>0.017789</td>\n",
       "      <td>-0.015848</td>\n",
       "      <td>48.765731</td>\n",
       "      <td>2.107014e-05</td>\n",
       "      <td>48.767039</td>\n",
       "      <td>0.001029</td>\n",
       "      <td>-0.000007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>143.9</td>\n",
       "      <td>13027.734968</td>\n",
       "      <td>-14.588257</td>\n",
       "      <td>7.783001</td>\n",
       "      <td>0.123634</td>\n",
       "      <td>10553.870271</td>\n",
       "      <td>-43.583035</td>\n",
       "      <td>15.039343</td>\n",
       "      <td>0.843754</td>\n",
       "      <td>278.780386</td>\n",
       "      <td>0.359805</td>\n",
       "      <td>0.016152</td>\n",
       "      <td>-0.015888</td>\n",
       "      <td>45.959745</td>\n",
       "      <td>2.685214e-05</td>\n",
       "      <td>45.961154</td>\n",
       "      <td>0.001236</td>\n",
       "      <td>-0.000007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>144.0</td>\n",
       "      <td>13026.913896</td>\n",
       "      <td>-13.167565</td>\n",
       "      <td>7.759087</td>\n",
       "      <td>0.096643</td>\n",
       "      <td>10550.496772</td>\n",
       "      <td>-41.100156</td>\n",
       "      <td>15.068623</td>\n",
       "      <td>0.802760</td>\n",
       "      <td>278.817160</td>\n",
       "      <td>0.362107</td>\n",
       "      <td>0.014520</td>\n",
       "      <td>-0.015920</td>\n",
       "      <td>43.157938</td>\n",
       "      <td>3.472815e-05</td>\n",
       "      <td>43.159457</td>\n",
       "      <td>0.001501</td>\n",
       "      <td>-0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>144.1</td>\n",
       "      <td>13026.270988</td>\n",
       "      <td>-11.752305</td>\n",
       "      <td>7.731553</td>\n",
       "      <td>0.069766</td>\n",
       "      <td>10547.431546</td>\n",
       "      <td>-38.614101</td>\n",
       "      <td>15.092119</td>\n",
       "      <td>0.761734</td>\n",
       "      <td>278.854037</td>\n",
       "      <td>0.364079</td>\n",
       "      <td>0.012893</td>\n",
       "      <td>-0.015945</td>\n",
       "      <td>40.362922</td>\n",
       "      <td>4.565620e-05</td>\n",
       "      <td>40.364564</td>\n",
       "      <td>0.001845</td>\n",
       "      <td>-0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>144.2</td>\n",
       "      <td>13025.805267</td>\n",
       "      <td>-10.343322</td>\n",
       "      <td>7.700427</td>\n",
       "      <td>0.043017</td>\n",
       "      <td>10544.674953</td>\n",
       "      <td>-36.126085</td>\n",
       "      <td>15.109834</td>\n",
       "      <td>0.720695</td>\n",
       "      <td>278.890967</td>\n",
       "      <td>0.365724</td>\n",
       "      <td>0.011273</td>\n",
       "      <td>-0.015963</td>\n",
       "      <td>37.577631</td>\n",
       "      <td>6.113466e-05</td>\n",
       "      <td>37.579410</td>\n",
       "      <td>0.002300</td>\n",
       "      <td>-0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>144.3</td>\n",
       "      <td>13025.515575</td>\n",
       "      <td>-8.941457</td>\n",
       "      <td>7.665740</td>\n",
       "      <td>0.016410</td>\n",
       "      <td>10542.227118</td>\n",
       "      <td>-33.637318</td>\n",
       "      <td>15.121770</td>\n",
       "      <td>0.679657</td>\n",
       "      <td>278.927902</td>\n",
       "      <td>0.367047</td>\n",
       "      <td>0.009659</td>\n",
       "      <td>-0.015975</td>\n",
       "      <td>34.805442</td>\n",
       "      <td>8.357160e-05</td>\n",
       "      <td>34.807378</td>\n",
       "      <td>0.002912</td>\n",
       "      <td>-0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>144.4</td>\n",
       "      <td>13025.400582</td>\n",
       "      <td>-7.547539</td>\n",
       "      <td>7.627530</td>\n",
       "      <td>-0.010040</td>\n",
       "      <td>10540.087934</td>\n",
       "      <td>-31.149003</td>\n",
       "      <td>15.127937</td>\n",
       "      <td>0.638639</td>\n",
       "      <td>278.964794</td>\n",
       "      <td>0.368053</td>\n",
       "      <td>0.008054</td>\n",
       "      <td>-0.015980</td>\n",
       "      <td>32.050362</td>\n",
       "      <td>1.169561e-04</td>\n",
       "      <td>32.052475</td>\n",
       "      <td>0.003753</td>\n",
       "      <td>-0.000005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>144.5</td>\n",
       "      <td>13025.458782</td>\n",
       "      <td>-6.162389</td>\n",
       "      <td>7.585836</td>\n",
       "      <td>-0.036321</td>\n",
       "      <td>10538.257057</td>\n",
       "      <td>-28.662335</td>\n",
       "      <td>15.128347</td>\n",
       "      <td>0.597656</td>\n",
       "      <td>279.001598</td>\n",
       "      <td>0.368746</td>\n",
       "      <td>0.006458</td>\n",
       "      <td>-0.015979</td>\n",
       "      <td>29.317307</td>\n",
       "      <td>1.681237e-04</td>\n",
       "      <td>29.319626</td>\n",
       "      <td>0.004934</td>\n",
       "      <td>-0.000005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>144.6</td>\n",
       "      <td>13025.688498</td>\n",
       "      <td>-4.786818</td>\n",
       "      <td>7.540699</td>\n",
       "      <td>-0.062418</td>\n",
       "      <td>10536.733917</td>\n",
       "      <td>-26.178501</td>\n",
       "      <td>15.123016</td>\n",
       "      <td>0.556723</td>\n",
       "      <td>279.038267</td>\n",
       "      <td>0.369133</td>\n",
       "      <td>0.004872</td>\n",
       "      <td>-0.015971</td>\n",
       "      <td>26.612545</td>\n",
       "      <td>2.492355e-04</td>\n",
       "      <td>26.615105</td>\n",
       "      <td>0.006639</td>\n",
       "      <td>-0.000005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>144.7</td>\n",
       "      <td>13026.087883</td>\n",
       "      <td>-3.421624</td>\n",
       "      <td>7.492167</td>\n",
       "      <td>-0.088318</td>\n",
       "      <td>10535.517711</td>\n",
       "      <td>-23.698680</td>\n",
       "      <td>15.111963</td>\n",
       "      <td>0.515858</td>\n",
       "      <td>279.074759</td>\n",
       "      <td>0.369217</td>\n",
       "      <td>0.003298</td>\n",
       "      <td>-0.015957</td>\n",
       "      <td>23.944414</td>\n",
       "      <td>3.828477e-04</td>\n",
       "      <td>23.947260</td>\n",
       "      <td>0.009175</td>\n",
       "      <td>-0.000005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>144.8</td>\n",
       "      <td>13026.654922</td>\n",
       "      <td>-2.067595</td>\n",
       "      <td>7.440289</td>\n",
       "      <td>-0.114007</td>\n",
       "      <td>10534.607414</td>\n",
       "      <td>-21.224040</td>\n",
       "      <td>15.095212</td>\n",
       "      <td>0.475074</td>\n",
       "      <td>279.111031</td>\n",
       "      <td>0.369006</td>\n",
       "      <td>0.001735</td>\n",
       "      <td>-0.015938</td>\n",
       "      <td>21.324512</td>\n",
       "      <td>6.127281e-04</td>\n",
       "      <td>21.327704</td>\n",
       "      <td>0.013076</td>\n",
       "      <td>-0.000005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>144.9</td>\n",
       "      <td>13027.387436</td>\n",
       "      <td>-0.725504</td>\n",
       "      <td>7.385118</td>\n",
       "      <td>-0.139474</td>\n",
       "      <td>10534.001771</td>\n",
       "      <td>-18.755736</td>\n",
       "      <td>15.072791</td>\n",
       "      <td>0.434388</td>\n",
       "      <td>279.147040</td>\n",
       "      <td>0.368504</td>\n",
       "      <td>0.000184</td>\n",
       "      <td>-0.015912</td>\n",
       "      <td>18.769763</td>\n",
       "      <td>1.027875e-03</td>\n",
       "      <td>18.773380</td>\n",
       "      <td>0.019305</td>\n",
       "      <td>-0.000004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>145.0</td>\n",
       "      <td>13028.283084</td>\n",
       "      <td>0.603889</td>\n",
       "      <td>7.326710</td>\n",
       "      <td>-0.164703</td>\n",
       "      <td>10533.699312</td>\n",
       "      <td>-16.294915</td>\n",
       "      <td>15.044730</td>\n",
       "      <td>0.393815</td>\n",
       "      <td>279.182748</td>\n",
       "      <td>0.367718</td>\n",
       "      <td>-0.001352</td>\n",
       "      <td>-0.015882</td>\n",
       "      <td>16.306101</td>\n",
       "      <td>1.817243e-03</td>\n",
       "      <td>16.310247</td>\n",
       "      <td>0.029643</td>\n",
       "      <td>-0.000004</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>145.1</td>\n",
       "      <td>13047.789648</td>\n",
       "      <td>7.549537</td>\n",
       "      <td>5.934406</td>\n",
       "      <td>-0.564544</td>\n",
       "      <td>10571.103173</td>\n",
       "      <td>-2.429443</td>\n",
       "      <td>12.313257</td>\n",
       "      <td>-0.406600</td>\n",
       "      <td>278.931022</td>\n",
       "      <td>0.279053</td>\n",
       "      <td>0.017832</td>\n",
       "      <td>-0.010012</td>\n",
       "      <td>7.930807</td>\n",
       "      <td>2.714192e-02</td>\n",
       "      <td>7.935714</td>\n",
       "      <td>0.214992</td>\n",
       "      <td>0.000023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>145.2</td>\n",
       "      <td>13049.703302</td>\n",
       "      <td>8.922351</td>\n",
       "      <td>5.836189</td>\n",
       "      <td>-0.595064</td>\n",
       "      <td>10572.926944</td>\n",
       "      <td>0.188983</td>\n",
       "      <td>12.198431</td>\n",
       "      <td>-0.460802</td>\n",
       "      <td>278.951377</td>\n",
       "      <td>0.275495</td>\n",
       "      <td>0.017113</td>\n",
       "      <td>-0.009806</td>\n",
       "      <td>8.924352</td>\n",
       "      <td>1.698451e-02</td>\n",
       "      <td>8.928603</td>\n",
       "      <td>0.151447</td>\n",
       "      <td>0.000023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>970</th>\n",
       "      <td>239.3</td>\n",
       "      <td>9693.412351</td>\n",
       "      <td>36.407598</td>\n",
       "      <td>7.225480</td>\n",
       "      <td>-0.883832</td>\n",
       "      <td>6854.578037</td>\n",
       "      <td>36.305154</td>\n",
       "      <td>7.688828</td>\n",
       "      <td>-0.823186</td>\n",
       "      <td>329.776813</td>\n",
       "      <td>-0.251085</td>\n",
       "      <td>-0.067414</td>\n",
       "      <td>0.005963</td>\n",
       "      <td>51.415731</td>\n",
       "      <td>2.519794e-06</td>\n",
       "      <td>51.416344</td>\n",
       "      <td>0.000130</td>\n",
       "      <td>0.000112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>239.4</td>\n",
       "      <td>9697.852807</td>\n",
       "      <td>37.297321</td>\n",
       "      <td>7.082587</td>\n",
       "      <td>-0.901375</td>\n",
       "      <td>6859.003217</td>\n",
       "      <td>37.239862</td>\n",
       "      <td>7.552525</td>\n",
       "      <td>-0.840559</td>\n",
       "      <td>329.742808</td>\n",
       "      <td>-0.259720</td>\n",
       "      <td>-0.066207</td>\n",
       "      <td>0.006159</td>\n",
       "      <td>52.705763</td>\n",
       "      <td>2.324094e-06</td>\n",
       "      <td>52.706403</td>\n",
       "      <td>0.000123</td>\n",
       "      <td>0.000110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>972</th>\n",
       "      <td>239.5</td>\n",
       "      <td>9702.362928</td>\n",
       "      <td>38.169822</td>\n",
       "      <td>6.938616</td>\n",
       "      <td>-0.918740</td>\n",
       "      <td>6863.502840</td>\n",
       "      <td>38.158050</td>\n",
       "      <td>7.415149</td>\n",
       "      <td>-0.857759</td>\n",
       "      <td>329.708087</td>\n",
       "      <td>-0.268217</td>\n",
       "      <td>-0.064983</td>\n",
       "      <td>0.006356</td>\n",
       "      <td>53.971957</td>\n",
       "      <td>2.153209e-06</td>\n",
       "      <td>53.972623</td>\n",
       "      <td>0.000116</td>\n",
       "      <td>0.000107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>973</th>\n",
       "      <td>239.6</td>\n",
       "      <td>9706.941762</td>\n",
       "      <td>39.025339</td>\n",
       "      <td>6.793548</td>\n",
       "      <td>-0.935947</td>\n",
       "      <td>6868.076022</td>\n",
       "      <td>39.059956</td>\n",
       "      <td>7.276681</td>\n",
       "      <td>-0.874803</td>\n",
       "      <td>329.672661</td>\n",
       "      <td>-0.276577</td>\n",
       "      <td>-0.063742</td>\n",
       "      <td>0.006552</td>\n",
       "      <td>55.214647</td>\n",
       "      <td>2.003297e-06</td>\n",
       "      <td>55.215340</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>0.000104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>239.7</td>\n",
       "      <td>9711.588388</td>\n",
       "      <td>39.864099</td>\n",
       "      <td>6.647360</td>\n",
       "      <td>-0.953010</td>\n",
       "      <td>6872.721906</td>\n",
       "      <td>39.945803</td>\n",
       "      <td>7.137099</td>\n",
       "      <td>-0.891708</td>\n",
       "      <td>329.636540</td>\n",
       "      <td>-0.284800</td>\n",
       "      <td>-0.062483</td>\n",
       "      <td>0.006747</td>\n",
       "      <td>56.434152</td>\n",
       "      <td>1.871221e-06</td>\n",
       "      <td>56.434871</td>\n",
       "      <td>0.000106</td>\n",
       "      <td>0.000102</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>975</th>\n",
       "      <td>239.8</td>\n",
       "      <td>9716.301914</td>\n",
       "      <td>40.686316</td>\n",
       "      <td>6.500030</td>\n",
       "      <td>-0.969949</td>\n",
       "      <td>6877.439667</td>\n",
       "      <td>40.815806</td>\n",
       "      <td>6.996378</td>\n",
       "      <td>-0.908490</td>\n",
       "      <td>329.599734</td>\n",
       "      <td>-0.292888</td>\n",
       "      <td>-0.061208</td>\n",
       "      <td>0.006943</td>\n",
       "      <td>57.630776</td>\n",
       "      <td>1.754397e-06</td>\n",
       "      <td>57.631520</td>\n",
       "      <td>0.000101</td>\n",
       "      <td>0.000099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>976</th>\n",
       "      <td>239.9</td>\n",
       "      <td>9721.081473</td>\n",
       "      <td>41.492198</td>\n",
       "      <td>6.351530</td>\n",
       "      <td>-0.986779</td>\n",
       "      <td>6882.228503</td>\n",
       "      <td>41.670170</td>\n",
       "      <td>6.854492</td>\n",
       "      <td>-0.925167</td>\n",
       "      <td>329.562252</td>\n",
       "      <td>-0.300843</td>\n",
       "      <td>-0.059915</td>\n",
       "      <td>0.007138</td>\n",
       "      <td>58.804809</td>\n",
       "      <td>1.650687e-06</td>\n",
       "      <td>58.805579</td>\n",
       "      <td>0.000097</td>\n",
       "      <td>0.000096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>977</th>\n",
       "      <td>240.0</td>\n",
       "      <td>9725.926229</td>\n",
       "      <td>42.281940</td>\n",
       "      <td>6.201833</td>\n",
       "      <td>-1.003517</td>\n",
       "      <td>6887.087641</td>\n",
       "      <td>42.509089</td>\n",
       "      <td>6.711412</td>\n",
       "      <td>-0.941756</td>\n",
       "      <td>329.524104</td>\n",
       "      <td>-0.308664</td>\n",
       "      <td>-0.058604</td>\n",
       "      <td>0.007334</td>\n",
       "      <td>59.956527</td>\n",
       "      <td>1.558313e-06</td>\n",
       "      <td>59.957321</td>\n",
       "      <td>0.000093</td>\n",
       "      <td>0.000094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>978</th>\n",
       "      <td>240.1</td>\n",
       "      <td>9730.835369</td>\n",
       "      <td>43.055728</td>\n",
       "      <td>6.050908</td>\n",
       "      <td>-1.020180</td>\n",
       "      <td>6892.016334</td>\n",
       "      <td>43.332748</td>\n",
       "      <td>6.567109</td>\n",
       "      <td>-0.958271</td>\n",
       "      <td>329.485299</td>\n",
       "      <td>-0.316354</td>\n",
       "      <td>-0.057276</td>\n",
       "      <td>0.007530</td>\n",
       "      <td>61.086191</td>\n",
       "      <td>1.475781e-06</td>\n",
       "      <td>61.087010</td>\n",
       "      <td>0.000090</td>\n",
       "      <td>0.000091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>979</th>\n",
       "      <td>240.2</td>\n",
       "      <td>9735.808108</td>\n",
       "      <td>43.813739</td>\n",
       "      <td>5.898723</td>\n",
       "      <td>-1.036782</td>\n",
       "      <td>6897.013858</td>\n",
       "      <td>44.141322</td>\n",
       "      <td>6.421548</td>\n",
       "      <td>-0.974731</td>\n",
       "      <td>329.445843</td>\n",
       "      <td>-0.323913</td>\n",
       "      <td>-0.055930</td>\n",
       "      <td>0.007727</td>\n",
       "      <td>62.194051</td>\n",
       "      <td>1.401839e-06</td>\n",
       "      <td>62.194895</td>\n",
       "      <td>0.000087</td>\n",
       "      <td>0.000089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>980</th>\n",
       "      <td>240.3</td>\n",
       "      <td>9740.843687</td>\n",
       "      <td>44.556140</td>\n",
       "      <td>5.745244</td>\n",
       "      <td>-1.053341</td>\n",
       "      <td>6902.079518</td>\n",
       "      <td>44.934978</td>\n",
       "      <td>6.274696</td>\n",
       "      <td>-0.991149</td>\n",
       "      <td>329.405745</td>\n",
       "      <td>-0.331342</td>\n",
       "      <td>-0.054565</td>\n",
       "      <td>0.007923</td>\n",
       "      <td>63.280343</td>\n",
       "      <td>1.335425e-06</td>\n",
       "      <td>63.281211</td>\n",
       "      <td>0.000085</td>\n",
       "      <td>0.000086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>981</th>\n",
       "      <td>240.4</td>\n",
       "      <td>9745.941372</td>\n",
       "      <td>45.283089</td>\n",
       "      <td>5.590432</td>\n",
       "      <td>-1.069872</td>\n",
       "      <td>6907.212644</td>\n",
       "      <td>45.713874</td>\n",
       "      <td>6.126515</td>\n",
       "      <td>-1.007542</td>\n",
       "      <td>329.365011</td>\n",
       "      <td>-0.338643</td>\n",
       "      <td>-0.053182</td>\n",
       "      <td>0.008121</td>\n",
       "      <td>64.345291</td>\n",
       "      <td>1.275636e-06</td>\n",
       "      <td>64.346182</td>\n",
       "      <td>0.000082</td>\n",
       "      <td>0.000084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>982</th>\n",
       "      <td>240.5</td>\n",
       "      <td>9751.100457</td>\n",
       "      <td>45.994738</td>\n",
       "      <td>5.434249</td>\n",
       "      <td>-1.086390</td>\n",
       "      <td>6912.412592</td>\n",
       "      <td>46.478157</td>\n",
       "      <td>5.976965</td>\n",
       "      <td>-1.023926</td>\n",
       "      <td>329.323648</td>\n",
       "      <td>-0.345815</td>\n",
       "      <td>-0.051779</td>\n",
       "      <td>0.008319</td>\n",
       "      <td>65.389104</td>\n",
       "      <td>1.221704e-06</td>\n",
       "      <td>65.390019</td>\n",
       "      <td>0.000080</td>\n",
       "      <td>0.000081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>983</th>\n",
       "      <td>240.6</td>\n",
       "      <td>9756.320264</td>\n",
       "      <td>46.691225</td>\n",
       "      <td>5.276653</td>\n",
       "      <td>-1.102911</td>\n",
       "      <td>6917.678745</td>\n",
       "      <td>47.227968</td>\n",
       "      <td>5.826005</td>\n",
       "      <td>-1.040316</td>\n",
       "      <td>329.281662</td>\n",
       "      <td>-0.352861</td>\n",
       "      <td>-0.050358</td>\n",
       "      <td>0.008518</td>\n",
       "      <td>66.411983</td>\n",
       "      <td>1.172969e-06</td>\n",
       "      <td>66.412920</td>\n",
       "      <td>0.000078</td>\n",
       "      <td>0.000079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>984</th>\n",
       "      <td>240.7</td>\n",
       "      <td>9761.600141</td>\n",
       "      <td>47.372685</td>\n",
       "      <td>5.117601</td>\n",
       "      <td>-1.119450</td>\n",
       "      <td>6923.010514</td>\n",
       "      <td>47.963439</td>\n",
       "      <td>5.673591</td>\n",
       "      <td>-1.056726</td>\n",
       "      <td>329.239058</td>\n",
       "      <td>-0.359780</td>\n",
       "      <td>-0.048916</td>\n",
       "      <td>0.008718</td>\n",
       "      <td>67.414113</td>\n",
       "      <td>1.128864e-06</td>\n",
       "      <td>67.415073</td>\n",
       "      <td>0.000076</td>\n",
       "      <td>0.000076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <td>240.8</td>\n",
       "      <td>9766.939466</td>\n",
       "      <td>48.039240</td>\n",
       "      <td>4.957046</td>\n",
       "      <td>-1.136022</td>\n",
       "      <td>6928.407340</td>\n",
       "      <td>48.684693</td>\n",
       "      <td>5.519677</td>\n",
       "      <td>-1.073173</td>\n",
       "      <td>329.195842</td>\n",
       "      <td>-0.366574</td>\n",
       "      <td>-0.047455</td>\n",
       "      <td>0.008919</td>\n",
       "      <td>68.395671</td>\n",
       "      <td>1.088900e-06</td>\n",
       "      <td>68.396654</td>\n",
       "      <td>0.000074</td>\n",
       "      <td>0.000074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>986</th>\n",
       "      <td>240.9</td>\n",
       "      <td>9772.337647</td>\n",
       "      <td>48.691006</td>\n",
       "      <td>4.794940</td>\n",
       "      <td>-1.152643</td>\n",
       "      <td>6933.868691</td>\n",
       "      <td>49.391845</td>\n",
       "      <td>5.364214</td>\n",
       "      <td>-1.089672</td>\n",
       "      <td>329.152016</td>\n",
       "      <td>-0.373243</td>\n",
       "      <td>-0.045973</td>\n",
       "      <td>0.009121</td>\n",
       "      <td>69.356820</td>\n",
       "      <td>1.052653e-06</td>\n",
       "      <td>69.357824</td>\n",
       "      <td>0.000073</td>\n",
       "      <td>0.000072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>987</th>\n",
       "      <td>241.0</td>\n",
       "      <td>9777.794121</td>\n",
       "      <td>49.328092</td>\n",
       "      <td>4.631232</td>\n",
       "      <td>-1.169327</td>\n",
       "      <td>6939.394068</td>\n",
       "      <td>50.085002</td>\n",
       "      <td>5.207149</td>\n",
       "      <td>-1.106236</td>\n",
       "      <td>329.107586</td>\n",
       "      <td>-0.379788</td>\n",
       "      <td>-0.044470</td>\n",
       "      <td>0.009325</td>\n",
       "      <td>70.297710</td>\n",
       "      <td>1.019754e-06</td>\n",
       "      <td>70.298736</td>\n",
       "      <td>0.000072</td>\n",
       "      <td>0.000070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>988</th>\n",
       "      <td>241.1</td>\n",
       "      <td>9783.308358</td>\n",
       "      <td>49.950594</td>\n",
       "      <td>4.465867</td>\n",
       "      <td>-1.186090</td>\n",
       "      <td>6944.983003</td>\n",
       "      <td>50.764262</td>\n",
       "      <td>5.048431</td>\n",
       "      <td>-1.122882</td>\n",
       "      <td>329.062555</td>\n",
       "      <td>-0.386210</td>\n",
       "      <td>-0.042945</td>\n",
       "      <td>0.009530</td>\n",
       "      <td>71.218482</td>\n",
       "      <td>9.898833e-07</td>\n",
       "      <td>71.219529</td>\n",
       "      <td>0.000071</td>\n",
       "      <td>0.000068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>989</th>\n",
       "      <td>241.2</td>\n",
       "      <td>9788.879862</td>\n",
       "      <td>50.558603</td>\n",
       "      <td>4.298789</td>\n",
       "      <td>-1.202946</td>\n",
       "      <td>6950.635060</td>\n",
       "      <td>51.429716</td>\n",
       "      <td>4.888000</td>\n",
       "      <td>-1.139625</td>\n",
       "      <td>329.016926</td>\n",
       "      <td>-0.392508</td>\n",
       "      <td>-0.041397</td>\n",
       "      <td>0.009737</td>\n",
       "      <td>72.119263</td>\n",
       "      <td>9.627609e-07</td>\n",
       "      <td>72.120331</td>\n",
       "      <td>0.000069</td>\n",
       "      <td>0.000066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>990</th>\n",
       "      <td>241.3</td>\n",
       "      <td>9794.508168</td>\n",
       "      <td>51.152202</td>\n",
       "      <td>4.129939</td>\n",
       "      <td>-1.219911</td>\n",
       "      <td>6956.349838</td>\n",
       "      <td>52.081445</td>\n",
       "      <td>4.725799</td>\n",
       "      <td>-1.156479</td>\n",
       "      <td>328.970700</td>\n",
       "      <td>-0.398684</td>\n",
       "      <td>-0.039827</td>\n",
       "      <td>0.009945</td>\n",
       "      <td>73.000169</td>\n",
       "      <td>9.381419e-07</td>\n",
       "      <td>73.001257</td>\n",
       "      <td>0.000068</td>\n",
       "      <td>0.000063</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>991</th>\n",
       "      <td>241.4</td>\n",
       "      <td>9800.192849</td>\n",
       "      <td>51.731461</td>\n",
       "      <td>3.959254</td>\n",
       "      <td>-1.236999</td>\n",
       "      <td>6962.126972</td>\n",
       "      <td>52.719521</td>\n",
       "      <td>4.561764</td>\n",
       "      <td>-1.173459</td>\n",
       "      <td>328.923880</td>\n",
       "      <td>-0.404738</td>\n",
       "      <td>-0.038234</td>\n",
       "      <td>0.010155</td>\n",
       "      <td>73.861303</td>\n",
       "      <td>9.158115e-07</td>\n",
       "      <td>73.862412</td>\n",
       "      <td>0.000068</td>\n",
       "      <td>0.000062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>992</th>\n",
       "      <td>241.5</td>\n",
       "      <td>9805.933513</td>\n",
       "      <td>52.296446</td>\n",
       "      <td>3.786669</td>\n",
       "      <td>-1.254226</td>\n",
       "      <td>6967.966134</td>\n",
       "      <td>53.344009</td>\n",
       "      <td>4.395830</td>\n",
       "      <td>-1.190581</td>\n",
       "      <td>328.876466</td>\n",
       "      <td>-0.410669</td>\n",
       "      <td>-0.036616</td>\n",
       "      <td>0.010367</td>\n",
       "      <td>74.702755</td>\n",
       "      <td>8.955813e-07</td>\n",
       "      <td>74.703883</td>\n",
       "      <td>0.000067</td>\n",
       "      <td>0.000060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>993</th>\n",
       "      <td>241.6</td>\n",
       "      <td>9811.729806</td>\n",
       "      <td>52.847210</td>\n",
       "      <td>3.612114</td>\n",
       "      <td>-1.271606</td>\n",
       "      <td>6973.867032</td>\n",
       "      <td>53.954964</td>\n",
       "      <td>4.227927</td>\n",
       "      <td>-1.207860</td>\n",
       "      <td>328.828460</td>\n",
       "      <td>-0.416479</td>\n",
       "      <td>-0.034973</td>\n",
       "      <td>0.010582</td>\n",
       "      <td>75.524603</td>\n",
       "      <td>8.772855e-07</td>\n",
       "      <td>75.525751</td>\n",
       "      <td>0.000066</td>\n",
       "      <td>0.000058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>994</th>\n",
       "      <td>241.7</td>\n",
       "      <td>9817.581412</td>\n",
       "      <td>53.383796</td>\n",
       "      <td>3.435518</td>\n",
       "      <td>-1.289155</td>\n",
       "      <td>6979.829414</td>\n",
       "      <td>54.552428</td>\n",
       "      <td>4.057984</td>\n",
       "      <td>-1.225310</td>\n",
       "      <td>328.779862</td>\n",
       "      <td>-0.422166</td>\n",
       "      <td>-0.033305</td>\n",
       "      <td>0.010798</td>\n",
       "      <td>76.326909</td>\n",
       "      <td>8.607785e-07</td>\n",
       "      <td>76.328077</td>\n",
       "      <td>0.000066</td>\n",
       "      <td>0.000056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>995</th>\n",
       "      <td>241.8</td>\n",
       "      <td>9823.488056</td>\n",
       "      <td>53.906240</td>\n",
       "      <td>3.256806</td>\n",
       "      <td>-1.306888</td>\n",
       "      <td>6985.853068</td>\n",
       "      <td>55.136438</td>\n",
       "      <td>3.885925</td>\n",
       "      <td>-1.242948</td>\n",
       "      <td>328.730672</td>\n",
       "      <td>-0.427732</td>\n",
       "      <td>-0.031610</td>\n",
       "      <td>0.011017</td>\n",
       "      <td>77.109723</td>\n",
       "      <td>8.459319e-07</td>\n",
       "      <td>77.110910</td>\n",
       "      <td>0.000065</td>\n",
       "      <td>0.000054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>996</th>\n",
       "      <td>241.9</td>\n",
       "      <td>9829.449502</td>\n",
       "      <td>54.414564</td>\n",
       "      <td>3.075898</td>\n",
       "      <td>-1.324821</td>\n",
       "      <td>6991.937826</td>\n",
       "      <td>55.707017</td>\n",
       "      <td>3.711670</td>\n",
       "      <td>-1.260788</td>\n",
       "      <td>328.680888</td>\n",
       "      <td>-0.433175</td>\n",
       "      <td>-0.029888</td>\n",
       "      <td>0.011239</td>\n",
       "      <td>77.873080</td>\n",
       "      <td>8.326329e-07</td>\n",
       "      <td>77.874285</td>\n",
       "      <td>0.000065</td>\n",
       "      <td>0.000053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>997</th>\n",
       "      <td>242.0</td>\n",
       "      <td>9835.465559</td>\n",
       "      <td>54.908782</td>\n",
       "      <td>2.892711</td>\n",
       "      <td>-1.342969</td>\n",
       "      <td>6998.083558</td>\n",
       "      <td>56.264180</td>\n",
       "      <td>3.535137</td>\n",
       "      <td>-1.278847</td>\n",
       "      <td>328.630509</td>\n",
       "      <td>-0.438496</td>\n",
       "      <td>-0.028137</td>\n",
       "      <td>0.011463</td>\n",
       "      <td>78.616997</td>\n",
       "      <td>8.207825e-07</td>\n",
       "      <td>78.618220</td>\n",
       "      <td>0.000065</td>\n",
       "      <td>0.000051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>998</th>\n",
       "      <td>242.1</td>\n",
       "      <td>9841.536079</td>\n",
       "      <td>55.388896</td>\n",
       "      <td>2.707158</td>\n",
       "      <td>-1.361348</td>\n",
       "      <td>7004.290183</td>\n",
       "      <td>56.807928</td>\n",
       "      <td>3.356238</td>\n",
       "      <td>-1.297139</td>\n",
       "      <td>328.579534</td>\n",
       "      <td>-0.443694</td>\n",
       "      <td>-0.026358</td>\n",
       "      <td>0.011690</td>\n",
       "      <td>79.341480</td>\n",
       "      <td>8.102936e-07</td>\n",
       "      <td>79.342721</td>\n",
       "      <td>0.000064</td>\n",
       "      <td>0.000049</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>999</th>\n",
       "      <td>242.2</td>\n",
       "      <td>9847.660952</td>\n",
       "      <td>55.854891</td>\n",
       "      <td>2.519150</td>\n",
       "      <td>-1.379973</td>\n",
       "      <td>7010.557658</td>\n",
       "      <td>57.338250</td>\n",
       "      <td>3.174882</td>\n",
       "      <td>-1.315681</td>\n",
       "      <td>328.527960</td>\n",
       "      <td>-0.448768</td>\n",
       "      <td>-0.024548</td>\n",
       "      <td>0.011921</td>\n",
       "      <td>80.046510</td>\n",
       "      <td>8.010903e-07</td>\n",
       "      <td>80.047768</td>\n",
       "      <td>0.000064</td>\n",
       "      <td>0.000048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1000 rows × 18 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         T             X         Vx        Ax        Jx             Y  \\\n",
       "0    142.3  13065.207231 -37.348273  7.662910  0.558759  10648.565303   \n",
       "1    142.4  13061.548323 -35.959268  7.698060  0.532016  10640.499794   \n",
       "2    142.5  13058.059018 -34.561718  7.729593  0.505136  10632.705141   \n",
       "3    142.6  13054.741148 -33.156487  7.757482  0.478135  10625.185351   \n",
       "4    142.7  13051.596381 -31.744444  7.781706  0.451030  10617.944223   \n",
       "5    142.8  13048.626221 -30.326460  7.802246  0.423839  10610.985345   \n",
       "6    142.9  13045.832000 -28.903414  7.819086  0.396578  10604.312091   \n",
       "7    143.0  13043.214882 -27.476185  7.832214  0.369263  10597.927615   \n",
       "8    143.1  13040.775853 -26.045654  7.841621  0.341912  10591.834848   \n",
       "9    143.2  13038.515727 -24.612703  7.847303  0.314539  10586.036493   \n",
       "10   143.3  13036.435139 -23.178215  7.849256  0.287160  10580.535029   \n",
       "11   143.4  13034.534547 -21.743073  7.847482  0.259792  10575.332698   \n",
       "12   143.5  13032.814229 -20.308156  7.841987  0.232449  10570.431514   \n",
       "13   143.6  13031.274284 -18.874343  7.832777  0.205147  10565.833255   \n",
       "14   143.7  13029.914631 -17.442509  7.819866  0.177900  10561.539464   \n",
       "15   143.8  13028.735006 -16.013525  7.803267  0.150724  10557.551445   \n",
       "16   143.9  13027.734968 -14.588257  7.783001  0.123634  10553.870271   \n",
       "17   144.0  13026.913896 -13.167565  7.759087  0.096643  10550.496772   \n",
       "18   144.1  13026.270988 -11.752305  7.731553  0.069766  10547.431546   \n",
       "19   144.2  13025.805267 -10.343322  7.700427  0.043017  10544.674953   \n",
       "20   144.3  13025.515575  -8.941457  7.665740  0.016410  10542.227118   \n",
       "21   144.4  13025.400582  -7.547539  7.627530 -0.010040  10540.087934   \n",
       "22   144.5  13025.458782  -6.162389  7.585836 -0.036321  10538.257057   \n",
       "23   144.6  13025.688498  -4.786818  7.540699 -0.062418  10536.733917   \n",
       "24   144.7  13026.087883  -3.421624  7.492167 -0.088318  10535.517711   \n",
       "25   144.8  13026.654922  -2.067595  7.440289 -0.114007  10534.607414   \n",
       "26   144.9  13027.387436  -0.725504  7.385118 -0.139474  10534.001771   \n",
       "27   145.0  13028.283084   0.603889  7.326710 -0.164703  10533.699312   \n",
       "28   145.1  13047.789648   7.549537  5.934406 -0.564544  10571.103173   \n",
       "29   145.2  13049.703302   8.922351  5.836189 -0.595064  10572.926944   \n",
       "..     ...           ...        ...       ...       ...           ...   \n",
       "970  239.3   9693.412351  36.407598  7.225480 -0.883832   6854.578037   \n",
       "971  239.4   9697.852807  37.297321  7.082587 -0.901375   6859.003217   \n",
       "972  239.5   9702.362928  38.169822  6.938616 -0.918740   6863.502840   \n",
       "973  239.6   9706.941762  39.025339  6.793548 -0.935947   6868.076022   \n",
       "974  239.7   9711.588388  39.864099  6.647360 -0.953010   6872.721906   \n",
       "975  239.8   9716.301914  40.686316  6.500030 -0.969949   6877.439667   \n",
       "976  239.9   9721.081473  41.492198  6.351530 -0.986779   6882.228503   \n",
       "977  240.0   9725.926229  42.281940  6.201833 -1.003517   6887.087641   \n",
       "978  240.1   9730.835369  43.055728  6.050908 -1.020180   6892.016334   \n",
       "979  240.2   9735.808108  43.813739  5.898723 -1.036782   6897.013858   \n",
       "980  240.3   9740.843687  44.556140  5.745244 -1.053341   6902.079518   \n",
       "981  240.4   9745.941372  45.283089  5.590432 -1.069872   6907.212644   \n",
       "982  240.5   9751.100457  45.994738  5.434249 -1.086390   6912.412592   \n",
       "983  240.6   9756.320264  46.691225  5.276653 -1.102911   6917.678745   \n",
       "984  240.7   9761.600141  47.372685  5.117601 -1.119450   6923.010514   \n",
       "985  240.8   9766.939466  48.039240  4.957046 -1.136022   6928.407340   \n",
       "986  240.9   9772.337647  48.691006  4.794940 -1.152643   6933.868691   \n",
       "987  241.0   9777.794121  49.328092  4.631232 -1.169327   6939.394068   \n",
       "988  241.1   9783.308358  49.950594  4.465867 -1.186090   6944.983003   \n",
       "989  241.2   9788.879862  50.558603  4.298789 -1.202946   6950.635060   \n",
       "990  241.3   9794.508168  51.152202  4.129939 -1.219911   6956.349838   \n",
       "991  241.4   9800.192849  51.731461  3.959254 -1.236999   6962.126972   \n",
       "992  241.5   9805.933513  52.296446  3.786669 -1.254226   6967.966134   \n",
       "993  241.6   9811.729806  52.847210  3.612114 -1.271606   6973.867032   \n",
       "994  241.7   9817.581412  53.383796  3.435518 -1.289155   6979.829414   \n",
       "995  241.8   9823.488056  53.906240  3.256806 -1.306888   6985.853068   \n",
       "996  241.9   9829.449502  54.414564  3.075898 -1.324821   6991.937826   \n",
       "997  242.0   9835.465559  54.908782  2.892711 -1.342969   6998.083558   \n",
       "998  242.1   9841.536079  55.388896  2.707158 -1.361348   7004.290183   \n",
       "999  242.2   9847.660952  55.854891  2.519150 -1.379973   7010.557658   \n",
       "\n",
       "            Vy         Ay        Jy           Z        Vz        Az        Jz  \\\n",
       "0   -81.879076  13.799308  1.480220  278.248441  0.275643  0.042042 -0.014298   \n",
       "1   -79.613038  13.917953  1.442313  278.276444  0.283573  0.040485 -0.014462   \n",
       "2   -77.324428  14.031347  1.404058  278.305394  0.291147  0.038914 -0.014616   \n",
       "3   -75.014413  14.139425  1.365475  278.335239  0.298362  0.037330 -0.014762   \n",
       "4   -72.684172  14.242129  1.326588  278.365926  0.305219  0.035735 -0.014899   \n",
       "5   -70.334893  14.339404  1.287418  278.397402  0.311716  0.034130 -0.015027   \n",
       "6   -67.967776  14.431199  1.247987  278.429614  0.317856  0.032515 -0.015146   \n",
       "7   -65.584024  14.517467  1.208315  278.462509  0.323637  0.030893 -0.015256   \n",
       "8   -63.184851  14.598164  1.168423  278.496031  0.329061  0.029265 -0.015359   \n",
       "9   -60.771474  14.673252  1.128332  278.530128  0.334128  0.027631 -0.015452   \n",
       "10  -58.345115  14.742694  1.088062  278.564745  0.338842  0.025994 -0.015538   \n",
       "11  -55.906999  14.806461  1.047633  278.599828  0.343202  0.024353 -0.015616   \n",
       "12  -53.458353  14.864525  1.007063  278.635325  0.347213  0.022712 -0.015685   \n",
       "13  -51.000405  14.916862  0.966371  278.671181  0.350875  0.021069 -0.015747   \n",
       "14  -48.534384  14.963453  0.925577  278.707345  0.354192  0.019428 -0.015801   \n",
       "15  -46.061519  15.004284  0.884699  278.743764  0.357168  0.017789 -0.015848   \n",
       "16  -43.583035  15.039343  0.843754  278.780386  0.359805  0.016152 -0.015888   \n",
       "17  -41.100156  15.068623  0.802760  278.817160  0.362107  0.014520 -0.015920   \n",
       "18  -38.614101  15.092119  0.761734  278.854037  0.364079  0.012893 -0.015945   \n",
       "19  -36.126085  15.109834  0.720695  278.890967  0.365724  0.011273 -0.015963   \n",
       "20  -33.637318  15.121770  0.679657  278.927902  0.367047  0.009659 -0.015975   \n",
       "21  -31.149003  15.127937  0.638639  278.964794  0.368053  0.008054 -0.015980   \n",
       "22  -28.662335  15.128347  0.597656  279.001598  0.368746  0.006458 -0.015979   \n",
       "23  -26.178501  15.123016  0.556723  279.038267  0.369133  0.004872 -0.015971   \n",
       "24  -23.698680  15.111963  0.515858  279.074759  0.369217  0.003298 -0.015957   \n",
       "25  -21.224040  15.095212  0.475074  279.111031  0.369006  0.001735 -0.015938   \n",
       "26  -18.755736  15.072791  0.434388  279.147040  0.368504  0.000184 -0.015912   \n",
       "27  -16.294915  15.044730  0.393815  279.182748  0.367718 -0.001352 -0.015882   \n",
       "28   -2.429443  12.313257 -0.406600  278.931022  0.279053  0.017832 -0.010012   \n",
       "29    0.188983  12.198431 -0.460802  278.951377  0.275495  0.017113 -0.009806   \n",
       "..         ...        ...       ...         ...       ...       ...       ...   \n",
       "970  36.305154   7.688828 -0.823186  329.776813 -0.251085 -0.067414  0.005963   \n",
       "971  37.239862   7.552525 -0.840559  329.742808 -0.259720 -0.066207  0.006159   \n",
       "972  38.158050   7.415149 -0.857759  329.708087 -0.268217 -0.064983  0.006356   \n",
       "973  39.059956   7.276681 -0.874803  329.672661 -0.276577 -0.063742  0.006552   \n",
       "974  39.945803   7.137099 -0.891708  329.636540 -0.284800 -0.062483  0.006747   \n",
       "975  40.815806   6.996378 -0.908490  329.599734 -0.292888 -0.061208  0.006943   \n",
       "976  41.670170   6.854492 -0.925167  329.562252 -0.300843 -0.059915  0.007138   \n",
       "977  42.509089   6.711412 -0.941756  329.524104 -0.308664 -0.058604  0.007334   \n",
       "978  43.332748   6.567109 -0.958271  329.485299 -0.316354 -0.057276  0.007530   \n",
       "979  44.141322   6.421548 -0.974731  329.445843 -0.323913 -0.055930  0.007727   \n",
       "980  44.934978   6.274696 -0.991149  329.405745 -0.331342 -0.054565  0.007923   \n",
       "981  45.713874   6.126515 -1.007542  329.365011 -0.338643 -0.053182  0.008121   \n",
       "982  46.478157   5.976965 -1.023926  329.323648 -0.345815 -0.051779  0.008319   \n",
       "983  47.227968   5.826005 -1.040316  329.281662 -0.352861 -0.050358  0.008518   \n",
       "984  47.963439   5.673591 -1.056726  329.239058 -0.359780 -0.048916  0.008718   \n",
       "985  48.684693   5.519677 -1.073173  329.195842 -0.366574 -0.047455  0.008919   \n",
       "986  49.391845   5.364214 -1.089672  329.152016 -0.373243 -0.045973  0.009121   \n",
       "987  50.085002   5.207149 -1.106236  329.107586 -0.379788 -0.044470  0.009325   \n",
       "988  50.764262   5.048431 -1.122882  329.062555 -0.386210 -0.042945  0.009530   \n",
       "989  51.429716   4.888000 -1.139625  329.016926 -0.392508 -0.041397  0.009737   \n",
       "990  52.081445   4.725799 -1.156479  328.970700 -0.398684 -0.039827  0.009945   \n",
       "991  52.719521   4.561764 -1.173459  328.923880 -0.404738 -0.038234  0.010155   \n",
       "992  53.344009   4.395830 -1.190581  328.876466 -0.410669 -0.036616  0.010367   \n",
       "993  53.954964   4.227927 -1.207860  328.828460 -0.416479 -0.034973  0.010582   \n",
       "994  54.552428   4.057984 -1.225310  328.779862 -0.422166 -0.033305  0.010798   \n",
       "995  55.136438   3.885925 -1.242948  328.730672 -0.427732 -0.031610  0.011017   \n",
       "996  55.707017   3.711670 -1.260788  328.680888 -0.433175 -0.029888  0.011239   \n",
       "997  56.264180   3.535137 -1.278847  328.630509 -0.438496 -0.028137  0.011463   \n",
       "998  56.807928   3.356238 -1.297139  328.579534 -0.443694 -0.026358  0.011690   \n",
       "999  57.338250   3.174882 -1.315681  328.527960 -0.448768 -0.024548  0.011921   \n",
       "\n",
       "            U2            C2         U3        C3        T3  \n",
       "0    89.994869  1.708234e-06  89.995291  0.000154 -0.000016  \n",
       "1    87.357340  1.929820e-06  87.357800  0.000169 -0.000015  \n",
       "2    84.696987  2.190796e-06  84.697487  0.000186 -0.000014  \n",
       "3    82.015333  2.499874e-06  82.015876  0.000206 -0.000014  \n",
       "4    79.313924  2.868085e-06  79.314511  0.000228 -0.000013  \n",
       "5    76.594330  3.309504e-06  76.594965  0.000254 -0.000012  \n",
       "6    73.858147  3.842247e-06  73.858831  0.000284 -0.000012  \n",
       "7    71.106997  4.489832e-06  71.107733  0.000320 -0.000011  \n",
       "8    68.342531  5.283088e-06  68.343323  0.000362 -0.000010  \n",
       "9    65.566434  6.262841e-06  65.567285  0.000411 -0.000010  \n",
       "10   62.780428  7.483758e-06  62.781343  0.000471 -0.000009  \n",
       "11   59.986280  9.019915e-06  59.987261  0.000542 -0.000009  \n",
       "12   57.185808  1.097299e-05  57.186862  0.000629 -0.000008  \n",
       "13   54.380899  1.348449e-05  54.382031  0.000735 -0.000008  \n",
       "14   51.573516  1.675442e-05  51.574733  0.000866 -0.000008  \n",
       "15   48.765731  2.107014e-05  48.767039  0.001029 -0.000007  \n",
       "16   45.959745  2.685214e-05  45.961154  0.001236 -0.000007  \n",
       "17   43.157938  3.472815e-05  43.159457  0.001501 -0.000006  \n",
       "18   40.362922  4.565620e-05  40.364564  0.001845 -0.000006  \n",
       "19   37.577631  6.113466e-05  37.579410  0.002300 -0.000006  \n",
       "20   34.805442  8.357160e-05  34.807378  0.002912 -0.000006  \n",
       "21   32.050362  1.169561e-04  32.052475  0.003753 -0.000005  \n",
       "22   29.317307  1.681237e-04  29.319626  0.004934 -0.000005  \n",
       "23   26.612545  2.492355e-04  26.615105  0.006639 -0.000005  \n",
       "24   23.944414  3.828477e-04  23.947260  0.009175 -0.000005  \n",
       "25   21.324512  6.127281e-04  21.327704  0.013076 -0.000005  \n",
       "26   18.769763  1.027875e-03  18.773380  0.019305 -0.000004  \n",
       "27   16.306101  1.817243e-03  16.310247  0.029643 -0.000004  \n",
       "28    7.930807  2.714192e-02   7.935714  0.214992  0.000023  \n",
       "29    8.924352  1.698451e-02   8.928603  0.151447  0.000023  \n",
       "..         ...           ...        ...       ...       ...  \n",
       "970  51.415731  2.519794e-06  51.416344  0.000130  0.000112  \n",
       "971  52.705763  2.324094e-06  52.706403  0.000123  0.000110  \n",
       "972  53.971957  2.153209e-06  53.972623  0.000116  0.000107  \n",
       "973  55.214647  2.003297e-06  55.215340  0.000111  0.000104  \n",
       "974  56.434152  1.871221e-06  56.434871  0.000106  0.000102  \n",
       "975  57.630776  1.754397e-06  57.631520  0.000101  0.000099  \n",
       "976  58.804809  1.650687e-06  58.805579  0.000097  0.000096  \n",
       "977  59.956527  1.558313e-06  59.957321  0.000093  0.000094  \n",
       "978  61.086191  1.475781e-06  61.087010  0.000090  0.000091  \n",
       "979  62.194051  1.401839e-06  62.194895  0.000087  0.000089  \n",
       "980  63.280343  1.335425e-06  63.281211  0.000085  0.000086  \n",
       "981  64.345291  1.275636e-06  64.346182  0.000082  0.000084  \n",
       "982  65.389104  1.221704e-06  65.390019  0.000080  0.000081  \n",
       "983  66.411983  1.172969e-06  66.412920  0.000078  0.000079  \n",
       "984  67.414113  1.128864e-06  67.415073  0.000076  0.000076  \n",
       "985  68.395671  1.088900e-06  68.396654  0.000074  0.000074  \n",
       "986  69.356820  1.052653e-06  69.357824  0.000073  0.000072  \n",
       "987  70.297710  1.019754e-06  70.298736  0.000072  0.000070  \n",
       "988  71.218482  9.898833e-07  71.219529  0.000071  0.000068  \n",
       "989  72.119263  9.627609e-07  72.120331  0.000069  0.000066  \n",
       "990  73.000169  9.381419e-07  73.001257  0.000068  0.000063  \n",
       "991  73.861303  9.158115e-07  73.862412  0.000068  0.000062  \n",
       "992  74.702755  8.955813e-07  74.703883  0.000067  0.000060  \n",
       "993  75.524603  8.772855e-07  75.525751  0.000066  0.000058  \n",
       "994  76.326909  8.607785e-07  76.328077  0.000066  0.000056  \n",
       "995  77.109723  8.459319e-07  77.110910  0.000065  0.000054  \n",
       "996  77.873080  8.326329e-07  77.874285  0.000065  0.000053  \n",
       "997  78.616997  8.207825e-07  78.618220  0.000065  0.000051  \n",
       "998  79.341480  8.102936e-07  79.342721  0.000064  0.000049  \n",
       "999  80.046510  8.010903e-07  80.047768  0.000064  0.000048  \n",
       "\n",
       "[1000 rows x 18 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = [\n",
    "    'T', 'X', 'Vx', 'Ax', 'Jx', 'Y', 'Vy','Ay', 'Jy',\n",
    "    'Z', 'Vz', 'Az', 'Jz', 'U2', 'C2', 'U3', 'C3', 'T3']\n",
    "i = 0\n",
    "trajectory_df = pd.DataFrame(X_df['data'][i], columns=columns)\n",
    "trajectory_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the meaning of the columns:\n",
    "- `T`: time\n",
    "- `X`, `Vx`, `Ax`, `Jx`: X coordinate of the resp. position, velocity, acceleration, and jerk (i.e. first to third order derivative) of the aircraft\n",
    "- `Y`, `Vy`, `Ay`, `Jy`: Y coordinate of the resp. position, velocity,  acceleration, and jerk (i.e. first to third order derivative) of the aircraft\n",
    "- `Z`, `Vz`, `Az`, `Jz`: Z coordinate of the resp. position, velocity,  acceleration, and jerk (i.e. first to third order derivative) of the aircraft- `U2`: absolute value of velocity vector projected onto the ground plane\n",
    "- `C3`: curvature of the 3D trajectory: $\\frac{||P'\\times P''||}{||P'||^3}$ where $P = (X, Y, Z)$\n",
    "- `T3`: torsion of the 3D trajectory: $\\frac{\\det(P',P'',P''')}{||P'\\times P''||^2}$\n",
    "- `C2`: curvature of the trajectory projected onto the ground plane: $\\frac{||\\tilde P'\\times \\tilde\n",
    "P''||}{||\\tilde P'||^3}$ where $\\tilde P=(X,Y,0)$\n",
    "- `U3`: absolute value of the 3D velocity vector\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The pipeline\n",
    "\n",
    "For submitting at the [RAMP site](http://ramp.studio), you will have to write two classes, saved in two different files:   \n",
    "* the class `FeatureExtractor`, which will be used to extract features for classification from the dataset and produce a numpy array of size (number of samples $\\times$ number of features). \n",
    "* a class `Classifier` to predict aircraft type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature extractor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature extractor implements a `transform` member function. It is saved in the file [`submissions/starting_kit/feature_extractor.py`](/edit/submissions/starting_kit/feature_extractor.py). It receives the pandas dataframe `X_df` defined at the beginning of the notebook. It should produce a numpy array representing the extracted features, which will then be used for the classification. The following simple feature extractor simply computes the mean of each feature and produces an 18 dimensional feature vector for each trajectory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class FeatureExtractor():\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    # use this if you need to learn something at training time that depends on the labels\n",
    "    # this will not be called on the test instances\n",
    "    def fit(self, X_df, y):\n",
    "        pass\n",
    "\n",
    "    # this will be called both on the training and test instance\n",
    "    def transform(self, X_df):\n",
    "        data_matrix = np.asarray(list(X_df['data'].values))\n",
    "        # taking the mean of each data matrix variable, averaging over axis 1\n",
    "        means = data_matrix.mean(axis=1)\n",
    "        return means\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it on the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4560, 18)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fe = FeatureExtractor()\n",
    "fe.fit(X_df, y_array)  \n",
    "X_array = fe.transform(X_df)\n",
    "X_array.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classifier\n",
    "\n",
    "The classifier follows a classical scikit-learn classifier template. In its simplest form it takes a scikit-learn pipeline, assigns it to `self.clf` in `__init__`, then calls its `fit` and `predict_proba` functions in the corresponding member funtions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "\n",
    "class Classifier(BaseEstimator):\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        self.clf = RandomForestClassifier(\n",
    "            n_estimators=2, max_leaf_nodes=2, random_state=61)\n",
    "        self.clf.fit(X, y)\n",
    "\n",
    "    def predict_proba(self, X):\n",
    "        return self.clf.predict_proba(X)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it on the training data, transformed into features. The output is a probability array, each row is 19 dimensional since we have 19 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4560, 19)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = Classifier()\n",
    "clf.fit(X_array, y_array)\n",
    "y_proba_array = clf.predict_proba(X_array)\n",
    "y_proba_array.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring\n",
    "\n",
    "To score, we forst convert the probability array into a label array by taking the label that received the highest probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred_array = [problem._prediction_label_names[y] for y in np.argmax(y_proba_array, axis=1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can compute the accuracy using `accuracy_score` of scikit learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10482456140350878"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score \n",
    "accuracy_score(y_array, y_pred_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell will evaluate the workflow on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10478468899521531"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_df, y_test_array = problem.get_test_data()\n",
    "X_test_array = fe.transform(X_test_df)\n",
    "y_test_proba_array = clf.predict_proba(X_test_array)\n",
    "y_test_pred_array = [problem._prediction_label_names[y] for y in np.argmax(y_test_proba_array, axis=1)]\n",
    "accuracy_score(y_test_array, y_test_pred_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local testing (before submission)\n",
    "\n",
    "You can start playing with the cells above, modify the feature extractor and the classifier and evaluate the accuracy of the workflow. However, it is <b><span style=\"color:red\">important that you test your submission files before submitting them</span></b>. First you should save your feature extractor and classifier classes into [`submissions/starting_kit/feature_extractor.py`](/edit/submissions/starting_kit/feature_extractor.py) and [`submissions/starting_kit/classifier.py`](/edit/submissions/starting_kit/classifier.py), respectively. Then run the `ramp_test_submission` script either at the command line or here. This script executes a more robust cross validation on the training set, defined in `problem.py`. If this test runs without error, you can submit your feature extractor and classifier to [ramp.studio](http://www.ramp.studio/problems/radar_trajectories)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;178m\u001b[1mTesting Aircraft classification from radar trajectories\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mReading train and test files from ./data ...\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mReading cv ...\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mTraining ./submissions/starting_kit ...\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 0\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.489\u001b[0m  \u001b[38;5;150m0.107\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.554\u001b[0m  \u001b[38;5;105m0.098\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.600\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 1\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.490\u001b[0m  \u001b[38;5;150m0.104\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.496\u001b[0m  \u001b[38;5;105m0.109\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.547\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 2\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.506\u001b[0m  \u001b[38;5;150m0.104\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.477\u001b[0m  \u001b[38;5;105m0.109\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.538\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 3\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.505\u001b[0m  \u001b[38;5;150m0.104\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.542\u001b[0m  \u001b[38;5;105m0.109\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.694\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 4\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.490\u001b[0m  \u001b[38;5;150m0.104\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.519\u001b[0m  \u001b[38;5;105m0.109\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.539\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 5\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.508\u001b[0m  \u001b[38;5;150m0.104\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.537\u001b[0m  \u001b[38;5;105m0.109\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.670\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 6\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.498\u001b[0m  \u001b[38;5;150m0.107\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.486\u001b[0m  \u001b[38;5;105m0.098\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.545\u001b[0m  \u001b[38;5;218m0.104\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 7\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.493\u001b[0m  \u001b[38;5;150m0.107\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.483\u001b[0m  \u001b[38;5;105m0.114\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.541\u001b[0m  \u001b[38;5;218m0.111\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mMean CV scores\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore             nll             acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m2.497 ± 0.0074\u001b[0m  \u001b[38;5;150m0.105 ± 0.0014\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.512 ± 0.0282\u001b[0m  \u001b[38;5;105m0.107 ± 0.0054\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.584 ± 0.0598\u001b[0m  \u001b[38;5;218m0.105 ± 0.0024\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mBagged scores\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m2.515\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m2.535\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ramp_test_submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also edit and test other submissions saved into `submissions/<submission_name>`. For example, there is a submission in `submissions/more_features` using a bigger random forest and more features. You can test it using the following command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[38;5;178m\u001b[1mTesting Aircraft classification from radar trajectories\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mReading train and test files from ./data ...\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mReading cv ...\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mTraining ./submissions/more_features ...\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 0\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.042\u001b[0m  \u001b[38;5;150m0.780\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.739\u001b[0m  \u001b[38;5;105m0.429\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.707\u001b[0m  \u001b[38;5;218m0.450\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 1\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.067\u001b[0m  \u001b[38;5;150m0.757\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.529\u001b[0m  \u001b[38;5;105m0.486\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.737\u001b[0m  \u001b[38;5;218m0.462\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 2\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.065\u001b[0m  \u001b[38;5;150m0.766\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.677\u001b[0m  \u001b[38;5;105m0.427\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.693\u001b[0m  \u001b[38;5;218m0.447\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 3\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.070\u001b[0m  \u001b[38;5;150m0.752\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.565\u001b[0m  \u001b[38;5;105m0.495\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.660\u001b[0m  \u001b[38;5;218m0.457\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 4\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.061\u001b[0m  \u001b[38;5;150m0.748\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.559\u001b[0m  \u001b[38;5;105m0.472\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.646\u001b[0m  \u001b[38;5;218m0.462\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 5\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.051\u001b[0m  \u001b[38;5;150m0.776\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.592\u001b[0m  \u001b[38;5;105m0.453\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.724\u001b[0m  \u001b[38;5;218m0.447\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 6\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.059\u001b[0m  \u001b[38;5;150m0.776\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.640\u001b[0m  \u001b[38;5;105m0.385\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.711\u001b[0m  \u001b[38;5;218m0.444\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mCV fold 7\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll    acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.046\u001b[0m  \u001b[38;5;150m0.774\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.597\u001b[0m  \u001b[38;5;105m0.440\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.630\u001b[0m  \u001b[38;5;218m0.476\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mMean CV scores\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore             nll             acc\u001b[0m\n",
      "\t\u001b[38;5;10m\u001b[1mtrain\u001b[0m  \u001b[38;5;10m\u001b[1m1.058 ± 0.0097\u001b[0m  \u001b[38;5;150m0.766 ± 0.0113\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.612 ± 0.0648\u001b[0m  \u001b[38;5;105m0.448 ± 0.0336\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.688 ± 0.0362\u001b[0m    \u001b[38;5;218m0.456 ± 0.01\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1mBagged scores\u001b[0m\n",
      "\u001b[38;5;178m\u001b[1m----------------------------\u001b[0m\n",
      "\t\u001b[38;5;178m\u001b[1mscore    nll\u001b[0m\n",
      "\t\u001b[38;5;12m\u001b[1mvalid\u001b[0m  \u001b[38;5;12m\u001b[1m1.585\u001b[0m\n",
      "\t\u001b[38;5;1m\u001b[1mtest\u001b[0m   \u001b[38;5;1m\u001b[1m1.607\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!ramp_test_submission --submission more_features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting to [ramp.studio](http://ramp.studio)\n",
    "\n",
    "Once you found a good feature extractor and classifier, you can submit them to [ramp.studio](http://www.ramp.studio). First, if it is your first time using RAMP, [sign up](http://www.ramp.studio/sign_up), otherwise [log in](http://www.ramp.studio/login). Then find an open event on the particular problem, for example, the event [radar_trajectories](https://www.ramp.studio/events/radar_trajectories_mines_201718) for this RAMP. Sign up for the event. Both signups are controled by RAMP administrators, so there **can be a delay between asking for signup and being able to submit**.\n",
    "\n",
    "Once your signup request is accepted, you can go to your [sandbox](http://www.ramp.studio/events/radar_trajectories_mines_201718/sandbox) and copy-paste (or upload) [`feature_extractor.py`](/edit/submissions/starting_kit/feature_extractor.py) and [`classifier.py`](/edit/submissions/starting_kit/classifier.py) from `submissions/starting_kit`. Save it, rename it, then submit it. The submission is trained and tested on our backend in the same way as `ramp_test_submission` does it locally. While your submission is waiting in the queue and being trained, you can find it in the \"New submissions (pending training)\" table in [my submissions](http://www.ramp.studio/events/radar_trajectories_mines_201718/my_submissions). Once it is trained, you get a mail, and your submission shows up on the [public leaderboard](http://www.ramp.studio/events/radar_trajectories_mines_201718/leaderboard). \n",
    "If there is an error (despite having tested your submission locally with `ramp_test_submission`), it will show up in the \"Failed submissions\" table in [my submissions](http://www.ramp.studio/events/radar_trajectories_mines_201718/my_submissions). You can click on the error to see part of the trace.\n",
    "\n",
    "After submission, do not forget to give credits to the previous submissions you reused or integrated into your submission.\n",
    "\n",
    "The data set we use at the backend is usually different from what you find in the starting kit, so the score may be different.\n",
    "\n",
    "The usual way to work with RAMP is to explore solutions, add feature transformations, select models, perhaps do some AutoML/hyperopt, etc., _locally_, and checking them with `ramp_test_submission`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More information\n",
    "\n",
    "You can find more information in the [wiki](https://github.com/paris-saclay-cds/ramp-workflow/wiki) of the `ramp-workflow library`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contact\n",
    "\n",
    "Don't hesitate to [contact us](mailto:admin@ramp.studio?subject=radar trajectory notebook)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}