notebooks/prepare_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Path configuration\n",
    "\n",
    "<pre>\n",
    "Please modify <b>config_path.py</b> first.\n",
    "\n",
    "Raw datasets will be organized as the following structure:\n",
    "\n",
    "dataset_path/\n",
    "    | kitti/               # KITTI object detection 3D dataset\n",
    "        | training/\n",
    "        | testing/\n",
    "    | argo/                # Argoverse dataset v1.1\n",
    "        | train1/\n",
    "        | train2/\n",
    "        | train3/\n",
    "        | train4/\n",
    "        | val/\n",
    "        | test/\n",
    "    | nusc/                # nuScenes dataset v1.0\n",
    "        | maps/\n",
    "        | samples/\n",
    "        | sweeps/\n",
    "        | v1.0-trainval/\n",
    "    | lyft/                # Lyft Level 5 dataset v1.02\n",
    "        | v1.02-train/\n",
    "    | waymo/               # Waymo dataset v1.0\n",
    "        | training/\n",
    "        | validation/\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, \"..\")\n",
    "from config_path import raw_path_dic as path_dic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Download the datasets\n",
    "We provide scripts for automatic downloading. \n",
    "\n",
    "If you already have some of these datasets, you may create soft links following our path configuration and skip the corresponding download scripts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KITTI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from download import download_kitti\n",
    "\n",
    "download_kitti(path_dic[\"kitti\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Argoverse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from download import download_argo\n",
    "\n",
    "download_argo(path_dic[\"argo\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nuScenes\n",
    "Please visit [this link](https://www.nuscenes.org/download)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lyft\n",
    "Please visit [this link](https://level5.lyft.com/dataset/download-dataset/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Waymo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from download import download_waymo\n",
    "\n",
    "download_waymo(path_dic[\"waymo\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Convert the datasets to KITTI format\n",
    "We use KITTI format as the uniform interface for data loading, and convert all other datasets to KITTI format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Argoverse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install git+git://github.com/argoai/argoverse-api.git\n",
    "from convert import argo_to_kitti\n",
    "\n",
    "argo_to_kitti(path_dic[\"argo\"], path_dic[\"argo-in-kitti-format\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nuScenes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install git+git://github.com/nutonomy/nuscenes-devkit.git\n",
    "from convert import nusc_to_kitti\n",
    "\n",
    "nusc_to_kitti(path_dic[\"nusc\"], path_dic[\"nusc-in-kitti-format\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lyft"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U git+https://github.com/lyft/nuscenes-devkit\n",
    "from convert import lyft_to_kitti\n",
    "\n",
    "lyft_to_kitti(path_dic[\"lyft\"], path_dic[\"lyft-in-kitti-format\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Waymo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install tensorflow==1.15      # CPU\n",
    "#!pip install tensorflow-gpu==1.15  # GPU\n",
    "!pip install git+git://github.com/waymo-research/waymo-open-dataset.git\n",
    "from convert import waymo_to_kitti\n",
    "\n",
    "waymo_to_kitti(path_dic[\"waymo\"], path_dic[\"waymo-in-kitti-format\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Subsample & Re-split\n",
    "We subsampled and re-split the datasets to balance training time and enable offline evaluation.\n",
    "\n",
    "We provide our split lists for result reproduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from split import replace_split\n",
    "\n",
    "replace_split(path_dic)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}