Robotics: Science and Systems (RSS), 2022
iSDF is real-time system the reconstructs the signed distance field (SDF) of room scale environments through the online training of a neural SDF. The model is represented by a MLP that is trained in a continual learning style from a live stream of posed depth images using our self-supervised loss. For details see our paper.
| ReplicaCAD | ScanNet |
|---|---|
 |
 |
There are three options for running iSDF, chose the desired mode and then follow only instructions in that section:
- Running iSDF on sequences with pose provided: for running iSDF on ScanNet and ReplicaCAD sequences used in the paper.
- Running iSDF with a live camera in ROS: a ros wrapper for iSDF subscribes to a topic of posed frames that is published by an ORB-SLAM3 wrapper.
- Running iSDF with a Franka and live camera in ROS: a ros wrapper for iSDF subscribes to a topic of realsense frames and poses from a calibrated realsense on a Franka Panda robot.
Please read the iSDF modes section to make sure this is the mode you want before installing!
git clone https://github.com/facebookresearch/iSDF.git && cd iSDF
Setup the environment.
conda env create -f environment.yml
conda activate isdf
Install pytorch by following instructions here, then:
pip install -e .
Make sure to be in the conda environment. To download a single replicaCAD sequence (5GB):
bash data/download_apt_2_nav.sh
To download all 12 sequences (15GB):
bash data/download_data.sh
The data is shared under the same license as the Replica-CAD datset. For instructions on how to generate the dataset see the data README.
To run the ScanNet sequences, you must download the ScanNet dataset. After downloading the scans, you should export the depth, color, poses and intrinsics for the sequences you wish to use with this script.
If using ScanNet data, then you must set the directory for the downloaded sequence via the key scannet_dir in the config file.
cd isdf/train/
python train_vis.py --config configs/replicaCAD.json
Run a batch of iSDF experiments sequentially in headless mode. To run these experiments you must have downloaded all 12 sequences using our bash script as well as separately downloading and exporting the ScanNet sequences. To run only the ReplicaCAD sequences, you can modify the load_params function in batch_utils.py. If you have multiple GPUs you may want to parallelise the experiment runs. Make sure to update project_dir and scannet_dir in jobs_local.py before running:
cd isdf/train
python batch_train/jobs_local.py
We have provided results for the baselines that we use in the paper for those wanted to reproduce results for iSDF only. Code to reproduce results for our baselines is coming soon.
Below we provide instructions to generate quantitative and qualitative results as in the paper. Note results may vary depending on the CPU / GPU used.
We provide the script to generate plots comparing SDF error, collision cost error and gradient cosine distance for iSDF and the two baselines. This script was used to generate all quantitative plots in the paper (e.g. Fig 8). Change the variable isdf_dir in the script before running:
python isdf/eval/figs/all_seq.py
We use ROS to interface iSDF with a tracking system and enable running with a live camera.
The iSDF node subscribes to the topic: frames, which is a custom message type containing a time synchronised rgb image, depth image and pose.
We use a ROS wrapper for ORB-SLAM3 in RGBD mode to publish to this topic.
We have developed this project on Ubuntu 20.04 (and ROS Noetic, so several code changes may be needed to adapt to other OS and ROS versions.
You may need to install some dependencies first, see full instructions if you run into issues.
git clone https://github.com/UZ-SLAMLab/ORB_SLAM3.git ORB_SLAM3
cd ORB_SLAM3
chmod +x build.sh
./build.sh
Follow install instructions.
Also install catkin tools if not already installed: sudo apt-get install python3-catkin-tools.
sudo apt-get install ros-$ROS_DISTRO-realsense2-camera
Create a catkin workspace and clone iSDF inside:
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
catkin init
cd src
git clone --recurse-submodules https://github.com/facebookresearch/iSDF.git
Open CMakeLists.txt and change the directory that points to ORB-SLAM3 library at the beginning of the file (default is home folder).
nano ~/catkin_ws/src/iSDF/ORB_SLAM3_ros/CMakeLists.txt
# Change this to your installation of ORB-SLAM3. Default is ~/
set(ORB_SLAM3_DIR
$ENV{HOME}/ORB_SLAM3
)
Unzip the ORBvoc.txt file:
cd ~/catkin_ws/src/iSDF/ORB_SLAM3_ros/config
tar -xf ORBvoc.txt.tar.gz
Build the ROS packages (while not in a conda environment):
cd ~/catkin_ws
catkin_make
source devel/setup.bash
Setup the iSDF conda environment.
conda env create -f environment.yml
conda activate isdf
Install pytorch by following instructions here, then:
pip install -e .
Before running you need to modify the camera intrinsics for both iSDF and ORB-SLAM3. If you launch the camera (roslaunch realsense2_camera rs_camera.launch), the intrinsics are published at /camera/color/camera_info. Copy these intrinsics into the iSDF config (isdf/train/configs/realsense.json) and the ORBSLAM3 config (ORB_SLAM3_ros/config/realsense_config.yaml).
To launch run:
roslaunch isdf train.launch show_orbslam_vis:=true
It can be helpful to show the ORB-SLAM3 vis as if the tracking fails iSDF will also fail.
During live operation:
- Press
cto clear the keyframe set. - Press
sto pause live operation and open a 3D visualization window. You can change the 3D visualization using key presses, instructions are printed in the terminal when the window is opened.
Our ORB-SLAM3 wrapper also contains a launchfile to run with the Azure Kinect camera. To run with a Kinect camera, modify the train.launch to launch run_kinect.launch and install the kinect ros drivers.
Current limitations:
- iSDF runs in a single thread and alternates between training (optimising the map) and updating the live visualisation. You can change the frequency of the live visualisation updates, however if set too high, training can become slow.
- The live visualistaion of the rendered depth and normals from the current view uses a very basic depth rendering method. This can lead to artifacts in the render even if the geometry is good. We plan to improve the depth rendering soon.
This runs similar to the live camera demo, but subscribes to Franka Panda inverse kinematics rather than ORB-SLAM. Additionally, this implementation has been tuned to work for tabletop scenes and is restricted to a defined workspace. Follow all instructions from the previous section, but the ORB-SLAM steps are not needed. The system requires the following:
- Extrinsic calibration of the realsense w.r.t. the robot base : use the eyehandcal repository.
- Run the automatic script using the
--proj-fun world_marker_proj_hand_camera. - Copy the contexts of
calibration.jsonand add it to"ext_calib"inisdf/train/configs/realsense_franka.json
- Run the automatic script using the
- Live ROS topics for rgb, depth, and EE poses of the Franka.
- With our modified franka control you can record kinesthetic demonstratations. Please refer to the repository for dependencies and setup.
- When you replay them with
playback.pythey publish the live posed frames.
After recording the kinesthetic demo with franka_control, run the repository's playback script:
python playback.py xx/yy.npz
To launch run:
roslaunch isdf train_franka.launch
Tabletop dataset script
Even without access to a Franka, you can run franka iSDF on our tabletop dataset. The datset consists of 5 datalogs with tabletop YCB objects and a 3D printed Stanford bunny). To download the franka_realsense tabletop dataset (3.4GB):
bash data/download_franka.sh
To launch run (select the "seq_dir" in realsense_franka_offline.json):
roslaunch isdf train_franka.launch live:=false inc:=true
Franka integration was carried out by Sudharshan Suresh, feel free to reach out with questions/comments.
@inproceedings{Ortiz:etal:iSDF2022,
title={iSDF: Real-Time Neural Signed Distance Fields for Robot Perception},
author={Ortiz, Joseph and Clegg, Alexander and Dong, Jing and Sucar, Edgar and Novotny, David and Zollhoefer, Michael and Mukadam, Mustafa},
booktitle={Robotics: Science and Systems},
year={2022}
}
iSDF is released under the MIT license. Please see the LICENSE file for more information.
We actively welcome your pull requests! Please see CONTRIBUTING.md and CODE_OF_CONDUCT.md for more info.