SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation

Jiehong Lin, Lihua Liu, Dekun Lu, Kui Jia

CVPR 2024

[Paper]

News

• [2024/03/07] We publish an updated version of our paper on ArXiv.
• [2024/02/29] Our paper is accepted by CVPR2024!

Update Log

• [2024/03/05] We update the demo to support FastSAM, you can do this by specifying SEGMENTOR_MODEL=fastsam in demo.sh.
• [2024/03/03] We upload a docker image for running custom data.
• [2024/03/01] We update the released model of PEM. For the new model, a larger batchsize of 32 is set, while that of the old is 12.

Overview

In this work, we employ Segment Anything Model as an advanced starting point for zero-shot 6D object pose estimation from RGB-D images, and propose a novel framework, named SAM-6D, which utilizes the following two dedicated sub-networks to realize the focused task:

• Instance Segmentation Model
• Pose Estimation Model

Getting Started

0. Prerequisites

• Linux (tested on Ubuntu 22.04)
• NVIDIA drivers (tested with 535)
• Anaconda (see below for Installation)
• If running on non-local machines: sudo apt install xorg

Optional: Installation of Anaconda

Run the code below, to install the latest version of Miniconda, or refer to the Anaconda documentation.

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh &&
chmod +x ~/miniconda.sh &&
bash ~/miniconda.sh -b -p ~/miniconda &&
rm ~/miniconda.sh &&
source ~/miniconda/bin/activate &&
conda init --all &&
source ~/.bashrc

1. Preparation

Please clone the repository locally:

git clone https://github.com/JiehongLin/SAM-6D.git
cd SAM-6D/SAM-6D

Install the environment:

conda env create -f environment.yaml
conda activate sam6d

Download the model checkpoints and build PointNet2:

sh prepare.sh

We also provide a docker image for convenience.

2. Run Inference on the example or custom data

To run inference the following environment variables must be set:

• $CAD_PATH path to a given cad model(mm)
• $RGB_PATH path to a given RGB image
• $DEPTH_PATH path to a given depth map(mm)
• $CAMERA_PATH path to given camera intrinsics
• $OUTPUT_DIR path to a pre-defined file for saving results

Run inference on the example data:

# export example variables
# must be executed from SAM-6D/SAM-6D directory containg the Data folder
export CAD_PATH=$PWD/Data/Example/obj_000005.ply
export RGB_PATH=$PWD/Data/Example/rgb.png
export DEPTH_PATH=$PWD/Data/Example/depth.png
export CAMERA_PATH=$PWD/Data/Example/camera.json
export OUTPUT_DIR=$PWD/Data/Example/outputs

sh demo.sh

All output will be saved unter the $OUTPUT_DIR.

To run inference on custom data, export the environment pointing to your data and then run sh demo.sh.

Citation

If you find our work useful in your research, please consider citing:

@article{lin2023sam,
title={SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation},
author={Lin, Jiehong and Liu, Lihua and Lu, Dekun and Jia, Kui},
journal={arXiv preprint arXiv:2311.15707},
year={2023}
}

Contact

If you have any questions, please feel free to contact the authors.

Jiehong Lin: [email protected]

Lihua Liu: [email protected]

Dekun Lu: [email protected]

Kui Jia: [email protected]