Dexterous Grasp Transformer

Guo-Hao Xu*, Yi-Lin Wei*, Dian Zheng, Xiao-Ming Wu, Wei-Shi Zheng

[Paper]     [Project]

(CVPR 2024) Official repository of paper "Dexterous Grasp Transformer".

Installation

1. Create a new conda environment

conda create -n dgtr python=3.8

2. Install pytorch

conda install pytorch==1.10.0 torchvision==0.11.0 cudatoolkit=11.3 -c pytorch -c conda-forge

3. Install pytorch3d

• Download source code of pytorch3d 0.7.2 from link and unzip it.

cd pytorch3d && pip install -e .

4. Install other dependencies

conda install -c plotly 
conda install -c conda-forge trimesh pyyaml tqdm plyfile multimethod
conda install scipy
conda install -c anaconda lxml
conda install tensorboard

5. Install pytorch_kinematics

cd thirdparty/pytorch_kinematics
pip install -e .

6. Install CSDF

cd thirdparty/CSDF
pip install -e .

7. Install pointnet2

cd thirdparty/pointnet2
python setup.py install

8. Install knn

cd thirdparty/knn
python setup.py install

9. Install rtree

pip install rtree

10. If you have encountered setuptools-related error, this may help:

pip install setuptools==59.5.0

Usage

Train

1. Train DGTR in Dynamic Match Training phase for 15 epochs.

CUDA_VISIBLE_DEVICES="0" python train.py --train_cfg config/dgtr.yaml

2. Train DGTR in Static Match Warm-up Training phase for 5 epochs.

CUDA_VISIBLE_DEVICES="0" python train.py \
--train_cfg config/dgtr.yaml \
-r \
-c <checkpoint of epoch 15> \
--override assignments \"static\" data.train.assignment \"./Experiments/dgtr/assignment_epoch_15.json\"

3. Train DGTR in Static Matching Penetration Training phase for 5 epochs.

CUDA_VISIBLE_DEVICES="0" python train.py \
--train_cfg config/dgtr.yaml \
-r \
-c <checkpoint of epoch 20> \
--override assignments \"static\" data.train.assignment \"./Experiments/dgtr/assignment_epoch_15.json\"

Inference

python ./test.py \
--train_cfg config/dgtr.yaml \
--test_cfg ./config/test_default.yaml \
--override model.checkpoint_path \"<checkpoint of epoch 25>\"

Evaluation

python ./tools/evaluate.py -r <the path of raw_results.json> --gpus <GPU_ID>

TODO

• Release the code of AB-TTA
• Release the training code of DGTR
• Release the inference code of DGTR

Acknowledgements

The code of this repository is based on the following repositories. We would like to thank the authors for sharing their works.

• DexGraspNet

• UniDexGrasp

Contact

• Email: {xugh23, weiylin5}@mail2.sysu.edu.cn

Citation

Please cite it if you find this work useful.

@inproceedings{xu2024dexterous,
 title = {Dexterous Grasp Transformer},
 author = {Xu, Guo-Hao and Wei, Yi-Lin and Zheng, Dian and Wu, Xiao-Ming and Zheng, Wei-Shi},
 booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
 year = {2024}
 }