iMatching: Imperative Correspondence Learning

Zitong Zhan^*, Dasong Gao^*, Yun-Jou Lin, Youjie Xia, Chen Wang,

📄 [ECCV 2024] | 📄 [Arxiv] | 💾 [Project Site]

Dependencies

The following library/packages are required:

• Python >= 3.10
• PyTorch >= 2.1
• GTSAM >= 4.1.1
• requirement.txt

Steps

0. Optionally, prepare build env

   conda create -n imatching python=3.10
   conda activate imatching
   conda install gcc==9.5 gxx==9.5 cmake boost

1. Follow the official PyTorch instructions to install PyTorch >= 1.11.

   pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0

2. Install the rest of Python dependencies:

   pip install -r requirements.txt

3. Install GTSAM with Python bindings:

   # install gtsam dependencies
   conda install boost==1.82.0
   
   # build gtsam from source
   export BUILD_TYPE=RelWithDebInfo
   git submodule update --init --recursive thirdparty/gtsam
   # cmake build
   cd thirdparty/gtsam
   
   mkdir -p build
   cd build
   # TBB allocator crashes python so we have to turn it off
   cmake .. -DGTSAM_BUILD_PYTHON=1 -DCMAKE_BUILD_TYPE=$BUILD_TYPE -DGTSAM_WITH_TBB=OFF -DGTSAM_CMAKE_CXX_FLAGS=-march-native
   make -j python-install

Note: In case libgtsamRelWithDebInfo.so is not found, run sudo ldconfig to register the GTSAM library.

Note: If you are on Python 3.11 and the above procedure throws compile error about pybind11, you may need to replace thirdparty/gtsam/wrap/pybind11 with pybind11 >= 2.10.1 (tested on 2.10.4).

4. Update submodules

   git submodule update --init --recursive ext/
   

Data

TartanAir

• TartanAir, download with tartanair_tools

 python download_training.py --output-dir OUTPUTDIR --rgb --depth --only-left

The dataset is aranged as follows:

$DATASET_ROOT/
└── tartanair/
    ├── abandonedfactory_night/
    |   ├── Easy/
    |   |   └── ...
    │   └── Hard/
    │       └── ...
    └── ...

Note: For TartanAir, only <ENVIRONMENT>/<DIFFICULTY>/<image|depth>_left.zip is required. After unziping downloaded zip files, make sure to remove the duplicate <ENVIRONMENT> directory level (tartanair/abandonedfactory/abandonedfactory/Easy/... -> tartanair/abandonedfactory/Easy/...).

find . -type f -name '*.zip' -exec unzip -n {} \;
for i in abandonedfactory        amusement   endofworld  hospital       neighborhood  office   oldtown      seasonsforest         soulcity abandonedfactory_night  carwelding  gascola     japanesealley  ocean         office2  seasidetown  seasonsforest_winter  westerndesert
do
    rm -r $i/Easy
    rm -r $i/Hard
    mv $i/$i/* $i/
    rm -r $i/$i
done

• ETH3D SLAM: The dataset will be downloaded automatically by the datamodule.

Weights

Copy the pretrained weights to pretrained/ directory. The following commands download the weights for CAPS, Patch2Pix, and AspanFormer. DKM weights will be downloaded automatically by the model.

pip install gdown
mkdir pretrained
cd pretrained/
# CAPS
gdown 1UVjtuhTDmlvvVuUlEq_M5oJVImQl6z1f
#p2p
sh ../ext/patch2pix/pretrained/download.sh
#aspan
gdown 1eavM9dTkw9nbc-JqlVVfGPU5UvTTfc6k
tar -xvf weights_aspanformer.tar
cd ..

Run

Train on TartanAir

• CAPS

scene=abandonedfactory
d=Easy
python ./train.py \
data_root=./data/datasets \
datamodule.include=\"$scene\_$d\"

• Patch2Pix

scene=abandonedfactory
d=Easy
python ./train.py \
--config-name p2p-train-tartanair \
data_root=./data/datasets \
trainer.max_epochs=2 \
datamodule.include=\"$scene\_$d\"

• AspanFormer

scene=abandonedfactory
d=Easy
python ./train.py \
--config-name aspan-train-tartanair \
data_root=./data/datasets \
trainer.max_epochs=2 \
datamodule.include=\"$scene\_$d\"

• DKM

scene=abandonedfactory
d=Easy
python ./train.py \
--config-name dkm-train-tartanair \
data_root=./data/datasets \
trainer.max_epochs=5 \
datamodule.include=\"$scene\_$d\"

Override configs

Configs listed in python ./train.py --help can be overriden by key=value.

Examples:

• Use only part of the dataset: datamodule.include=\"<regex>\" or datamodule.exclude=\"<regex>\", where <regex> is the regular expression for matching the sequences to be included or removed (e.g. abandonedfactory_Easy_P00[0-2])
• Change the train/validation/test split ratio of dataset: datamodule.split_ratio=[0.5,0.3,0.2].
• Change the validation interval: trainer.val_check_interval=<interval>, see PyTorch Lightning docs.

---

Citing iMatching

@inproceedings{zhan2024imatching,
  title = {{iMatching}: Imperative Correspondence Learning},
  author = {Zhan, Zitong and Gao, Dasong and Lin, Yun-Jou and Xia, Youjie and Wang, Chen},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year = {2024},
  url = {https://arxiv.org/abs/2312.02141},
}