This repository contains the implementation of the paper:

DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images
Bing Wang^*, Lu Chen^*, Bo Yang
Paper | Supplementary | Video

 

python >=3.7
pip install pytorch==1.8.1 torchVision==0.9.1 torchaudio===0.8.1
pip install -r environment.txt

To evaluate our model or train a new model from scratch, you have to obtain the respective dataset. In this paper, we consider 3 different datasets:

DM-SR

Replica

ScanNet

After you set all parameters you want, you can train model use one of blow command, for example:

If you want use full of decomposition function, you can run commands like:

CUDA_VISIBLE_DEVICES=0 python -u train_dmsr.py --config configs/train/dmsr/study.txt

If you do not decompose emptiness area, you can delete penalize parameter in config file, and run above command.

We used PSNR, SSIM, LPIPS, and mAPs to evaluate our tasks:

For decomposition operation:

You need to add render=True and log_time="your log folder name" into config txt.

And then run:

CUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/test/dmsr/study.txt

Manipulation operation includes two parts, evaluation and demo generation:

We only provide manipulated ground truth of DM-SR dataset for manipulation evaluation.

Change render = True to mani_eval = True, add target_label and editor_mode to assign which object manipulated and

which manipulated operation you want, specific format can renference ./configs/manipulation/dmsr/editor_multi/study.txt.

You can run:

CUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/manipulation/dmsr/manipulation_multi/study.txt

Change render = True to mani_demo = True, edit objs_info.json to assign objects manipulation, file path is

./data/dmsr/study/objs_info.json.

You can move view poses by given view_id = null, but given views a number in config file.

More explanation, ins_map is a global matching list statistic from matching_logs.json.

You can run:

CUDA_VISIBLE_DEVICES=0 python -u test_dmsr.py --config configs/test/dmsr/study.txt

SOTA method Mask R-CNN

If you find our work useful in your research, please consider citing:

In this project we use (parts of) the implementations of the following works:

• nerf_pytorch by Yen-Chen Lin.
• ScanNet by Angela Dai.
• 3D-BoNet by Bo Yang.
• Semantic-NeRF by Shuaifeng Zhi.
• Mask R-CNN by Kaiming He.

We thank the respective authors for open sourcing their methods.