FFNET: Flow-Based Feature Fusion for VEHICLE-INFRASTRUCTURE COOPERATIVE 3D OBJECT DETECTION

Figure 1: FFNET OVERVIEW.

Project page | Paper |

FFNET: Flow-Based Feature Fusion for VEHICLE-INFRASTRUCTURE COOPERATIVE 3D OBJECT DETECTION.
Haibao Yu, Yingjuan Tang, Enze Xie, Jilei Mao, Ping Luo, and Zaiqing Nie
NeurIPS 2023.

This repository contains the official Pytorch implementation of training & evaluation code and the pretrained models for FFNET.

FFNET is a simple, efficient and powerful VIC3D Object Detection method, as shown in Figure 1.

We use MMDetection3D v0.17.1 as the codebase.
We evaluation all the models with OpenDAIRV2X.

Installation

For more information about installing mmdet3d, please refer to the guidelines in MMDetectionn3D v0.17.1. For more information about installing OpenDAIRV2X, please refer to the guideline in OpenDAIRV2X.

Other requirements: pip install --upgrade git+https://github.com/klintan/pypcd.git

An example (works for me): CUDA 11.1 and pytorch 1.9.0

pip install torchvision==0.10.0
pip install mmcv-full==1.3.14
pip install mmdet==2.14.0
pip install mmsegmentation==0.14.1
cd FFNET-VIC3D && pip install -e . --user

Data Preparation

We train and evaluate the models on DAIR-V2X dataset. For downloading DAIR-V2X dataset, please refer to the guidelines in DAIR-V2X. After downloading the dataset, we should preprcocess the dataset as the guidelines in data_preprocess. We provide the preprocessed example data example-cooperative-vehicle-infrastructure, you can download and decompress it under './data/dair-v2x'.

Evaluation

Step 1: Download trained weights.

• FFNET trained on DAIR-V2X (FFNET Trainded Checkpoint | FFNET without prediction | FFNET-V2 without prediction)

• FFNET trained on V2X-Seq-SPD (ffnet-v2x-spd.pth | ffnet-pretrained-v2x-spd.pth)

Step 2: Inference

• Please refer OpenDAIRV2X for evaluating FFNet with OpenDAIRV2X on DAIR-V2X-C dataset.

• Example: evaluate FFNET on DAIR-V2X-C-Example with 100ms latency:

  # modify the DATA to point to DAIR-V2X-C-Example in script ${OpenDAIRV2X_root}/v2x/scripts/lidar_feature_flow.sh
  # bash scripts/lidar_feature_flow.sh [YOUR_CUDA_DEVICE] [YOUR_FFNET_WORKDIR] [DELAY_K] 
  cd ${OpenDAIRV2X_root}/v2x
  bash scripts/lidar_feature_flow.sh 0 /home/yuhaibao/FFNet-VIC3D 1
  

Training

Step1: Train the basemodel on DAIR-V2X without latency

• Single-gpu training

  cd ${FFNET-VIC_repo}
  export PYTHONPATH=$PYTHONPATH:./
  CUDA_VISIBLE_DEVICES=${GPU_ID} python tools/train.py configs/config_basemodel.py
  

• or, Multi-gpu Training

  cd ${FFNET-VIC_repo}
  export PYTHONPATH=$PYTHONPATH:./
  CUDA_VISIBLE_DEVICES=${GPU_IDs} bash tools/dist_train.sh configs/config_basemodel.py ${GPU_NUM}
  

Step 2: Put the trained basemodel in a folder ffnet_work_dir/release-checkpoints as ffnet_work_dir/release-checkpoints/ffnet-pretrained.pth.

Step 3: Train FFNET on DAIR-V2X with latency

• Single-gpu training (Not supported multi-GPU training now)

  cd ${FFNET-VIC_repo}
  export PYTHONPATH=$PYTHONPATH:./
  CUDA_VISIBLE_DEVICES=${GPU_ID} python tools/train.py configs/config_ffnet.py
  

Citation

@inproceedings{yu2023ffnet,
  title={Flow-Based Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection},
  author={Yu, Haibao and Tang, Yingjuan and Xie, Enze and Mao, Jilei and Luo, Ping and Nie, Zaiqing},
  booktitle={Advances in Neural Information Processing Systems},
  year={2023}
}

@inproceedings{yu2023ffnet,
  title={Vehicle-Infrastructure Cooperative 3D Object Detection via Feature Flow Prediction},
  author={Yu, Haibao and Tang, Yingjuan and Xie, Enze and Mao, Jilei and Yuan, Jirui and Luo, Ping and Nie, Zaiqing},
  booktitle={https://arxiv.org/abs/2303.10552},
  year={2023}
}