BriVL for application development

This repo is used for illustrating how to build applications by using BriVL model.

This repo is re-implemented from following projects:

• Source Code of BriVL 1.0
• Model of BriVL 1.0
• Bottom Up Attention For Application
• bottom-up-attention
• bottom-up-attention.pytorch

Online Demo built by BriVL

• AI's Imaginary World

• Soul Music

• Few Word

Contents

This repo contains two parts:

• Bounding Box Extractor: ./bbox_extractor
• BriVL Feature Extractor: ./BriVL

Test this Pipeline

Test image has been saved in ./bbox_extractor/feature_extractor, test with following command:

python3 main.py --brivl_cfg BriVL/cfg/BriVL_cfg.yml --brivl_weights BriVL/weights/brivl-weights.pth

Download Models

• bua-caffe-frcn-r101_with_attributes.pth -> /bbox_extractor/weights
• chinese-roberta-wwm-ext -> /BriVL/weights/hfl
• tf_efficientnet_b5_ns-6f26d0cf.pth -> /BriVL/weights
• brivl-weights.pth* -> /BriVL/weights

Requirements

• Python >= 3.6
• PyTorch >= 1.4
• Cuda >= 9.2 and cuDNN
• Detectron2 <= 0.3
• Transformers

Important: The version of Detectron2 should be 0.3 or below.

Install Pre-Built Detectron2 (Linux only)

Choose from this table to install v0.3 (Nov 2020):

CUDA	torch 1.7	torch 1.6	torch 1.5
11.0	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html
10.2	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.7/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.6/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu102/torch1.5/index.html
10.1	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.7/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.6/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.5/index.html
9.2	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu92/torch1.7/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu92/torch1.6/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cu92/torch1.5/index.html
cpu	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/torch1.7/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/torch1.6/index.html	install python -m pip install detectron2==0.3 -f \ https://dl.fbaipublicfiles.com/detectron2/wheels/cpu/torch1.5/index.html

More Resources

Source Code of BriVL 1.0

Model of BriVL 1.0*

Online API of BriVL 1.0

Online API of BriVL 2.0

* indicates an application is needed.

Contact

This repo is maintained by Chuhao JIn(@jinchuhao).