Official github repo of the ECCV24 paper, "Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions".
- Website: https://selfcontradiction.github.io
- Dataset: https://huggingface.co/datasets/sci-benchmark/self-contradictory
This repo provides the code for generating Self-Contradictory-Instruction(SCI) dataset.
LL
: This section of code provides the generation scripts for Language-Language ConflictsVL
: This section of code provides the generation scripts for Vision-Language Conflicts
The basic file structure is shown as follows:
SCI
├── LL
│ ├── *.yaml
│ ├── generate.py
│ ├── information_conflict.py
│ ├── instruction_conflict.py
│ ├── seeds.json
│ └── utils.py
├── README.md
└── VL
├── *.yaml
├── generate.py
├── imageNet_class.py
├── tools.py
└── visionlanguage.py
conda create -n SCI python=3.9
conda activate SCI
conda install pyyaml tqdm openai numpy matplotlib
Two scripts are available for the generation of SCI:
./LL/generate.py
generates Language-Language(L-L) conflicts../VL/generate.py
generates Vision-Language(V-L) conflicts.
./LL/generate.py
generates only one type of subtask at a time. Details about the generation process can be set with the following optional arguments:
--task
must be 1, 2, 3 or 4, for generating OCR Conflict, Attribute Conflict, Exclusion Conflict and Forbidden Conflict.
--total_conflicts
sets the total number of conflicts required.
--config
is the configure file for openai API.
--dataset_path
is the target path to save the generated conflicts.
Here is an example that generates 2500 conflicts for Rule Conflict, and saves them in ../SCI/Rule
python generate.py --task 1 --total_conflicts 2500 --config openai.yaml --dataset_path ../SCI/Rule
./VL/generate.py
generates only one type of subtask at a time. Details about the generation process can be set with the following optional arguments:
--task
must be 1, 2, 3 or 4, for generating OCR Conflict, Figure Conflict, Geometric Conflict and Semantic Conflict.
--total_num
sets the total number of conflicts required, only valid for task1 and task3.
--config
is the configure file for openai API.
--target_dir
is the target path to save the generated conflicts.
Here is an example that generates 2000 Geometric Conflicts, and saves them in ../SCI/OCR
python generate.py --task 3 --total_num 2000 --config openai.yaml --dataset_path ../SCI/OCR
To use scripts above, a configure file for openai API is required.
Specifically, a .yaml
file with the following information is required.
Here is an example of openai.yaml
.
OPENAI_KEY: your_openai_key
MODEL_NAME: 'your desired model'
OPENAI_API_BASE: 'your api base'
MAX_RETRY : 10
Many LMMs are tested on SCI in our paper. You can access them through APIs or local deployment.
If our code or models help your work, please cite our paper.