Skip to content

shiyegao/Self-Contradictory-Instructions-SCI

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 

Repository files navigation

Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions

Introduction

Official github repo of the ECCV24 paper, "Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions".

This repo provides the code for generating Self-Contradictory-Instruction(SCI) dataset.

  • LL: This section of code provides the generation scripts for Language-Language Conflicts
  • VL: This section of code provides the generation scripts for Vision-Language Conflicts

File Structure

The basic file structure is shown as follows:

SCI
├── LL
│   ├── *.yaml
│   ├── generate.py
│   ├── information_conflict.py
│   ├── instruction_conflict.py
│   ├── seeds.json
│   └── utils.py
├── README.md
└── VL
    ├── *.yaml
    ├── generate.py
    ├── imageNet_class.py
    ├── tools.py
    └── visionlanguage.py

Installation

conda create -n SCI python=3.9 
conda activate SCI
conda install pyyaml tqdm openai numpy matplotlib

Usage

Two scripts are available for the generation of SCI:

  1. ./LL/generate.py generates Language-Language(L-L) conflicts.
  2. ./VL/generate.py generates Vision-Language(V-L) conflicts.

L-L conflict generation

./LL/generate.py generates only one type of subtask at a time. Details about the generation process can be set with the following optional arguments:

--task must be 1, 2, 3 or 4, for generating OCR Conflict, Attribute Conflict, Exclusion Conflict and Forbidden Conflict.

--total_conflicts sets the total number of conflicts required.

--config is the configure file for openai API.

--dataset_path is the target path to save the generated conflicts.

Here is an example that generates 2500 conflicts for Rule Conflict, and saves them in ../SCI/Rule

python generate.py --task 1 --total_conflicts 2500 --config openai.yaml --dataset_path ../SCI/Rule

V-L conflict generation

./VL/generate.py generates only one type of subtask at a time. Details about the generation process can be set with the following optional arguments:

--task must be 1, 2, 3 or 4, for generating OCR Conflict, Figure Conflict, Geometric Conflict and Semantic Conflict.

--total_num sets the total number of conflicts required, only valid for task1 and task3.

--config is the configure file for openai API.

--target_dir is the target path to save the generated conflicts.

Here is an example that generates 2000 Geometric Conflicts, and saves them in ../SCI/OCR

python generate.py --task 3 --total_num 2000 --config openai.yaml --dataset_path ../SCI/OCR

OPENAI API

To use scripts above, a configure file for openai API is required.

Specifically, a .yaml file with the following information is required.

Here is an example of openai.yaml.

OPENAI_KEY: your_openai_key
MODEL_NAME: 'your desired model'
OPENAI_API_BASE: 'your api base'

MAX_RETRY : 10

LMM Evaluation

Many LMMs are tested on SCI in our paper. You can access them through APIs or local deployment.

GPT4-V/GPT-4、ChatGPT

ChatGLM/GLM4

Gemini

SPHINX-v2

LLaVA-1.5

LLaMA-Adapter v2

BLIP-2

Citation

If our code or models help your work, please cite our paper.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages