Artificial Kuramoto Oscillatory Neurons (AKOrN)

Takeru Miyato · Sindy Löwe · Andreas Geiger · Max Welling

arXiv

This page contains the initial environment setup and code for the CLEVR-Tex experiments.

• Code for other synthetic datasets (Tetrominoes, dSprits, CLEVR): here

Setup Conda env

yes | conda create -n akorn python=3.12
conda activate akorn
pip3 install -r requirements.txt

Donwload the CLEVRTex dataset

cd data
bash download_clevrtex.sh
cd ..

Training

export NUM_GPUS=<number_of_gpus> # If you use a single GPU, run a command without the multi GPU option  (`--multi-gpu`).

CLEVRTex

AKOrN

export L=1 # The number of layers. L=1 or 2. This can be >2, but we only experimented with a single or two-layer model.
accelerate launch --multi-gpu --num_processes=$NUM_GPUS  train_obj.py --exp_name=clvtex_akorn --data_root=./data/clevrtex_full/ --model=akorn --data=clevrtex_full --J=attn --L=${L}

# Larger model (L=2, ch=512, bs=512)
accelerate launch --multi-gpu --num_processes=$NUM_GPUS  train_obj.py --exp_name=clvtex_large_akorn --data_root=./data/clevrtex_full/ --model=akorn --data=clevrtex_full --J=attn --L=2 --ch=512 --batchsize=512 --epochs=1024 --lr=0.0005

ItrSA

export L=1
accelerate launch --multi-gpu --num_processes=$NUM_GPUS  train_obj.py --exp_name=clvtex_itrsa --data_root=./data/clevrtex_full/ --model=vit --data=clevrtex_full --L=${L} --gta=False

Evaluation

CLEVRTex (-OOD, -CAMO)

export DATA_TYPE=full #{full, outd, camo}
export L=1
# AKOrN
python eval_obj.py  --data_root=./data/clevrtex_${DATA_TYPE}/  --model=akorn  --data=clevrtex_${DATA_TYPE} --J=attn --L=${L} --model_path=runs/clvtex_akorn/ema_499.pth --model_imsize=128
# ItrSA
python eval_obj.py  --data_root=./data/clevrtex_${DATA_TYPE}/  --model=vit  --data=clevrtex_${DATA_TYPE} --gta=False --L=${L} --model_path=runs/clvtex_itrsa/ema_499.pth --model_imsize=128

Eval with Up-tiling (See Appendix section).

# Might take long time depending on the CPU spec
python eval_obj.py  --data_root=./data/clevrtex_${DATA_TYPE}/  --saccade_r=4 --model=akorn  --data=clevrtex_${DATA_TYPE} --J=attn --L=${L} --model_path=runs/clvtex_akorn/ema_499.pth --model_imsize=128

Performance table

Model	CLEVRTex FG-ARI	CLEVRTex MBO	OOD FG-ARI	OOD MBO	CAMO FG-ARI	CAMO MBO
ViT	46.4±0.6	25.1±0.7	44.1±0.5	27.2±0.5	32.5±0.6	16.1±1.1
ItrSA (L = 1)	65.7±0.3	44.6±0.9	64.6±0.8	45.1±0.4	49.0±0.7	30.2±0.8
ItrSA (L = 2)	76.3±0.4	48.5±0.1	74.9±0.8	46.4±0.5	61.9±1.3	37.1±0.5
AKOrN (attn, L = 1)	75.6±0.2	55.0±0.0	73.4±0.4	56.1±1.1	59.9±0.1	44.3±0.9
AKOrN (attn, L = 1)	80.5±1.5	54.9±0.6	79.2±1.2	55.7±0.5	67.7±1.5	46.2±0.9

With Up-tiling (x4)

Model	CLEVRTex FG-ARI	CLEVRTex MBO	OOD FG-ARI	OOD MBO	CAMO FG-ARI	CAMO MBO
AKOrN (attn, L = 2)	87.7±1.0	55.3±2.1	85.2±0.9	55.6±1.5	74.5±1.2	45.6±3.4
Large AKOrN (attn, L = 2)	88.5±0.9	59.7±0.9	87.7±0.5	60.8±0.6	77.0±0.5	53.4±0.7