ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes

ICCV 2023

from dataset import ArnoldDataset, prepare_batch

cfg = {}
train_dataset = ArnoldDataset('path_to_a_task_train or test or val_folder', task, cfg, canonical=False)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, pin_memory=True,num_workers=min(batch_size,os.cpu_count()//world_size), sampler=train_sampler, shuffle=shuffle, persistent_workers=True)

for batch_ndx, sample in enumerate(train_loader):
            batch = {k: v.to(device, non_blocking = True, dtype=torch.float) for k, v in sample.items() if type(v) == torch.Tensor}
            batch["language"] = sample["language"]
            batch['ignore_collisions'] = torch.zeros(batch_size, 1).to(device).long()
            batch = prepare_batch(train_dataset, batch)
            # rest of the code

Ran Gong^✶, Jiangyong Huang^✶, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu
Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, Baoxiong Jia, Siyuan Huang

 

 

we present ARNOLD, a benchmark for language-grounded task learning with continuous states in realistic 3D scenes. We highlight the following major points:

• ARNOLD is built on NVIDIA Isaac Sim, equipped with photo-realistic and physically-accurate simulation, covering 40 distinctive objects and 20 scenes.
• ARNOLD is comprised of 8 language-conditioned tasks that involve understanding object states and learning policies for continuous goals. For each task, there are 7 data splits, including unseen generalization.
• ARNOLD provides 10k expert demonstrations with diverse template-generated language instructions, based on thousands of human annotations.
• We assess the task performances of the latest language-conditioned policy learning models. The results indicate that current models for language-conditioned manipulation still struggle in understanding continuous states and producing precise motion control. We hope these findings can foster future research to address the unsolved challenges in instruction grounding and precise continuous motion control.

We provide brief guidance on this page. Please refer to our documentation for more information about ARNOLD.

Get Started

There are two setup approaches: docker-based and conda-based. We recommend the docker-based approach as it wraps everything up and is friendly to users. See step-by-step instructions here.

After setup, you can refer to quickstart for a glance of using ARNOLD.

Major components of the ARNOLD environment are introduced here. Based on this environment, you can check the tasks and data.

We use hydra for configurations of the experiments. See configs. After double-checking the configurations, you can explore the [training] and [evaluation] on your own.

TODO

• Demonstration generator.

BibTex

@inproceedings{gong2023arnold,
  title={ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes},
  author={Gong, Ran and Huang, Jiangyong and Zhao, Yizhou and Geng, Haoran and Gao, Xiaofeng and Wu, Qingyang and Ai, Wensi and Zhou, Ziheng and Terzopoulos, Demetri and Zhu, Song-Chun and others},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2023}
}