Code for paper "KOSMOS-E : Learning to Follow Instruction for Robotic Grasping" at IEEE International Conference on Intelligent Robots and System (IROS), 2024.

[website] [paper] [video]

git clone https://github.com/TX-Leo/kosmos-e.git
cd kosmos-e
bash vl_setup_xl.sh

We create INSTRUCT-GRASP dataset based on Cornell Grasping Dataset. It includes three components: Non, Single and Multi with 8 kinds of intructions. It has 1.8 million grasping samples, with 250k unique language-image non-instruction samples and 1.56 million instruction-following samples. Among these instruction-following samples, 654k pertain to the single-object scene, while the remaining 654k relate to the multi-object scene. You can download the dataset HERE (coming soon).

The dataset structure:

• INSTRUCT-GRASP
  • INSTRUCT-GRASP-NON-SINGLE
    • 01-10
      • pcdxxxx
        • pcdxxxx_xx_xxgrasp_xya_encoded_with_instruction_xxxxxxx.tsv
        • pcdxxxx_xx_xxgrasp_xya_encoded.tsv
        • pcdxxxx_xx_xxgrasp_r.png
        • pcdxxxx_xx_xxgrasp_rgrasp.png
      • else
        • instructions.json
  • INSTRUCT-GRASP-MULTI
    • 01-10
      • pcdxxxx
        • pcdxxxx_xx_xxgrasp_xya_encoded_with_instruction_xxxxxxx.tsv
        • pcdxxxx_xx_xxgrasp_r.png
        • pcdxxxx_xx_xxgrasp_rgrasp.png
    • else
      • instructions.json
  • dataloder
    • dataloader_config

The checkpoint can be downloaded from HERE (coming soon):

After downloading the dataset, you should change the --laion-data-dir to the config directory path and --save-dir to the directory path saving models, --tensorboard-logdir to the directory path saving tensorboard logs in run_train.sh.

bash local_mount.sh
bash vl_setup_xl.sh
bash run_train.sh

We evaluate our model KOSMOS-E on the INSTRUCT-GRASP Dataset.

cd ../evaluation
bash vl_setup.sh

You can modify evaluation parameters in \eval\eval_cornell.py, mainly focusing on:

• dataset_path
• dataloader_num
• train_output_num
• instruction_type (angle/part/name/color/shape/purpose/position/strategy)

bash run_eval_cornell.sh

We follow a cross-validation setup as in previous works and partition the datasets into 5 folds

Our model was trained using a combination of non-instruction and instruction-following datasets. In contrast, four other baselines were each trained on a distinct dataset: non-instruction, single-object, multi-object, and a combination of single-object and multi-object datasets. We adopted image-wise grasp accuracy as our primary evaluation metric.

There are some instruction-following grasping examples which includes single-object examples and multi-object examples.