finetuning using notebook on custom dataset

[amoghskanda]

System Info

python 3.10.15
torch 2.5.1
transformers 4.46.2
tokenizers 0.20.3

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

I had to finetune llama3.2 11B Vision Instruct and I downloaded the model from huggingface(https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct)> I'm trying to finetune the model on a custom dataset of mine by following the finetuning notebook. When I start finetuning, I run into list conversion to tensor issue which I'm guessing is because the dataset is not in the right format. Could anybody suggest the dataset format?
I have ~4k images, metadata.csv which contains 20 columns encompassing all the information about the images, a prompt for finetuning.
The code I used for generating the dataset :

import os
import pandas as pd
from datasets import Dataset, DatasetDict
from transformers import AutoTokenizer
from PIL import Image
from torchvision import transforms
import torch

image_folder = 'path to images folder'
csv_file = 'path to metadata.csv'
prompt = "The prompt used for FT"
metadata = pd.read_csv(csv_file)
metadata['image_path'] = metadata['file_name'].apply(lambda x: os.path.join(image_folder, x))


def load_image(image_path):
    image = Image.open(image_path).convert("RGB")
    return image

def preprocess_image(image):
    transform = transforms.Compose([
        transforms.Resize((224, 224)),
        transforms.ToTensor(),
    ])
    return transform(image)


def tokenize_prompt(prompt, tokenizer):
    return tokenizer(prompt, return_tensors="pt", padding="max_length", truncation=True, max_length=512)


tokenizer = AutoTokenizer.from_pretrained("path to llama model")

data = []
for idx, row in metadata.iterrows():
    image_path = os.path.join(image_folder, row["image_path"])
    image = load_image(image_path)
    image = preprocess_image(image)

    tokenized_prompt = tokenize_prompt(prompt, tokenizer)
    
    data_entry = {
        "image": image,
        "text": prompt,
        "input_ids": tokenized_prompt["input_ids"].squeeze().tolist(),
        "attention_mask": tokenized_prompt["attention_mask"].squeeze().tolist(),
        "metadata": row.to_dict()
    }
    data.append(data_entry)

dataset = Dataset.from_pandas(pd.DataFrame(data))

dataset_dict = DatasetDict({
    "train": dataset
})

dataset_dict.save_to_disk("train_dataset")

Error logs

{
"name": "AttributeError",
"message": "'list' object has no attribute 'to'",
"stack": "---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[9], line 15
12 scheduler = StepLR(optimizer, step_size=1, gamma=train_config.gamma)
14 # Start the training process
---> 15 results = train(
16 model,
17 train_dataloader['train'],
18 eval_dataloader['test'],
19 tokenizer,
20 optimizer,
21 scheduler,
22 train_config.gradient_accumulation_steps,
23 train_config,
24 None,
25 None,
26 None,
27 wandb_run=None,
28 )

File ~/anaconda3/envs/llama/lib/python3.10/site-packages/llama_recipes/utils/train_utils.py:151, in train(model, train_dataloader, eval_dataloader, tokenizer, optimizer, lr_scheduler, gradient_accumulation_steps, train_config, fsdp_config, local_rank, rank, wandb_run)
149 batch[key] = batch[key].to('xpu:0')
150 elif torch.cuda.is_available():
--> 151 batch[key] = batch[key].to('cuda:0')
152 with autocast():
153 loss = model(**batch).loss

AttributeError: 'list' object has no attribute 'to'"
}

I have tried keeping input_ids and attention_mask as pytorch tensors but there was a problem during conversion of tensors to arrow objects during dataset creation.

Expected behavior

Any guide on how to create a dataset compatible with llama3.2 11B Vision Instruct with images, metadata and a prompt

[HamidShojanazeri]

cc: @wukaixingxp

[wukaixingxp]

@amoghskanda You need to convert list into tensor, something like batch["labels"] = torch.tensor(label_list). Please check this example about how to convert the dialogs into tokens