How can I train two or more differnt ForecastDFDataset at the same time?

[yhlee52]

I am working with data that consists of multiple CSV files with the same characteristics for each column but different collection environments. Since combining individual CSV files into a single dataset creates an inappropriate dataset at the merging point, I would like to know how to create ForecastDFDataset objects from multiple individual CSV files and combine them together for training at once. Is there a way to combine the generated ForecastDFDataset objects into one or to put multiple ForecastDFDataset objects into the trainer after model? Such as 'train_datasets' in below
self.trainer = Trainer( model = self.model, args = self.training_args, train_dataset = train_datasets, eval_dataset = eval_dataset, )

[Dylan0211]

I am working with data that consists of multiple CSV files with the same characteristics for each column but different collection environments. Since combining individual CSV files into a single dataset creates an inappropriate dataset at the merging point, I would like to know how to create ForecastDFDataset objects from multiple individual CSV files and combine them together for training at once. Is there a way to combine the generated ForecastDFDataset objects into one or to put multiple ForecastDFDataset objects into the trainer after model? Such as 'train_datasets' in below self.trainer = Trainer( model = self.model, args = self.training_args, train_dataset = train_datasets, eval_dataset = eval_dataset, )

I also face this issue and my solution is to use ConcatDataset from torch to combine these ForecastDFDataset objects.

[yhlee52]

I tried the method you suggested and it seems to work as intended.
I was thinking of a difficult method, but I had no idea there was such a simple way.
Thank you very much @Dylan0211 !

[wgifford]

Hi @yhlee52 @Dylan0211

This functionality is built into ForecastDFDataset -- it supports distinguishing time series by IDs. You can add an ID column (calling it "id" for example) to each of your CSV files (each CSV receiving a unique ID value) before you concatenate them, and then use the id_columns = ["id"] parameter to point to this column when you instantiate ForecastDFDataset. Internally ForecastDFDataset will use ConcatDataset to separate the time series.

Other components in our library support the id_columns argument to make working with multi-time series datasets easier.

[yhlee52]

Hi @wgifford , it is very helpful features! I will try to fix my codes to handle my data with the arg. Thank you!