Split Dataset in RandomWindowGeoDataset and SlidingWindowGeoDataset

[lcoandrade]

What would be the best approach to generate a dataset considering that I have 2 folders in my dataset (gt and images). I would like to create the dataset with 60% of it as a RandomWindowGeoDataset to train and 20% as SlidingWindowGeoDataset to validate and test.

So far, considering that I'm running my code on Kaggle (meaning that I haven't set GDAL explicitly), my code is like this:

DataTuple = namedtuple("TrainingTuple", "image label")

def define_segmentation_data(imageroot, labelroot):
    training_dict = {}
    for file in os.listdir(labelroot):
        training_dict[file] = DataTuple(
            image=os.path.join(imageroot, file),
            label=os.path.join(labelroot, file),
        )
    return training_dict

training_dict = define_segmentation_data(
    os.path.join(INPUT_DIR, TRAIN_DIR, IMG_DIR),
    os.path.join(INPUT_DIR, TRAIN_DIR, LABEL_DIR),
)
print(len(training_dict))

def make_random_window_dataset_from_dict(data_dict):
    datasets = []
    for file in data_dict.keys():    
        dataset = SemanticSegmentationRandomWindowGeoDataset.from_uris(
            class_config=class_config,
            image_uri=data_dict[file].image,
            label_raster_uri=data_dict[file].label,
            size_lims=(IMG_SIZE, 2*IMG_SIZE),
            out_size=IMG_SIZE,
            max_windows=MAX_WINDOWS,
            label_raster_source_kw=dict(raster_transformers=[ReclassTransformer({255: 1})]),
        )
        datasets.append(dataset)
    return ConcatDataset(datasets)

# Making out dataset
concat_dataset = make_random_window_dataset_from_dict(training_dict)
print('Training dataset: ', len(concat_dataset))

train_set, val_set, test_set = torch.utils.data.random_split(concat_dataset, [0.6, 0.2, 0.2])

[AdeelH]

You could manually split the files into separate train, val, and test splits. You could do that by moving them to separate directories and then running your define_segmentation_data() on each of them to get separate dicts for each split. Alternatively, you could just split the list of files.