Metric embeddings for satellite image classification

This repository contains methods for generating and evaluating metric embeddings on satellite images.

Overview

Sampling

1. Random sampling
   Choose every tile randomly across the entire map. Source: Sentinel 2 Cloudless

2. Cluster sampling
   Choose tiles in clusters. (required for unsupervised magnet) Source: Sentinel 2 Cloudless

Mining

Hard Mining

Hermans, Alexander, L. Beyer und B. Leibe (2017). In defense of the
triplet loss for person re-identification. arXiv preprint arXiv:1703.07737.

1. Hard online mining
   Find hard samples within the batch.

2. Hard offline mining
   Random search for hard samples over the map.

TODO: More mining methods from PyTorch metric learning

Networks

1. resnext
2. ViT

TODO: Integrate model zoo TODO: EfficientNet TODO: DEiT

Loss functions

1. TripletLoss
2. QuadrupletLoss
3. MagNetLoss

Supervised/Unsupervised

1. Train supervised with label from OSM/CLC
2. Train unsupervised with distance of tile (Tile2Vec)

   Jean, Neal, S. Wang, A. Samar, G. Azzari, D. Lobell und S. Ermon
   (2019). Tile2Vec: Unsupervised representation learning for spatially distributed data.
   In: Proceedings of the AAAI Conference on Artificial Intelligence, Bd. 33, S. 3967–3974.
   

Evaluation

1. Single Class Evaluation
   1. Accuracy
   2. MAP@R
2. Multi Class Evaluation
   1. Sum error
   2. MAP@R
      Multi label extension with the key idea of gSil
   3. gSil
3. TODO: Semantic segmentation

Train

Data

Place the maps or edit dataProcessors.tiffreader.GEOMAP

	[SENTINEL](https://s2maps.eu) = 'data/s2-de.tif'
	[OSM](https://osmlanduse.org) = 'data/osm_WGS84.tif'
	[CLC](https://land.copernicus.eu/pan-european/corine-land-cover) = 'data/CLCBGR.tif'

Default map is germany. Change rect in dataProcesor.imageSampler.ImageSampler

Config

Edit Config file

{
	// neural network architecture
	"network": "transformer|resnext50",

	// loss funktion 
	"loss": "tripletloss|quadrupletloss|magnetloss",

	// use softmax distance (True) or euclidian (False)
	"alt_loss": "True|False",

	// supervised classifier and regressors for validation
	"single_class_classifiers": ["knn", "svc", "rfc", "mlp", "dt"],
	"multi_class_classifiers": ["knn", "svc", "rfc", "mlp", "dt"],

	// tile size of samples
	"image_size": 96,
	"batch_size": 24,
	"mining": ["HARD_NEGATIVE", "HARD_POSITIVE", "RANDOM_HARD_NEGATIVE", "RANDOM_HARD_POSITIVE"],

	// sampling distances for unsupervised learning (distance = image_size*dist)
	"min_pos_dist": 1.0,
	"max_pos_dist": 5.0,
	"min_neg_dist": 10.0,
	"max_neg_dist": 25.0,

	// margin of triplet and quadrupletloss
	"margin": 0.5,

	// number of tiles for validation
	"validation_nb": 5000,

	// label map for validation
	"validation_map": "osm|clc",

	// L2-Norm on network output
	"norm_output": "True|False",

	// use map labels of validation_map (True) or tile distances (False) for training 
	"supervised": "True|False",

	// size of output vector of neural network
	"output_size": 64,

	// use in validation only tiles that dominated by a single class (minimum proportion of the class in the tile)
	"singleClassTreshold": 0.0,

	// run the validation multiple times on different images
	"valIteration": 5,

	// 1.0 for random sampling | 0.0 for cluster sampling
	"random_reset": 1.0

}

Start training

python main.py

Show Evaluation

tensorboard --port XXXX --logdir results/XXXX

Results

TODO: Add more results

Best tested results unsupervised

{
	"network": "resnext50",
	"loss": "magnetloss",
	"alt_loss": "False",
	"single_class_classifiers": ["rfc", "knn"],
	"multi_class_classifiers": ["rfc", "knn"],
	"image_size": 96,
	"batch_size": 12,
	"mining": [],
	"min_pos_dist": 1.0,
	"max_pos_dist": 5.0,
	"min_neg_dist": 10.0,
	"max_neg_dist": 25.0,
	"margin": 0.5,
	"validation_nb": 5000,
	"validation_map": "osm",
	"norm_output": "True",
	"supervised": "False",
	"output_size": 64,
	"singleClassTreshold": 0.0|0.8,
	"valIteration": 5,
	"random_reset": 0.0
}

Single label accuracy: 0.67
Multi label MAP@1: 0.44
Multi label MAP@10: 0.35
Multi label summed error: 1.08

osm label images in 3D embedding space, dimension reduction with PCA

Best tested results supervised

{
	"network": "resnext50",
	"loss": "tripletloss",
	"alt_loss": "False",
	"single_class_classifiers": ["rfc", "knn"],
	"multi_class_classifiers": ["rfc", "knn"],
	"image_size": 96,
	"batch_size": 12,
	"mining": [],
	"min_pos_dist": 1.0,
	"max_pos_dist": 5.0,
	"min_neg_dist": 10.0,
	"max_neg_dist": 25.0,
	"margin": 0.5,
	"validation_nb": 5000,
	"validation_map": "osm",
	"norm_output": "True",
	"supervised": "True",
	"output_size": 64,
	"singleClassTreshold": 0.0,
	"valIteration": 5,
	"random_reset": 0.0
}

Single label accuracy: 0.77
Multi label MAP@1: 0.72
Multi label MAP@10: 0.7
Multi label summed error: 0.54

osm label images in 3D embedding space, dimension reduction with PCA