Release version 0.2.2 (#348)

* Add files via upload

* implementing new CRS management with pyproj

* loosening version restrictions

* removing WIP GeoTIFFDataset added by accident

* implementing new CRS management with pyproj

* Update environment-gpu.yml

* loosening version restrictions

* removing WIP GeoTIFFDataset added by accident

* pinning to dev version of shapely

* debugging syntax

* debugging requirements and setup.py - diffs btwn conda and pypi

* pinning shapely to 1.7.1dev

* adding dependency link

* finally got shapely installed from gh

* why is this still not working

* fixing pip install flags to accommodate git install

* trying another way to get shapely...

* fixing pyyaml reqt

* hotfix split_multi_geoms()

* hotfix split_multi_geoms()

* fixing split_multi_geometries and adding test

* debugged split_multi_geometries and wrote new tests

* adding dependency_link for shapely in setup.py

* fixing version regex to include characters, e.g. "0.2.2dev0"

* [WIP] Bug fixes, geojson2coco improvements, restrict_by_aoi, and nodata filling (#331)

* added nodata threshold option for raster tiler for cases where large portions of image have nodata

* fixed issue where the aoi boundary was not intersected with src_img bounds prior to generating tile bounds list and added warnign about multipoly

* reverse sign on nodata threshold check and add a docstring to tile function

* impements restrict_to_aoi in rastertiler

* fixed naming of raster tiles in case restrict_to_aoi is used

* original tests pass for raster tiler

* added remove multipolygon option to geojson2coco, they cause it to fail

* fill_no_data func for tiler with options for mean, arbitrary value

* added remove multipolygon function for geojson2coco

* option to override crs in geojson_to_px_gdf to fix for epsg being used for wkt projections when gdfs are saved as geojson and fix for missing crs info in vector tiler

* add override options for coco

* bug fixes for tiler when fill no data is used, nodata_percentage is calculated correctly now

* added comment on not cropping in tiler

* return fill values from raster tiler

* bug fix for fill_all_nodata and instance mask should return empty arr with same shape as reference_im

* added filtering for geometry collections to remove_multipolygon function and updated description for geojson2coco

* bug fix for case where fill value is not the mean

* new function for nodata aware instance mask conversion with tiler inputs, test for instance masking, handling necessary use of rasterio crs in _check_crs

* Fixing torchvision pin in environment.yml

* fixed test

* update CHANGELOG, environments for shapely dev for np support, removed coco crs override, update tqdm

* small updates, tests pass

* review updates: tqdm import switch, explode multipoly option, _check_crs change

* fixed get_overlapping_subset to use new return_Rasterio flag

* added override crs option back in to geojson_to_coco

* add chceck for src gdf to have a crs attribute

Co-authored-by: Ubuntu <ryan@cropmask-mgpu-18-v2.03lau3lhyktubpzetswi1ohm1h.xx.internal.cloudapp.net>
Co-authored-by: Nick Weir <[email protected]>

* Pinning GDAL version to resolve compatibility issue with Fiona

* Fix stage of val_datagen in Trainer (#345)

* Fix stage of val_datagen in Trainer

* updating changelog and solaris version

Co-authored-by: jshermeyer <[email protected]>
Co-authored-by: Ryan Avery <[email protected]>
Co-authored-by: Ubuntu <ryan@cropmask-mgpu-18-v2.03lau3lhyktubpzetswi1ohm1h.xx.internal.cloudapp.net>
Co-authored-by: Daniel Hogan <[email protected]>
Co-authored-by: Kaizaburo Chubachi <[email protected]>

.travis.yml

@@ -32,7 +32,8 @@ install:
   - conda list -n solaris
   - source activate solaris
   - python --version
-  - pip install -q -e .[test]
+  - pip install --user -r requirements.txt -vv
+  - pip install .[test] -vv
   - pip install codecov pytest-cov pytest==5.3.1
 # command to run tests
 script:

CHANGELOG.md

@@ -25,6 +25,24 @@ When a new version of `solaris` is released, all of the changes in the Unrelease
 
 ---
 
+## Version 0.2.2
+
+### Changed
+- 20200401, rbavery: new tiler method `fill_all_nodata` to fill nodata with `"mean"` or custom value (#328)
+- 20200401, rbavery: option to ignore MultiPolygon and GeometryCollection types in `geojson2coco` since these cannot be converted to COCO.
+- 20200401, rbavery: new function `solaris.vector.mask.geojsons_to_masks_and_fill_nodata`, which rasterized vector labels according to raster tile extents. Fills nodata areas in raster tile and corresponding rasterized label raster.
+- 20200401, rbavery: new test/example of tiling and creating instance masks with nodata values filled in tile outputs
+
+### Fixed
+- 20200401, rbavery: restrict_to_aoi implemented, sets values outside aoi to nodata value (#327, #240)
+- 20200401, rbavery: tqdm prints correctly in notebook and lab (if ipywidgets is enabled according to https://github.com/tqdm/tqdm/issues/394#issuecomment-384743637) (#335)
+- 20200401, rbavery: fixed bug where aoi boundary was not intersected with src_img extent prior to tiling
+- 20200401, rbavery/nrweir: adapted `_check_crs` to convert `pyproj.CRS` or pass through `rasterio.crs.CRS` class when rasterio crs object is required (for example, reprojecting in the tilers)
+- 20200414, zaburo-ch: fixed `val_datagen` to point to the correct augmentation pipeline
+---
+
+---
+
 ## Version 0.2.1
 
 ### Changed

docs/conf.py

@@ -24,8 +24,8 @@
 copyright = u'2018-{}, CosmiQ Works: an IQT Lab'.format(time.strftime("%Y"))
 
 # The full version, including alpha/beta/rc tags
-release = '0.2.1'
-version = '0.2.1'
+release = '0.2.2'
+version = '0.2.2'
 
 # -- General configuration ---------------------------------------------------
 

environment-gpu.yml

@@ -9,25 +9,26 @@ dependencies:
   - affine>=2.3.0
   - albumentations=0.4.3
   - fiona>=1.7.13
-  - gdal>=3.0.2
-  - geopandas=0.6.2
+  - gdal=3.0.3
+  - geopandas>=0.7.0
   - matplotlib>=3.1.2
   - networkx>=2.4
-  - numpy>=1.17.3,<1.18.0
   - opencv>=4.1
   - pandas>=0.25.3
   - pyproj>=2.1
   - pytorch>=1.3.1
   - pyyaml=5.2
   - rasterio>=1.0.23
   - requests=2.22.0
-  - rio-cogeo>=1.1.6
+#  - rio-cogeo>=1.1.6
   - rtree>=0.9.3
   - scikit-image>=0.16.2
   - scipy>=1.3.2
-  - shapely>=1.6.4
   - torchvision>=0.5.0
   - tqdm>=4.40.0
   - urllib3>=1.25.7
   - tensorflow-gpu=1.13.1
   - cuda100
+  - pip
+  - pip:
+    - git+https://github.com/Toblerity/shapely.git@master#egg=shapely-1.7.1dev # temporary, dev required for numpy array support

environment.yml

@@ -9,19 +9,19 @@ dependencies:
   - affine>=2.3.0
   - albumentations=0.4.3
   - fiona>=1.7.13
-  - gdal>=3.0.2
-  - geopandas>=0.6.2
+  - gdal=3.0.3
+  - geopandas>=0.7.0
   - matplotlib>=3.1.2
   - networkx>=2.4
   - numpy>=1.17.3
   - opencv>=4.1
   - pandas>=0.25.3
   - pyproj>=2.1
-  - pytorch=1.3.1
+  - pytorch>=1.3.1
   - pyyaml=5.2
   - rasterio>=1.0.23
   - requests=2.22.0
-  - rio-cogeo>=1.1.6
+#  - rio-cogeo>=1.1.6
   - rtree>=0.9.3
   - scikit-image>=0.16.2
   - scipy>=1.3.2
@@ -30,3 +30,5 @@ dependencies:
   - torchvision>=0.5.0
   - tqdm>=4.40.0
   - urllib3>=1.25.7
+  - pip:
+    - git+https://github.com/Toblerity/shapely.git@master#egg=shapely-1.7.1dev # temporary, 1.8dev required for numpy array support

requirements.txt

@@ -1,26 +1,24 @@
-python>=3.6
 pip>=19.0.3
 affine>=2.3.0
-albumentations=0.4.3
-fiona>=1.8.13
+albumentations==0.4.3
+fiona>=1.7.13
 gdal>=3.0.2
-geopandas>=0.6.2
+geopandas>=0.7.0
 matplotlib>=3.1.2
 networkx>=2.4
 numpy>=1.17.3
-opencv=4.1.0.25
+opencv-python>=4.1
 pandas>=0.25.3
 pyproj>=2.1
-pytorch=1.3.1
-pyyaml=5.2
+torch>=1.3.1
+pyyaml==5.2
 rasterio>=1.0.23
-requests>=2.22.0
-rio-cogeo>=1.1.6
+requests==2.22.0
 rtree>=0.9.3
 scikit-image>=0.16.2
 scipy>=1.3.2
-shapely>=1.6.4
-tensorflow=1.13.1
-torchvision>=0.4.2
+git+git://github.com/toblerity/shapely.git@master#egg=shapely-1.7.1dev
+torchvision>=0.5.0
 tqdm>=4.40.0
 urllib3>=1.25.7
+tensorflow==1.13.1

setup.py

@@ -9,7 +9,7 @@
 def get_version():
     VERSIONFILE = os.path.join('solaris', '__init__.py')
     initfile_lines = open(VERSIONFILE, 'rt').readlines()
-    VSRE = r'^__version__ = [\"\']*([\d.]+)[\"\']'
+    VSRE = r'^__version__ = [\"\']*([\d\w.]+)[\"\']'
     for line in initfile_lines:
         mo = re.search(VSRE, line, re.M)
         if mo:
@@ -56,40 +56,43 @@ def check_output(cmd):
 
 
 on_rtd = os.environ.get('READTHEDOCS') == 'True'
-
 if on_rtd:
     inst_reqs = ['sphinx_bootstrap_theme']
 else:
     inst_reqs = ['pip>=19.0.3',
                  'affine>=2.3.0',
                  'albumentations==0.4.3',
-                 'fiona>=1.8.13',
-                 'gdal>=3.0.2',
-                 'geopandas>=0.6.2',
+                 'fiona>=1.7.13',
+                 'gdal>=3.0.2'
+                 'geopandas>=0.7.0',
                  'matplotlib>=3.1.2',
                  'networkx>=2.4',
                  'numpy>=1.17.3',
-                 'opencv-python>=4.1.0.25',
+                 'opencv-python>=4.1',
                  'pandas>=0.25.3',
                  'pyproj>=2.1',
                  'torch>=1.3.1',
                  'pyyaml==5.2',
                  'rasterio>=1.0.23',
-                 'requests>=2.22.0',
-                 'rio-cogeo>=1.1.6',
+                 'requests==2.22.0',
                  'rtree>=0.9.3',
                  'scikit-image>=0.16.2',
                  'scipy>=1.3.2',
-                 'shapely>=1.6.4',
-                 'tensorflow==1.13.1',
+                 'shapely>=1.7.1dev',
                  'torchvision>=0.5.0',
                  'tqdm>=4.40.0',
-                 'urllib3>=1.25.7']
+                 'urllib3>=1.25.7',
+                 'tensorflow==1.13.1'
+                 ]
 
 
 extra_reqs = {
     'test': ['mock', 'pytest', 'pytest-cov', 'codecov']}
 
+# workaround until new shapely release is out
+os.system('pip install  git+git://github.com/toblerity/shapely@master')
+
+
 project_name = 'solaris'
 setup(name='solaris',
       version=get_version(),
@@ -109,6 +112,7 @@ def check_output(cmd):
       include_package_data=True,
       install_requires=inst_reqs,
       extras_require=extra_reqs,
+      dependency_links=['https://github.com/toblerity/shapely/tarball/master#egg=shapely-1.7.1dev'],
       entry_points={'console_scripts': [
           'geotransform_footprints = solaris.bin.geotransform_footprints:main',
           'make_graphs = solaris.bin.make_graphs:main',

solaris/__init__.py

@@ -1,3 +1,3 @@
 from . import bin, data, eval, nets, raster, tile, utils, vector
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"

solaris/data/coco.py

@@ -1,7 +1,7 @@
 from ..utils.core import _check_df_load, _check_geom, get_files_recursively
-from ..utils.geo import bbox_corners_to_coco, polygon_to_coco
+from ..utils.geo import bbox_corners_to_coco, polygon_to_coco, split_multi_geometries
 from ..utils.log import _get_logging_level
-from ..vector.polygon import geojson_to_px_gdf
+from ..vector.polygon import geojson_to_px_gdf, remove_multipolygons
 import numpy as np
 import rasterio
 from tqdm import tqdm
@@ -15,7 +15,9 @@
 def geojson2coco(image_src, label_src, output_path=None, image_ext='.tif',
                  matching_re=None, category_attribute=None, score_attribute=None,
                  preset_categories=None, include_other=True, info_dict=None,
-                 license_dict=None, recursive=False, verbose=0):
+                 license_dict=None, recursive=False, override_crs=False,
+                 explode_all_multipolygons=False, remove_all_multipolygons=False, 
+                 verbose=0):
     """Generate COCO-formatted labels from one or multiple geojsons and images.
 
     This function ingests optionally georegistered polygon labels in geojson
@@ -108,6 +110,13 @@ def geojson2coco(image_src, label_src, output_path=None, image_ext='.tif',
         If `image_src` and/or `label_src` are directories, setting this flag
         to ``True`` will induce solaris to descend into subdirectories to find
         files. By default, solaris does not traverse the directory tree.
+    explode_all_multipolygons : bool, optional
+        Explode the multipolygons into individual geometries using sol.utils.geo.split_multi_geometries. 
+        Be sure to inspect which geometries are multigeometries, each individual geometries within these 
+        may represent artifacts rather than true labels.
+    remove_all_multipolygons : bool, optional
+        Filters MultiPolygons and GeometryCollections out of each tile geodataframe. Alternatively you 
+        can edit each polygon manually to be a polygon before converting to COCO format.
     verbose : int, optional
         Verbose text output. By default, none is provided; if ``True`` or
         ``1``, information-level outputs are provided; if ``2``, extremely
@@ -191,6 +200,14 @@ def geojson2coco(image_src, label_src, output_path=None, image_ext='.tif',
     for gj in tqdm(label_list):
         logger.debug('Reading in {}'.format(gj))
         curr_gdf = gpd.read_file(gj)
+
+        if remove_all_multipolygons is True and explode_all_multipolygons is True:
+            raise ValueError("Only one of remove_all_multipolygons or explode_all_multipolygons can be set to True.")
+        if remove_all_multipolygons is True and explode_all_multipolygons is False:
+            curr_gdf = remove_multipolygons(curr_gdf)
+        elif explode_all_multipolygons is True:
+            curr_gdf = split_multi_geometries(curr_gdf)
+
         curr_gdf['label_fname'] = gj
         curr_gdf['image_fname'] = ''
         curr_gdf['image_id'] = np.nan
@@ -207,6 +224,7 @@ def geojson2coco(image_src, label_src, output_path=None, image_ext='.tif',
             if len(curr_gdf) > 0:  # if there are geoms, reproj to px coords
                 curr_gdf = geojson_to_px_gdf(
                     curr_gdf,
+                    override_crs=override_crs,
                     im_path=match_df.loc[match_df['label_fname'] == gj,
                                          'image_fname'].values[0])
                 curr_gdf['image_id'] = image_ref[match_df.loc[
@@ -221,6 +239,7 @@ def geojson2coco(image_src, label_src, output_path=None, image_ext='.tif',
             logger.debug('Converting to pixel coordinates.')
             # match the two images
             curr_gdf = geojson_to_px_gdf(curr_gdf,
+                                         override_crs=override_crs,
                                          im_path=list(image_ref.keys())[0])
             curr_gdf['image_id'] = list(image_ref.values())[0]
         curr_gdf = curr_gdf.rename(
@@ -403,6 +422,7 @@ def df_to_coco_annos(df, output_path=None, geom_col='geometry',
     def _row_to_coco(row, geom_col, category_id_col, image_id_col, score_col):
         "get a single annotation record from a row of temp_df."
         if score_col is None:
+
             return {'id': row['annotation_id'],
                     'image_id': int(row[image_id_col]),
                     'category_id': int(row[category_id_col]),

solaris/data/restrict_aoi_test.geojson

@@ -0,0 +1,8 @@
+{
+"type": "FeatureCollection",
+"name": "restrict_aoi_test",
+"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
+"features": [
+{ "type": "Feature", "properties": { "id": null }, "geometry": { "type": "MultiPolygon", "coordinates": [ [ [ [ -101.122733978747405, 41.737200029527671 ], [ -101.064449359873706, 41.767397042817478 ], [ -101.002339286354697, 41.751349690809022 ], [ -101.047567608320904, 41.677423309487807 ], [ -101.096527512847558, 41.649695858211281 ], [ -101.125247881319964, 41.674698280229421 ], [ -101.126670019207381, 41.707062031182794 ], [ -101.122733978747405, 41.737200029527671 ] ] ] ] } }
+]
+}

solaris/data/vectortile_test_nonfilled_expected/geoms_-101.003_41.774.geojson

@@ -0,0 +1 @@
+{"type": "FeatureCollection", "crs": {"type": "name", "properties": {"name": "urn:ogc:def:crs:EPSG:4326"}}, "features": []}

solaris/data/vectortile_test_nonfilled_expected/geoms_-101.044_41.691.geojson

@@ -0,0 +1 @@
+{"type": "FeatureCollection", "crs": {"type": "name", "properties": {"name": "urn:ogc:def:crs:EPSG:4326"}}, "features": []}

solaris/data/vectortile_test_nonfilled_expected/geoms_-101.044_41.732.geojson

@@ -0,0 +1 @@
+{"type": "FeatureCollection", "crs": {"type": "name", "properties": {"name": "urn:ogc:def:crs:EPSG:4326"}}, "features": []}