Skip to content

Commit

Permalink
Prepare for v0.15.0 Release (#84)
Browse files Browse the repository at this point in the history
* Create CI.yml

Complete unittests whenever any pull request is created

* IT WORKS

This wont work

This wont work

This wont work

This wont work

This wont work

This wont work

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Oh hellfire

Please lord work

Please lord work

Testing9

Testing8

Testing7

Testing6

Testing5

Testing4

Remove Pytest

Testing3

Testing2

Testing1

Fix indentation

Add push check

Add push check

Test Linux compatible

* Don't save figure, just shows it

* Change filepath

* Add data folder

* Fix layer tests

* Fix tests

* Change working dir

* Change working dir

* Change working dir

* Change working dir

* Please god work

* Please god work x2

* Please god work x3

* Please god work x4

* Please god work x5

* Please god work x6

* Please god work x7

* Change workflow name

* Test CI

* Trigger CI on push

* Join paths properly

* Typing is hard

* Check file exists

* fiona supported drivers

* Spit out GIS package versions

* LPT: Only import packages that exist

* Remove unused tests

* Include lfs

* Fix lfs

* create folders for save files

* Assert array almost equal within 1e-14

* Import correct numpy testing

* Checking for proper test correction

* The names Intergration, Continuous Intergration.

* The names Intergration, Continuous Intergration.

Checking for proper test correction

Import correct numpy testing

Assert array almost equal within 1e-14

create folders for save files

Fix lfs

Include lfs

Remove unused tests

* Remove commented code

* Change name of risk threshold input field

* Change resolution to 30m as standard

* Change plot_server res

* Change algo input regex

* Add tests for filepath references (#64)

* Add tests for filepath references

* Check for platform in filepath tests

* Fix graph regen (#66)

* Do not verify SSL (temporary fix)

* Disable InsecureRequestWarning console printout

* Change default aircraft base FatalityLayer

* Change if loop to a function

* Ask user to regen plot upon errors

* Add third API URL

* Add all API global instances

* Add TODO to remind about SSL certs

* Change GET req for POST req

* Change headers

* Header now connected to installed version

* Feature allow risk threshold input (#67)

* Change name of risk threshold input field

* Change resolution to 30m as standard

* Change plot_server res

* Change algo input regex

* Correctly input the threshold value

* Change README to update and correct grammar (#71)

* Add resolution to distance calculations

* Input raster grid and indices

* Correct path distance now calculated

* Use linspace to calculate x values

* Remove unnecessary code

* Add Postcode Dialog Box

* Change button selections to allow for map or post

* Call Postcode dialog when button clicked

* Use QInputDialog instead of own QWidget

* Clean and remove unnecessary code

* Add postcode API to requirements

* Change dialog title when invalid postcode is inputted

* Add unittest to check api

* Change unittest to only check for what we require

* Remove unneeded code

* Add in the ability to cancel the input

* Revert "feature_postcode_locations" (#74)

* Feature postcode selector (#75)

* Add Postcode Dialog Box

* Change button selections to allow for map or post

* Call Postcode dialog when button clicked

* Use QInputDialog instead of own QWidget

* Clean and remove unnecessary code

* Add postcode API to requirements

* Change dialog title when invalid postcode is inputted

* Add unittest to check api

* Change unittest to only check for what we require

* Remove unneeded code

* Add in the ability to cancel the input

* Break invalid postcode loop

* Add dummy aircraft (#79)

* Add dummy aircraft

* Change the point the dummy variable is added in

* Fix menu selection (#83)

* Account for dummy variable addition

* REDO THE TEST

* Risk Map Match Feature  (#77)

* Insert aircraft parameters into fatality risk layer

* Aircraft and wind now dhown in layer label

* Aircraft now have name built into the dictionary

* Replaces Fatality Risk layer with that of the Pathfinding layer's aircraft

* Change initial Fatality Risk Layer name

* Remove all old layers and add new ones

* Add TODO

* Loop the data layers when annotating layers

* Remove duplication of layers

* Only plot top data layer

* Path analysis popups now show the correct layer

* Remove test cases

* Remove serial res.append code

* Remove unnecessary for loop

* Add in more granular reporting to the UI

* Add name to DAV

* Remove the ability to duplicate aircraft and inform user of their saved aircraft's name

* Spelling of the dummy

* Change to V0.15.0

Co-authored-by: Aliaksei Pilko <[email protected]>
Co-authored-by: Aliaksei Pilko <[email protected]>
  • Loading branch information
3 people authored Nov 28, 2021
1 parent caa56ff commit 0ec0bb1
Show file tree
Hide file tree
Showing 34 changed files with 518 additions and 337 deletions.
42 changes: 42 additions & 0 deletions .github/workflows/CI-linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: SEEDPOD Linux

on: [ push, pull_request ]

jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
with:
lfs: true
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
sudo apt-get install libproj-dev proj-data proj-bin
sudo apt-get install libgeos-dev
sudo apt-add-repository ppa:ubuntugis/ubuntugis-unstable
sudo apt-get update
sudo apt-get install gdal-bin libgdal-dev
pip install wheel GDAL==3.2.3
pip install flake8 pytest Cython numpy pyproj pygeos
if [ -f requirements-linux.txt ]; then pip install -r requirements-linux.txt; fi
mkdir "tests/layers/figs"
mkdir "tests/layers/tiffs"
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
pytest
39 changes: 39 additions & 0 deletions .github/workflows/CI-windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: SEEDPOD Windows

on: [ push, pull_request ]

jobs:
build:

runs-on: windows-latest

steps:
- uses: actions/checkout@v2
with:
lfs: true
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install wheel
pip install -r requirements.txt
pip wheel --wheel-dir=extern -r requirements.txt
pip install flake8
pip install pytest
mkdir "tests/layers/figs"
mkdir "tests/layers/tiffs"
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
pytest
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,14 @@
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.4776529.svg)](https://doi.org/10.5281/zenodo.4776529)

Proof of Concept demonstrating modelling of UAS ground risk from open source data sources. This constructs a
spatiotemporal population density map and evaluates the risk posed by a parameterised UAS.
spatiotemporal population density map and evaluates the risk posed by a parameterised UAS. This risk map can then be
used by pathfinding algorithms to determine routes that have a fixed risk threshold or minimize flight risk completely.

All data processing is performed locally, so performance depends on your machine spec. At least 8GiB of RAM is essential
however.

Developed as part of the [SEEDPOD project](https://cascadeuav.com/seedpod/) funded by the E-Drone project grant
grant ([EP/V002619/1)](https://gow.epsrc.ukri.org/NGBOViewGrant.aspx?GrantRef=EP/V002619/1).
([EP/V002619/1)](https://gow.epsrc.ukri.org/NGBOViewGrant.aspx?GrantRef=EP/V002619/1).

## Disclaimer

Expand Down
2 changes: 1 addition & 1 deletion make_installer.iss
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
; SEE THE DOCUMENTATION FOR DETAILS ON CREATING INNO SETUP SCRIPT FILES!

#define MyAppName "SEEDPOD Ground Risk"
#define MyAppVersion "0.14.0"
#define MyAppVersion "0.15.0"
#define MyAppPublisher "CASCADE UAV"
#define MyAppURL "https://cascadeuav.com/seedpod/"
#define MyAppExeName "SEEDPOD Ground Risk.exe"
Expand Down
97 changes: 97 additions & 0 deletions requirements-linux.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# . # Set in setup.py

altgraph==0.17
attrs==20.3.0
bokeh==2.3.1
Cartopy==0.19.0.post1
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
click-plugins==1.1.1
cligj==0.7.1
cloudpickle==1.6.0
colorama==0.4.4
colorcet==2.0.6
commonmark==0.9.1
cycler==0.10.0
dask==2021.4.0
datashader==0.12.1
datashape==0.5.2
defusedxml==0.7.1
dill==0.3.3
distributed==2021.4.0
fastparquet==0.6.3
Fiona==1.8.19
fsspec==2021.4.0
future==0.18.2
geopandas==0.9.0
geoviews==1.9.1
HeapDict==1.0.1
holoviews==1.14.3
import-profiler==0.0.3
Jinja2==2.11.3
kiwisolver==1.3.1
llvmlite==0.36.0
locket==0.2.1
Markdown==3.3.4
MarkupSafe==1.1.1
matplotlib==3.4.1
msgpack==1.0.2
multipledispatch==0.6.0
multiprocess==0.70.11.1
munch==2.5.0
numba==0.53.1
numpy==1.20.2
odfpy==1.4.1
packaging==20.9
pandas==1.2.4
panel==0.11.3
param==1.10.1
partd==1.2.0
pathos==0.2.7
pefile==2019.4.18
Pillow==8.2.0
postcodes-io-api==0.0.4
pox==0.2.9
ppft==1.6.6.3
psutil==5.8.0
pyarrow==3.0.0
pyct==0.4.8
pygeos==0.8 # Keep at 0.8 for geopandas to play nice
Pygments==2.8.1
pyinstaller
pyinstaller-hooks-contrib==2021.1
pyparsing==2.4.7
pyproj==3.0.1
pyshp==2.1.3
PySide2==5.15.2
python-dateutil==2.8.1
pytz==2021.1
pyviz-comms==2.0.1
pywin32-ctypes==0.2.0
PyYAML==5.4.1
rasterio==1.2.3
requests==2.25.1
retrying==1.3.3
rich==10.1.0
Rtree==0.9.7
scipy==1.6.2
Shapely==1.7.1
shiboken2==5.15.2
six==1.15.0
sortedcontainers==2.3.0
spatialpandas==0.3.6
tabulate==0.8.9
tblib==1.7.0
thrift==0.13.0
toolz==0.11.1
topojson==1.0
tornado==6.1
tqdm==4.60.0
typing-extensions==3.7.4.3
urllib3==1.26.4
xarray==0.17.0
zict==2.0.0
casex~=1.0.5
scikit-learn~=0.24.1
scikit-image~=0.18.1
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ partd==1.2.0
pathos==0.2.7
pefile==2019.4.18
Pillow==8.2.0
postcodes-io-api==0.0.4
pox==0.2.9
ppft==1.6.6.3
psutil==5.8.0
Expand Down
75 changes: 52 additions & 23 deletions seedpod_ground_risk/core/plot_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ class PlotServer:
def __init__(self, tiles: str = 'Wikipedia', tools: Optional[Iterable[str]] = None,
active_tools: Optional[Iterable[str]] = None,
cmap: str = 'CET_L18',
raster_resolution: float = 60,
raster_resolution: float = 40,
plot_size: Tuple[int, int] = (760, 735),
progress_callback: Optional[Callable[[str], None]] = None,
update_callback: Optional[Callable[[str], None]] = None,
Expand Down Expand Up @@ -145,6 +145,7 @@ def plot(self, doc):
hvPlot = self.compose_overlay_plot(self._x_range, self._y_range)
if self._preload_complete:
self._progress_bar_callback(100)
self._progress_callback("Plotting complete")
fig = hv.render(hvPlot, backend='bokeh')
fig.output_backend = 'webgl'

Expand Down Expand Up @@ -197,27 +198,43 @@ def compose_overlay_plot(self, x_range: Optional[Sequence[float]] = (-1.6, -1.2)

t0 = time()
self._progress_bar_callback(10)
# TODO: This will give multiple data layers, these need to be able to fed into their relevent pathfinding layers
for annlayer in self.annotation_layers:
new_layer = FatalityRiskLayer('Fatality Risk', ac=annlayer.aircraft['name'])
self.add_layer(new_layer)
self.remove_duplicate_layers()
self._progress_bar_callback(20)
self.generate_layers(bounds_poly, raster_shape)
self._progress_bar_callback(50)
plot = Overlay([res[0] for res in self._generated_data_layers.values()])
plt_lyr = list(self._generated_data_layers)[0]
plot = Overlay([self._generated_data_layers[plt_lyr][0]])
print("Generated all layers in ", time() - t0)
if self.annotation_layers:
plot = Overlay([res[0] for res in self._generated_data_layers.values()])
raw_datas = [res[2] for res in self._generated_data_layers.values()]
raster_indices = dict(Longitude=np.linspace(x_range[0], x_range[1], num=raster_shape[0]),
Latitude=np.linspace(y_range[0], y_range[1], num=raster_shape[1]))
raster_grid = np.sum(
[remove_raster_nans(res[1]) for res in self._generated_data_layers.values() if
res[1] is not None],
axis=0)
raster_grid = np.flipud(raster_grid)
raster_indices['Latitude'] = np.flip(raster_indices['Latitude'])

self._progress_callback('Annotating Layers...')
res = jl.Parallel(n_jobs=1, verbose=1, backend='threading')(
jl.delayed(layer.annotate)(raw_datas, (raster_indices, raster_grid)) for layer in
self.annotation_layers)

plot = Overlay([self._generated_data_layers[plt_lyr][0]])
res = []
prog_bar = 50
for dlayer in self.data_layers:
raster_indices = dict(Longitude=np.linspace(x_range[0], x_range[1], num=raster_shape[0]),
Latitude=np.linspace(y_range[0], y_range[1], num=raster_shape[1]))
raw_data = [self._generated_data_layers[dlayer.key][2]]
raster_grid = np.sum(
[remove_raster_nans(self._generated_data_layers[dlayer.key][1])],
axis=0)
raster_grid = np.flipud(raster_grid)
raster_indices['Latitude'] = np.flip(raster_indices['Latitude'])

for alayer in self.annotation_layers:
if alayer.aircraft == dlayer.ac_dict:
self._progress_bar_callback(prog_bar)
prog_bar += 40 / len(self.annotation_layers)
self._progress_callback(f'Finding a path for {alayer.aircraft["name"]}')
res.append(alayer.annotate(raw_data, (raster_indices, raster_grid)))

self._progress_callback('Plotting paths')
self._progress_bar_callback(90)
# res = jl.Parallel(n_jobs=1, verbose=1, backend='threading')(
# jl.delayed(layer.annotate)(raw_datas, (raster_indices, raster_grid)) for layer in
# self.annotation_layers )
plot = Overlay(
[self._base_tiles, plot, *[annot for annot in res if annot is not None]]).collate()
else:
Expand All @@ -239,6 +256,8 @@ def compose_overlay_plot(self, x_range: Optional[Sequence[float]] = (-1.6, -1.2)
# Just display map tiles in case this was transient
import traceback
traceback.print_exc()
self._progress_callback(
f'Plotting failed with the following error: {e}. Please attempt to re-generate the plot')
print(e)
plot = self._base_tiles

Expand Down Expand Up @@ -292,6 +311,8 @@ def generate_layer(layer: DataLayer, bounds_poly: sg.Polygon, raster_shape: Tupl
Tuple[str, Tuple[Geometry, np.ndarray, gpd.GeoDataFrame]], Tuple[str, None]]:

try:
if isinstance(layer, FatalityRiskLayer):
layer.key = f'{layer.key} {layer.ac} {layer.wind_dir:03d}@{layer.wind_vel}kts'
result = layer.key, layer.generate(bounds_poly, raster_shape, from_cache=False, hour=hour,
resolution=resolution)
return result
Expand Down Expand Up @@ -334,15 +355,23 @@ def generate_path_data_popup(self, layer):
from seedpod_ground_risk.pathfinding.environment import GridEnvironment
from seedpod_ground_risk.ui_resources.info_popups import DataWindow
from seedpod_ground_risk.layers.fatality_risk_layer import FatalityRiskLayer
for i in self.data_layers:
if isinstance(i, FatalityRiskLayer):
path = layer.path
cur_layer = GridEnvironment(self._generated_data_layers['Fatality Risk'][1])
for dlayer in self.data_layers:
if isinstance(dlayer, FatalityRiskLayer) and layer.aircraft == dlayer.ac_dict:
cur_layer = GridEnvironment(self._generated_data_layers[dlayer.key][1])
grid = cur_layer.grid
popup = DataWindow(path, grid)
popup = DataWindow(layer, grid)
popup.exec()
break

def remove_duplicate_layers(self):
# TODO Make the list/set method work as the nested for loop is clunky
# self.data_layers = list(set(self.data_layers))

for i, layer1 in enumerate(self.data_layers):
for j, layer2 in enumerate(self.data_layers):
if layer1.ac_dict == layer2.ac_dict and i != j:
self.remove_layer(layer2)

def _get_raster_dimensions(self, bounds_poly: sg.Polygon, raster_resolution_m: float) -> Tuple[int, int]:
"""
Return a the (x,y) shape of a raster grid given its EPSG4326 envelope and desired raster resolution
Expand Down
14 changes: 7 additions & 7 deletions seedpod_ground_risk/data/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,35 @@


def aircraft_list_filepath():
aircraft_list_fp = os.sep.join(("static_data", "aircraft_list.json"))
aircraft_list_fp = os.path.join("static_data", "aircraft_list.json")
return aircraft_list_fp


def england_wa_2011_clipped_filepath():
england_wa_2011_clipped_fp = os.sep.join(('static_data', 'england_wa_2011_clipped.shp'))
england_wa_2011_clipped_fp = os.path.join('static_data', 'england_wa_2011_clipped.shp')
return england_wa_2011_clipped_fp


def nhaps_data_filepath():
nhaps_data_fp = os.sep.join(('static_data', 'nhaps.json'))
nhaps_data_fp = os.path.join('static_data', 'nhaps.json')
return nhaps_data_fp


def density_filepath():
density_fp = os.sep.join(('static_data', 'density.csv'))
density_fp = os.path.join('static_data', 'density.csv')
return density_fp


def traffic_count_filepath():
traffic_count_fp = os.sep.join(('static_data', 'dft_traffic_counts_aadf.csv'))
traffic_count_fp = os.path.join('static_data', 'dft_traffic_counts_aadf.csv')
return traffic_count_fp


def road_geometry_filepath():
road_geometry_fp = os.sep.join(('static_data', '2018-MRDB-minimal.shp'))
road_geometry_fp = os.path.join('static_data', '2018-MRDB-minimal.shp')
return road_geometry_fp


def relative_variation_filepath():
relative_variation_fp = os.sep.join(('static_data', 'tra0307.ods'))
relative_variation_fp = os.path.join('static_data', 'tra0307.ods')
return relative_variation_fp
Loading

0 comments on commit 0ec0bb1

Please sign in to comment.