Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update dsra psra oct2021 (model-factory) #96

Merged
merged 140 commits into from
Jan 21, 2022
Merged

Update dsra psra oct2021 (model-factory) #96

merged 140 commits into from
Jan 21, 2022

Conversation

wkhchow
Copy link
Collaborator

@wkhchow wkhchow commented Jan 18, 2022 •
edited by anthonyfok
Loading

Fixes:

wkhchow added 30 commits October 5, 2021 11:50
@wkhchow
Update Create_scenario_risk_building_Indicators_ALL_shkmp.sql
4e69f8c 
- added placeholder 'MMI' indicator
- changed all_indicators to indicators
@wkhchow
Update Create_scenario_risk_sauid_Indicators_ALL_shkmp.sql
53e771e 
- added placeholder 'MMI' indicator
- changed all_indicators to indicators
- removed vs30lon, vs30lat
- fixed prefix sD_Collapse to sDt_Collapse
- added CSD aggregation level
@wkhchow
Update PSRA_combineSrcLossTable.py
c50b81a 
- change r2 to r1
@wkhchow
Update psra_4.Create_psra_sauid_all_indicators.sql
d50eb2e 
- changed all_indicators to indicators
- renamed r2 to r1 indicators
- added eq risk index calculations / indicators
- removed commented out sections of code
@wkhchow
Update PSRA_copyTables.py
f8fdcfb 
- rename r2 to r1
- removed/reordered stat, loss_value from agg_curves_stats
- added occupants to avg_loss_stats
- removed trt from src_loss
@wkhchow
Update psra_2.Create_table_updates.sql
b3cf9cb 
- removed commented out codes
- renamed r2 to r1
- add occupants, removed trt
@wkhchow
Update Create_all_tables_update.sql
674dee8 
- remove commented out lines
@wkhchow
Update Create_hexbin_MH_risk_sauid_prioritization_aggregation_area.sql
9c0739d 
- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_hexbin_physical_exposure_aggregation_area_proxy.sql
61540fc 
- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_hexbin_social_vulnerability_aggregation_area_proxy.sql
5c6e84e 
- changed all_indicators to indicators
@wkhchow
Update Create_MH_risk_sauid_prioritization_Canada.sql
ef7af38 
- changed all_indicators to indicators
@wkhchow
Update Create_physical_exposure_building_indicators_PhysicalExposure.sql
c288153 
- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_physical_exposure_building_indicators_PhysicalExposure_…
99d190a 
…ste.sql

- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_physical_exposure_sauid_indicators_view_PhysicalExposur…
4471360 
…e.sql

- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_physical_exposure_sauid_indicators_view_PhysicalExposur…
4241185 
…e_ste.sql

- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_physical_exposure_site_level_indicators_PhysicalExposur…
c0630dd 
…e_ste.sql

- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_risk_dynamics_indicators.sql
78dd6f4 
- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_scenario_risk_building_Indicators_ALL_ste_shkmp.sql
f16e878 
- remove commented out lines
- changed all_indicators to indicators
- remove sH_RupAbbr
@wkhchow
Update Create_scenario_risk_master_tables.sql
083fb58 
- remove commented out lines
- remove sh_rupabbr
@wkhchow
Update Create_scenario_risk_sauid_Indicators_ALL_ste_shkmp.sql
521f1d1 
- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_scenario_risk_site_level_Indicators_ALL_ste_shkmp.sql
fac7438 
- remove commented out lines
- changed all_indicators to indicators
@wkhchow
Update Create_site_exposure_to_building_and_sauid.sql
bc36bf7 
- remove commented out lines
@wkhchow
Update Create_social_vulnerability_sauid_indicators_SocialFabric.sql
8b320ca 
- changed all_indicators to indicators
- updated VHt_NonResHa
@wkhchow
Update psra_1.Create_tables.sql
55560c1 
- remove commented out lines
- changed r2 to r1
@wkhchow
Update psra_3.Create_psra_building_all_indicators.sql
b9545bb 
- remove commented out lines
- changed all_indicators to indicators
- changed r2 to r1
@wkhchow
Update psra_5.Create_psra_sauid_references_indicators.sql
d6d48f9 
- remove commented out lines
- changed all_indicators to indicators
- changed r2 to r1
@wkhchow
Update psra_6.Create_psra_merge_into_national_indicators.sql
e84aa91 
- comment out P/T not ready yet for initial testing
@wkhchow
Update Create_hexbin_physical_exposure_aggregation_area_proxy.sql
0c72b62 
- Et_RESHD to Et_ResHD
@wkhchow
Update psra_6.Create_psra_merge_into_national_indicators.sql
af327f6 
- change all_indicators to indicators
@wkhchow
Update psra_6.Create_psra_merge_into_national_indicators.sql
7014f47 
update on_uhs to pe_uhs
@wkhchow wkhchow added this to the Sprint 50 milestone Jan 18, 2022
@wkhchow wkhchow mentioned this pull request Jan 18, 2022
Merged
@wkhchow wkhchow changed the title Update dsra psra oct2021 Update dsra psra oct2021 (model-factory) Jan 18, 2022
This was linked to issues Jan 18, 2022
PSRA - add id field to psra reference layers that are missing #77
Closed
DSRA - Generate indicators to CSDUID level, check all available layers required in ES #78
Closed
Eliminate the use of 'all_indicators' #80
Closed
Add agg-curves q05 and q95 to PSRA_copyTables.py #82
Closed
DSRA - revise SA(0.3) >= 0.02 filter #85
Closed
investigate new changes for psra #97
Closed
jvanulde
jvanulde reviewed Jan 19, 2022
View reviewed changes
scripts/PSRA_copyTables_Canada.py
args = parse_args()

# Copy agg curves stats b0 table
with open("/usr/src/app/ebRisk/Canada/ebR_Canada_agg_curves-stats_b0.csv".format(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid system paths.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pass the path in as an argument?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably best.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe get the path from an environment variable, and with a default fallback if the environment variable is unset?
(Or maybe accept both command-line argument and environment variable?)

I took a brief look at it back in June 2021, and from what I recall, it is not just the Python scripts, but also some SQL files where the /usr/src/app path is hardcoded. See OpenDRR/opendrr-api#114, and the newly opened #98.

jvanulde
jvanulde approved these changes Jan 19, 2022
View reviewed changes
Copy link
Contributor

@jvanulde jvanulde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine to me. Eventually, it might be worth looking at ways to reduce the reliance on system paths for files if possible. I would like to to see the renovate.json configuration file removed and the requirements.txt updated to reflect current state. I ran some of the python scripts through Flake8 and there are unused imports in many files. Suggest that we enable Flake8 linter in VS Code.

@anthonyfok anthonyfok mentioned this pull request Jan 19, 2022
Open
@drotheram drotheram linked an issue Jan 21, 2022 that may be closed by this pull request
PSRA - add aggregation column for agg_curves_stats, agg_losses_stats, src_loss_table from filename #84
Closed
@anthonyfok anthonyfok mentioned this pull request Jan 21, 2022
Merged
anthonyfok
anthonyfok approved these changes Jan 21, 2022
View reviewed changes
Copy link
Member

@anthonyfok anthonyfok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wkhchow and @drotheram Wow! There are so many intricate details that it is nothing short of amazing how you meticulously keep track of them and make them work! Kudos!

@anthonyfok anthonyfok force-pushed the update_dsra_psra_oct2021 branch from 6f2ae3a to 6710c72 Compare January 21, 2022 21:00
@anthonyfok anthonyfok merged commit 6710c72 into master Jan 21, 2022
drotheram
drotheram reviewed Jan 21, 2022
View reviewed changes
Copy link
Contributor

@drotheram drotheram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Big changes. Everything looks good to me. Note, PSRA results for B.C. are still not included in this pipeline yet

@wkhchow wkhchow deleted the update_dsra_psra_oct2021 branch April 1, 2022 21:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jvanulde jvanulde jvanulde approved these changes

@drotheram drotheram drotheram left review comments

@anthonyfok anthonyfok anthonyfok approved these changes

@DamonU2 DamonU2 DamonU2 approved these changes

Assignees

@drotheram drotheram

@wkhchow wkhchow

Labels
None yet
Projects
None yet
Milestone
Sprint 50
5 participants
@wkhchow @anthonyfok @jvanulde @drotheram @DamonU2