Skip to content

Latest commit

 

History

History
75 lines (56 loc) · 3.91 KB

README.md

File metadata and controls

75 lines (56 loc) · 3.91 KB
Raw

IBF-river-flood-pipeline

Forecast riverine flooding. Part of IBF-system.

Description

The pipeline roughly consists of three steps:

  • Extract data on river discharge from an external provider, both in specific locations (stations) and over pre-defined areas (administrative divisions).
  • Forecast floods by determining if the river discharge is higher than pre-defined thresholds; if so, calculate flood extent and impact (affected people).
  • Send this data to the IBF app.

The pipeline stores data in:

  • ibf-cosmos (Azure Cosmos DB): river discharge per station / administrative division, flood forecasts, and trigger thresholds
  • 510ibfsystem (Azure Storage Account): raw data from GloFAS and other geospatial data

The pipeline depends on the following services:

For more information, see the functional architecture diagram.

Basic Usage

To run the pipeline locally

  1. fill in the secrets in .env.example and rename the file to .env; in this way, they will be loaded as environment variables
  2. install requirements
pip install poetry
poetry install --no-interaction
  1. run the pipeline with python flood_pipeline.py
Usage: flood_pipeline.py [OPTIONS]

Options:
  --country TEXT  country ISO3
  --prepare       prepare discharge data
  --extract       extract discharge data
  --forecast      forecast floods
  --send          send to IBF app
  --save          save to storage
  --help          Show this message and exit.

Advanced Usage

How do I set up the pipeline for a new country?

  1. Check that the administrative boundaries are in the IBF system; if not, ask IBF developers to add them
  2. Add country-specific configuration in config/config.yaml
  3. Create historical flood extent maps
python data_updates\add_flood_maps.py --country <country ISO3>
  1. Compute trigger and alert thresholds
python data_updates\add_flood_thresholds.py --country <country ISO3>
  1. Update Glofas data pipeline in IBF-data-factory so that it will trigger a pipeline run for the new country

How do I insert an exception for a specific country?

You don't. The pipeline is designed to work in the same way for all countries. If you need to change the pipeline's behavior for a specific country, please discuss your needs with your fellow data specialist, they will try their best to accommodate your request.

There is a new version of GloFAS, how do I update the pipeline?

GloFAS should take care to update the river discharge data in a backward-compatible way. If that is not the case, you need to have a look at floodpipeline/extract.py and change what's needed.

What will probably change with the new GloFAS version are the trigger/alert thresholds. To update them you need to

  1. Change the