-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ETL Pipe for download and ingest EEZ [MARXAN-1618] #1121
base: develop
Are you sure you want to change the base?
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Co-authored-by: andrea rota <[email protected]>
…ity/marxan-cloud into feature/data/MARXAN-1618_ETL_EEZ
curl --location --request POST 'https://www.marineregions.org/download_file.php?name=World_EEZ_v11_20191118.zip' \ | ||
--header 'Cookie: PHPSESSID=870e305efbc0519d59b361427dbd8336; vliz_webc=vliz_webc1' \ | ||
--form 'name="Jen"' \ | ||
--form 'organisation="TNC"' \ | ||
--form 'email="[email protected]"' \ | ||
--form 'country="EEUU"' \ | ||
--form 'user_category="academia"' \ | ||
--form 'purpose_category="Conservation"' \ | ||
--form 'agree="1"' \ | ||
--output './World_EEZ_v11_20191118.zip' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry I had missed parts of this PR while I reviewed it - probably my tab wasn't refreshed when you pushed later changes... this bit, I recommend to handle it differently, without hardcoding cookie and most of the form data such as name, org, email, country, user_category, purpose_category
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how to do this... my first thought would be to move the whole curl command to a simple shell script that is committed to the repo with placeholders, and needs to be copied to a specific file (that should be added to .gitignore
, similarly to env.default
vs .env
), mounted in the eez ingestion container, and executed in the eez Makefile
@aagm for the failing tests (https://github.com/Vizzuality/marxan-cloud/runs/6691428000?check_suite_focus=true for example), IIRC each migration file is wrapped in a transaction so we may need two migrations in order for the enum to be fully committed by the time it's then used. I don't think (again IIRC, and IIRC the docs are thin on this - I'd need to read the source code...) that using two distinct Or avoiding touching that enum altogether and going the separate table route 😜 |
Overview
This pr adds the makefile pipe to manage EEZ marine boundaries.
Designs
Link to the related design prototypes (if applicable)
Testing instructions
Either make a full db restoration:
make clean-slate
make start-api
make seed-api-init-data
make seed-geodb-data
Or run just for Testing the individual pipe:
docker-compose -p marxan-cloud -f ./data/docker-compose-data_management.yml up --no-start --build marxan-seed-data marxan-seed-data && docker-compose -p marxan-cloud -f ./data/docker-compose-data_management.yml run marxan-seed-data make seed-eez
Checklist before submitting
develop
.deploying to staging/production, please add brief testing instructions
to the deploy checklist (
docs/deployment-checklist.md
)