Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make data retrieval more robust #1361

Open
2 tasks
ekatef opened this issue Feb 28, 2025 · 2 comments · May be fixed by #1366
Open
2 tasks

Make data retrieval more robust #1361

ekatef opened this issue Feb 28, 2025 · 2 comments · May be fixed by #1366

Comments

@ekatef
Copy link
Member

ekatef commented Feb 28, 2025

Describe the feature you'd like to see

There have been some recent changes in google drive policy which have lead to the rule retrieve_databundle_light being crashed more often. We need to revise it to improve stability and overall user experience. A revised design may look like following.

  • Revise structure of retrieve_databundle_light to make it modular and allow to fail for download of any particular data bundle, while storing outputs of the successful downloads
  • Provide a summary to the user on which files need manual interventions
@ekatef
Copy link
Member Author

ekatef commented Feb 28, 2025

The inspiration can be taken from run_all_scenarios which allows for safe failures while executing a set of scenarios and also provides an overview of the steps which have been successful for every scenario including failed ones

@ekatef
Copy link
Member Author

ekatef commented Feb 28, 2025

To have a complete picture, copy-pasting the current workaround (many thanks @davide-f for providing a detailed description in this Discord thread):

A temporary workaround is to locally execute the retrievedatabundle rule with python and download the bundle manually while being logged with your gdrive account. The gdrive limitstion is more strict when downloads are from unknown accounts as in the typical workflow run
To do so you can:

conda activate pypsa-earth
cd pypsa-earth/scripts
python retrieve_databundlelight.py

Here the files are being downloaded and keep notice of the bundles that are not downloaded; most likely the cutout only.

Then go into your config.yaml and ensure to switch: enable_retrieve_databundlelight: false. Then now you can manually download the missing bundles from gdrive. You can find the links into the file configs/bunlde_config.yaml.

As you download your desired bundle, you can place it in the respective folder e.g. pypsa-earth/cutouts.

@virio-andreyana virio-andreyana linked a pull request Mar 4, 2025 that will close this issue
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant