Finalize duplication of retrieve_data

[davide-f]

Closes #1073 and #1074

Changes proposed in this Pull Request

This PR builds on top of #1120 to avoid duplication of retrieve_data function, bump technology data version and drop cost_dir option.

Note: dropping cost_dir is in line with pypsa-eur and we also had a TODO on that, but before merging this PR, there is the need for a check by @hazemakhalek , @Eddy-JV or @energyLS given its use in the -sec model.
Moreover, bumping the latest version of the default technology data to the latest will change values of the CI

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[ekatef]

Hello @davide-f, looks like a very nice improvement!

My major concern is that the PR also contains some improvements which relate to improvements for alternative clustering. It feels like there may be some connections with other issues and fixes, but the context is not quite clear from the PR. Given the fact that alternative clustering seems to be one of the major points to be tackled, I'd suggest to keep in this PR only cost-related improvements and open another dedicated PR for the improvements on alternative clustering. What do you think?

The costs-related part looks great and basically ready to merge, unless there are some comments from @hazemakhalek @Eddy-JV and @energyLS

[ekatef]

@davide-f thanks a lot for introducing the changes! Totally agree that tackling the hydro improvements deserves a dedicated issue. Thanks for taking care about that 🙂

I think this PR is ready to merge if there is no objections from the experts on the sector-coupled part 🚀 Great work! 😄

[energyLS]

@energyLS doublecheck

[davide-f]

Update. @ekatef given the recent alternative_clustering issue and the possible fix #1287
it could be advisable to consider restoring the proposal on alternative clustering in #1120 . Recent tests have identified that ror technology can be lost in #1287

[ekatef]

Looks very good! 😄

I remember that the PR combines changes related both to file management for costs and improvements for hydro calculations. To help disentangle it, may it be a good idea to add a duplicated release note on hydro along with the existing one?

[ekatef]

Have checked the objective, and it appears that the only significant change of ~8% relates to the second test case (the one with config.custom.yaml with alternative clustering on): 2157472538.0 in this PR vs 2341159535.0. Interestingly, that the objective remains exactly the same for myopic test case though it's also using alternative clustering. [I guess the reason may be fixing the hydro potential which could have been overestimated previously]

Also, there is ~4% change for one of the Monte-Carlo tests (the one with very low objective function), but I remember that we had it previously (looking on this comment) and it's normal.

So, the objective function didn't change for the test cases with Voronoi clustering and slightly increased for config.custom.yaml which had to be expected as previously the hydro potential has been overestimated. So, I think the results of the objective testing are satisfactory 😄