Os to pathlib and helpers.py method from pypsa-earth-sec

[finozzifa]

Changes proposed in this Pull Request

In this pull request I have:

• replaced the os module with the pathlib module
• added to scripts/_helpers.py the methods from the helpers.py script of pypsa-earth-sec
• added a unit-test execution step in ci-linux.yaml, ci-mac.yaml, ci-windows.yaml
• moved the method load_network_for_plots from _helpers.py to plot_network.py. The method is in fact used just in plot_network.py. Leaving it in _helpers.py created a circular dependency, as add_electricity.py imports _helpers.py, which then imported the add_electricity.py methods load_costs and update_transmission_costs in load_network_for_plots

os to pathlib

The os module has been replaced with the pathlib module. There are still a few os-module-calls in build_natura_raster.py (this is because there is no pathlib alternative for os.walk) and in the _helpers.py

The migration has been carried out together with the abstraction of the path methods. Such methods are available in the _helpers.py script. In fact, whenever I see lines of code that repeat themselfs several times, I try to create methods that implement such lines of code and that can be separately unit-tested. In this was I can ensure consistency and make future changes easier. Instead of tracing throughout the code each invocation of (say) os.path.basename(), I just need to modify once the method where such line is implemented. I believe that in this way, code reviews, code merges and further code expansions will be easier, as we can rely on soundproof methods that we have unit-tested.

Addition of the methods from the helpers.py script of pypsa-earth-sec

• two_digits_2_name_country : I add a new function argument (called name_string) to specify whether we wish to return the full name of the country or its full name. For example when name_string=name_short CD -> DR Congo, whereas when name_string=name_official CD -> Democratic Republic of the Congo
• prepare_costs: I add this method from the helpers.py scripts of pypsa-earth-sec
• create_network_topology: I add this method from the helpers.py script of pypsa-earth-sec
• cycling_shift: I add this method from the helpers.py script of pypsa-earth-sec
• download_gadm: I add this method from the helpers.py script of pypsa-earth-sec
• get_gadm_layer: I add this method from the helpers.py script of pypsa-earth-sec
• locate_bus: I add this method from the helpers.py script of pypsa-earth-sec
• safe_divide: I add this method from the helpers.py script of pypsa-earth-sec

I did not add the following methods as they are not used in pypsa-earth-sec

• create_dummy_data(n, sector, carriers): I did not copy this method from pypsa-earth-sec/scripts/helpers.py because it is not used in pypsa-earth-sec
• get_country(target, **keys): this method from pypsa-earth-sec/scripts/helpers.py is not needed

Unit tests

I have added dedicated unit tests for the path methods and the methods coming from pypsa-earth-sec.

I have also added the execution of such unit tests to ci-linux.yaml, ci-mac.yaml, ci-windows.yaml, making the unit test execution part of the CI/CD.

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[pz-max]

Some comments and questions

[pz-max]

Why commented?

[finozzifa]

Hi @pz-max
thanks for your comment. In order to execute this unit test I need to modify the way the _helpers.py methods are invoked in the other files contained in the scripts folder.

Currently we have from _helpers import ....

I think we need to have from scripts._helpers import ....

I will attempt this change in another pull request.

[pz-max]

Why? Is this not already in PyPSA-Earth in build shapes? Why having it twice?

[finozzifa]

Hey @pz-max thanks for noticing it. Actually the two methods are kind of different in what they do even though they look similar and are called the same way.

I think that a dedicated PR is needed to fix it.

For the moment I will flag it as a TODO and change the same of the method in _helpers.py

[pz-max]

Same as before? (It's in build shapes?)

[finozzifa]

Hey @pz-max , ja same comment as before. I would like to have a dedicated pull request for address this "duplication". Most likely I need to properly unit test both version in order to merge them accordingly.

[finozzifa]

Hi all,

as an extra check I have compared the values of the objective function we got from CI-linux/ ubuntu-latest for this pull request and ones obtained for CI-linux/ ubuntu-latest which corresponds to the last CI/CD exeuction for pypsa-meets-earth/pypsa-earth.

In the attached files, I will refer to:

• New values: the objective function values obtained from CI-linux/ ubuntu-latest for this pull request
• Reference values: the objective function values obtained from CI-linux/ ubuntu-latest of pypsa-meets-earth/pypsa-earth

obj_func_new_values.txt
obj_func_reference_values.txt

As you can see, the percentage differences between the two executions are below 1%.