Revise export ports data

[GbotemiB]

Closes # (if applicable).

Changes proposed in this Pull Request

The changes in this PR include

• use port.csv data as input to add_export script
• preprocessing steps for ports.csv data in add_export script

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

Hello @GbotemiB :)
Thanks for the proposal to address this point.

Currently, the "export_ports" list is expected to be filled by the user; export_ports is indeed a file available in the data folder.
However, I agree that it is unconfortable to be filled manually and this can lead to issues especially when an export potential is requested.

The fact is that not all ports in ports.csv can actually be suitable for exports.
Ideally, the user shall propose either to use the custom value or a file calculated by the workflow.

I believe @hazemakhalek, @energyLS and/or @Eddy-JV may have some insights here.
Feel free to propose something by first writing in here, before coding.

Long-term, we may have prepare_ports to have 2 outputs: 1 ports.csv and 1 export.csv.
Moreover, if relevant for the -sec team, we also keep export.csv and have an option about whether using the export produced by ports or the custom file.

For the short term, we can have the option about either using ports.csv or export.csv

[GbotemiB]

Thanks @davide-f. I will await comments from the -sec team on the best suitable approach.

But running the US model, what do you recommend at the moment to solve the model?

[davide-f]

Thanks @davide-f. I will await comments from the -sec team on the best suitable approach.

But running the US model, what do you recommend at the moment to solve the model?

The simplest approach is copying the ports.csv file and rename it as export_ports.csv, or selecting a subset of such ports; not that it is likely that only large ports can actually be exporting ones so it makes sense to select a subset of them.
To make the model run, the procedure above should work in the meantime

[Eddy-JV]

Thank you @davide-f for already replying earlier and thank you @GbotemiB for rasing this.

First let me understand the context:

• Currently the export_ports.csv is kept in data because usually the user has in mind very specific export ports for Hyrdrogen export. That was actually always the case till now, that export countries have already selected certain hydrogen export ports. What I understand, is that you want the model to decide which export port to invest in randomly. Please correct me if am wrong.

I that is the case, then I agree with @davide-f . We should then allow the user to choose in the config file under custom_data as True or False:

1. Use custom data for export ports (True): which entales using export_ports.csv file that we place in data_custom instead of data folder.
2. Use wrokflow (False): Then we add another output for prepare_ports.py , where we select export ports to be only "Large" ports for example and then save it in resources folder.

What do you think?

[energyLS]

Thank you @davide-f for already replying earlier and thank you @GbotemiB for rasing this.

First let me understand the context:

* Currently the `export_ports.csv` is kept in data because usually the user has in mind very specific export ports for Hyrdrogen export. That was actually always the case till now, that export countries have already selected certain hydrogen export ports. What I understand, is that you want the model to decide which export port to invest in randomly. Please correct  me if am wrong.

I that is the case, then I agree with @davide-f . We should then allow the user to choose in the config file under custom_data as True or False:

1. **Use custom data for export ports (True):** which entales using `export_ports.csv` file that we place in `data_custom` instead of `data` folder.

2. **Use wrokflow (False):** Then we add another output for `prepare_ports.py` , where we select export ports to be only "Large" ports for example and then save it in `resources `folder.

What do you think?

@Eddy-JV I support this idea, seems reasonable to me.

[GbotemiB]

Thank you @davide-f for already replying earlier and thank you @GbotemiB for rasing this.

First let me understand the context:

• Currently the export_ports.csv is kept in data because usually the user has in mind very specific export ports for Hyrdrogen export. That was actually always the case till now, that export countries have already selected certain hydrogen export ports. What I understand, is that you want the model to decide which export port to invest in randomly. Please correct me if am wrong.

I that is the case, then I agree with @davide-f . We should then allow the user to choose in the config file under custom_data as True or False:

1. Use custom data for export ports (True): which entales using export_ports.csv file that we place in data_custom instead of data folder.
2. Use wrokflow (False): Then we add another output for prepare_ports.py , where we select export ports to be only "Large" ports for example and then save it in resources folder.

What do you think?

Hi @Eddy-JV, Thank you very much for this contribution. I have a clarification question. There is a possibility that not all countries have large ports. Incases like this, we can use the medium ports, and if there is no medium ports we can use small ports. This way every country is covered.

[Eddy-JV]

Thank you @davide-f for already replying earlier and thank you @GbotemiB for rasing this.
First let me understand the context:

• Currently the export_ports.csv is kept in data because usually the user has in mind very specific export ports for Hyrdrogen export. That was actually always the case till now, that export countries have already selected certain hydrogen export ports. What I understand, is that you want the model to decide which export port to invest in randomly. Please correct me if am wrong.

I that is the case, then I agree with @davide-f . We should then allow the user to choose in the config file under custom_data as True or False:

1. Use custom data for export ports (True): which entales using export_ports.csv file that we place in data_custom instead of data folder.
2. Use wrokflow (False): Then we add another output for prepare_ports.py , where we select export ports to be only "Large" ports for example and then save it in resources folder.

What do you think?

Hi @Eddy-JV, Thank you very much for this contribution. I have a clarification question. There is a possibility that not all countries have large ports. Incases like this, we can use the medium ports, and if there is no medium ports we can use small ports. This way every country is covered.

Yeah that sounds good. Thank you for moving this forward.

[Eddy-JV]

Hi @GbotemiB @davide-f . I took the liberty to review this PR. All comments are subject to discussion if you do not agree.

[Eddy-JV]

I would suggest to name it: export_ports. I think it is more intuitive for the users.

[Eddy-JV]

Here it is better to have an example file (not empty) where the needed columns are shown and at least an example row is also available. This way it is easier for the users to insert their custom data in the right format.

[Eddy-JV]

Ah we should also delete the export_ports.csv from data folder as it is not needed anymore. Maybe just move it here as it is.

[GbotemiB]

What I have done is to move the export_ports.csv file to the custom folder which should serve as an example. Let me know your thoughts on this.

[Eddy-JV]

I personally prefer that all files in resources to be the ones that are actually used in the workflow. This way the user can track a file easily if the file is not in resources then the workflow used the file from the data folder.

Therefore, I suggest to have this rule read only export_ports="resources/" + SECDIR + "export_ports.csv",
and the decision on using custom data or workflow generated one stays in prepare_ports.py.

Thus in prepare_ports.py, either:

• the file is copied from data/custom to resources if the config["custom_data"]["export_data"] is True
• or filter_ports function is activated and the output is saved in resources.

Please note that all my comments are subject to discussion if needed.

[Eddy-JV]

Here we can put an if clause that i suggested in my Snakefile comment.

[Eddy-JV]

From my side, the PR looks good now from the scripts changes point.

@davide-f @GbotemiB I can do the merge if the failing run checks are not important.

[davide-f]

Great new development!