Draft modular config structure

[ekatef]

An attempt to organise the configuration file in a readable way

Changes proposed in this Pull Request

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

That is indeed a great draft. Clearly this is not (yet) integrated as a workflow, but it is a great proposal.
It will have implications to the other repo (-distribution and alike), but it is a great proposal in my opinion, and we need to start :)

We can reflect on the naming, for example:

• geometry may be replaced with spatial
• weather may be replaced with load_weather or alike; just to clarify also the load option.

They are quite many files; I am wondering if the sector configs may be merged (e.g. heating/hydrogen/transport) into a config.sector.yaml for example

Moreover, I'm wondering about the "main" settings of the scenario, run and enable keys for example; maybe a "header" config may be interesting to aggregate some of those options?

It would be great to collect ideas

[ekatef]

Thank you so much for looking into that @davide-f! Happy to hear that you like it 🙂

I expect that it'll take some iterations to find a convenient and robust approach to deal with the configs. But it feels easier to think about different options when playing with a possible implementation. So, let's experiment with that!

Agree that it would be cleaner to keep a reasonable number of the modular configs. Have renamed geometry with spatial and merged:

• heating+transport+H2 into the sectoral config
• monte-carlo parameters included into solving config.

Not sure it would increase clarity if we add load to weather in the file name. Currently, mixing the weather year with socioecomonic parameters seems to be a reason of many confusions. So, my preference would be to leave weather only, but happy to revise if you think it'll make things more clear

Absolutely agree that we need also to think about management of the configuration files, also keeping in mind the requirements of the distribution model.

A great point on having a header file. Happy to implement that and have to find a proper way to do so 😄 That directly relates to creating a good architecture, and I'd be very grateful for your support in that.

[ekatef]

Some additionally points which would be great to discuss are:

• Would be great to ensure that management of the configs do not lead to any data loss. In particular, if config.yaml presents in the working folder, it must be kept untouched (can we do a reserve copy?)
• Some sanity checks of the merged config.yaml are needed. E.g. according to this issue Snakemake allows for duplicated keys in the config.

Obviously, more ideas are welcome

[ekatef]

As discussed previously, it would be also great to add a parser for the config to check it for consistency.

The requirements can include:

• all the keys are unique
• years used to define different parameters are the same (not realistic, probably) or there is an overview on which year is assumed for what (e.g. GDP, installed renewable capacity, industry capacity, energy balances, weather year)
• parameters of the sector-coupled scenarios are consistent (e.g. bev_avail_max > bev_avail_mean)
• assumptions for particular scenarios can be easily checked(is also linked with the need to document the configuration parameters for the sector-coupled capabilities Add sector-coupled configuration to docs #1208)