Error in rule simplify_network for Kenya

[Haiping007]

Hi, I tried to run pypsa-earth for Kenya many times, and it keep failed in the step of simplify_network, showing "NameError: name 'config' is not defined" and "errors in rule simply_network" . I run the the same approach for Madagascar which is fine. I am wondering: are there any issues for the data of Kenya? Does anyone has the experience or suggestion on how to solve it?

The information about the errors in the terminal is as the follows:

......
INFO:snakemake.logging:
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\networkclustering.py:16: UserWarning: The namespace pypsa.networkclustering is deprecated and will be removed in PyPSA v0.24. Please use pypsa.clustering.spatial instead.
warnings.warn(
This is the repository path: C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth
Had to go 0 folder(s) up.
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
INFO:pypsa.io:Imported network elec.nc has buses, carriers, generators, lines, links, loads, storage_units, transformers
INFO:main:Mapping all network lines onto a single 380kV layer
INFO:main:Simplifying connected link components
Traceback (most recent call last):
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp_3tz6bla.simplify_network.py", line 857, in
n, simplify_links_map = simplify_links(
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp_3tz6bla.simplify_network.py", line 384, in simplify_links
connection_costs_per_link = _prepare_connection_costs_per_link(
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp_3tz6bla.simplify_network.py", line 169, in _prepare_connection_costs_per_link
if config["electricity"]["hvdc_as_lines"]:

NameError: name 'config' is not defined
[Fri Sep 29 21:11:54 2023]
INFO:snakemake.logging:[Fri Sep 29 21:11:54 2023]
Error in rule simplify_network:
jobid: 5
input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
log: logs/simplify_network/elec_s.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule simplify_network:
jobid: 5
input: networks/elec.nc, resources/costs.csv, resources/bus_regions/regions_onshore.geojson, resources/bus_regions/regions_offshore.geojson
output: networks/elec_s.nc, resources/bus_regions/regions_onshore_elec_s.geojson, resources/bus_regions/regions_offshore_elec_s.geojson, resources/bus_regions/busmap_elec_s.csv, resources/bus_regions/connection_costs_s.csv
log: logs/simplify_network/elec_s.log (check log file(s) for error details)

RuleException:
CalledProcessError in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 590:
Command 'C:/Users/haipings/AppData/Local/miniconda3/envs/pypsa-earth/python.exe "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp_3tz6bla.simplify_network.py"' returned non-zero exit status 1.
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile", line 590, in __rule_simplify_network
File "C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 590:
Command 'C:/Users/haipings/AppData/Local/miniconda3/envs/pypsa-earth/python.exe "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp_3tz6bla.simplify_network.py"' returned non-zero exit status 1.
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile", line 590, in __rule_simplify_network
File "C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-09-29T193401.403530.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake\log\2023-09-29T193401.403530.snakemake.log

[ekatef]

Hello @Haiping007!
Sorry to hear that you are having troubles, and thanks a lot for the detailed report.
I can reproduce this issue. That is a bug linked with some recent changes in the project. We'll fix it during the next couple of days. Thanks a lot for notifying!

[Haiping007]

Hi @ekatef @davide-f

Thank you very much for fixing simplify_link bug. I rerun the pypsa-earth for Kenya. The step of simplify_network and cluster_network went through, but there are some errors with the step of prepare_network that showing "KeyError: 'automatic_emission'" and "Error in rule prepare_network". Do you have any suggestion on how to solve it?

The information about the errors in the terminal is as the follows:

INFO:snakemake.logging:
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\networkclustering.py:16: UserWarning: The namespace pypsa.networkclustering is deprecated and will be removed in PyPSA v0.24. Please use pypsa.clustering.spatial instead.
warnings.warn(
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
INFO:pypsa.io:Imported network elec_s_10_ec.nc has buses, carriers, generators, lines, links, loads, storage_units, stores
INFO:main:N-1 security margin of lines set to 0.7
INFO:main:Resampling the network to 1H
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
Traceback (most recent call last):
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp5ypqzr32.prepare_network.py", line 361, in
if snakemake.params.electricity["automatic_emission"]:
KeyError: 'automatic_emission'
[Sun Oct 1 16:49:36 2023]
INFO:snakemake.logging:[Sun Oct 1 16:49:36 2023]
Error in rule prepare_network:
jobid: 2
input: networks/elec_s_10_ec.nc, resources/costs.csv
output: networks/elec_s_10_ec_lcopt_Co2L-1H.nc
log: logs/prepare_network/elec_s_10_ec_lcopt_Co2L-1H.log (check log file(s) for error details)

ERROR:snakemake.logging:Error in rule prepare_network:
jobid: 2
input: networks/elec_s_10_ec.nc, resources/costs.csv
output: networks/elec_s_10_ec_lcopt_Co2L-1H.nc
log: logs/prepare_network/elec_s_10_ec_lcopt_Co2L-1H.log (check log file(s) for error details)

RuleException:
CalledProcessError in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 775:
Command 'C:/Users/haipings/AppData/Local/miniconda3/envs/pypsa-earth/python.exe "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp5ypqzr32.prepare_network.py"' returned non-zero exit status 1.
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile", line 775, in __rule_prepare_network
File "C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
ERROR:snakemake.logging:RuleException:
CalledProcessError in file C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile, line 775:
Command 'C:/Users/haipings/AppData/Local/miniconda3/envs/pypsa-earth/python.exe "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\scripts\tmp5ypqzr32.prepare_network.py"' returned non-zero exit status 1.
File "C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth\Snakefile", line 775, in __rule_prepare_network
File "C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\concurrent\futures\thread.py", line 58, in run
Shutting down, this might take some time.
WARNING:snakemake.logging:Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
ERROR:snakemake.logging:Exiting because a job execution failed. Look above for error message
Complete log: .snakemake\log\2023-10-01T152904.681165.snakemake.log
WARNING:snakemake.logging:Complete log: .snakemake\log\2023-10-01T152904.681165.snakemake.log

[ekatef]

Hello @Haiping007!
Glad that the fix has worked for you.

Regarding the new issue, most likely it can be resolved by if you update you your configuration file, using config.default.yaml (config.yaml is not tracked by git). The has been a recent improvement which required adding a new parameter to the config, which is now referenced in the code.

[Haiping007]

Hi @ekatef @davide-f
Thank you so much for your quick reply! I have updated the config.yaml and rerun. The step of prepare_network went smoothly!
But the process was stopped at "pypsa.linopf:Solve linear problem using Glpk solver".
I have tried many times (such as install Glpk solver and let it run over night.....), but it seems dead at that step of solve_network.
Could you help me with this?
Thanks again!

The information in the terminal is as the follows:

INFO:snakemake.logging:
[Mon Oct 2 23:00:54 2023]
INFO:snakemake.logging:[Mon Oct 2 23:00:54 2023]
rule solve_network:
input: networks/elec_s_10_ec_lcopt_Co2L-1H.nc
output: results/networks/elec_s_10_ec_lcopt_Co2L-1H.nc
log: logs\solve_network\elec_s_10_ec_lcopt_Co2L-1H_solver.log, logs/solve_network/elec_s_10_ec_lcopt_Co2L-1H_python.log
jobid: 1
benchmark: benchmarks/solve_network/elec_s_10_ec_lcopt_Co2L-1H
reason: Missing output files: results/networks/elec_s_10_ec_lcopt_Co2L-1H.nc; Input files updated by another job: networks/elec_s_10_ec_lcopt_Co2L-1H.nc
wildcards: simpl=, clusters=10, ll=copt, opts=Co2L-1H
resources: tmpdir=C:\Users\haipings\AppData\Local\Temp, mem=35850
INFO:snakemake.logging:rule solve_network:
input: networks/elec_s_10_ec_lcopt_Co2L-1H.nc
output: results/networks/elec_s_10_ec_lcopt_Co2L-1H.nc
log: logs\solve_network\elec_s_10_ec_lcopt_Co2L-1H_solver.log, logs/solve_network/elec_s_10_ec_lcopt_Co2L-1H_python.log
jobid: 1
benchmark: benchmarks/solve_network/elec_s_10_ec_lcopt_Co2L-1H
reason: Missing output files: results/networks/elec_s_10_ec_lcopt_Co2L-1H.nc; Input files updated by another job: networks/elec_s_10_ec_lcopt_Co2L-1H.nc
wildcards: simpl=, clusters=10, ll=copt, opts=Co2L-1H
resources: tmpdir=C:\Users\haipings\AppData\Local\Temp, mem=35850

INFO:snakemake.logging:
Changing to shadow directory: C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\shadow\tmpnow0w1qg
WARNING:snakemake.logging:Changing to shadow directory: C:\Users\haipings\pypsa\pypsa-earth_KE_new\pypsa-earth.snakemake\shadow\tmpnow0w1qg
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\networkclustering.py:16: UserWarning: The namespace pypsa.networkclustering is deprecated and will be removed in PyPSA v0.24. Please use pypsa.clustering.spatial instead.
warnings.warn(
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\components.py:318: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
attrs.loc[bool_b, "default"] = attrs.loc[bool_b].isin({True, "True"})
INFO:pypsa.io:Imported network elec_s_10_ec_lcopt_Co2L-1H.nc has buses, carriers, generators, global_constraints, lines, links, loads, storage_units, stores
INFO:pypsa.linopf:Prepare linear problem
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\linopf.py:530: FutureWarning: DataFrame.groupby with axis=1 is deprecated. Do frame.T.groupby(...) without axis instead.
pd.concat([bus_injection(*arg) for arg in args], axis=1)
C:\Users\haipings\AppData\Local\miniconda3\envs\pypsa-earth\lib\site-packages\pypsa\linopf.py:538: FutureWarning: DataFrame.groupby with axis=1 is deprecated. Do frame.T.groupby(...) without axis instead.
(-get_as_dense(n, "Load", "p_set", sns) * n.loads.sign)
INFO:pypsa.linopf:Total preparation time: 11.12s
INFO:pypsa.linopf:Solve linear problem using Glpk solver

[davide-f]

Awesome to hear that you fixed that error :)
The problem lays in the use of glpk for large problems: glpk is inefficient and for large problems (e.g. hourly simulations and/or some clusters) it easily becomes impractical.

If you are a student, you should be able to get a gurobi license for free and you could use gurobi.
Otherwise, you may need to reduce the spatial and time resolution.
To do the latter, in the config, you may change scenario->opts into "Co2L-3H" or 6H. 3H average the time steps every 3h while 6H every 6H, reducing the computational burden at the cost of inaccuracy

[Haiping007]

Hi @davide-f
Thank you for the fast response!
Yes, I have a gurobi license and rerun with gurobi. Now, all steps done!
Thank you so much for support!