add NREL/ATB electricity costs for the US

[finozzifa]

Goals

This pull request contains the changes performed by @danielelerede-oet and myself (@finozzifa) as agreed with @martacki and Johannes (could not find his github alias).

Proposed final goal: this pull request is the first (intermediate) step of a set of changes that aim at granting the possibility to the model users to use country-specific cost assumptions.

Goal of this pull request: This work in particular proposes an intermediate step and creates a sub-folder outputs/US, where the outputs/costs_yyyy.csv files are copied and updated with NREL/ATB data.

Input and output schemas

NREL/ATB input values and schema

The NREL/ATB electricity data source is available here.

We require the cost assumptions for the years 2020, 2025, 2030, 2035, 2040, 2045 and 2050. The cost assumptions for 2020 are obtained from atb_e_2022 dataset, whereas those for the other years from the atb_e_2024 dataset. The schema of these files is unfortunately slightly different. Namely:

Schema of atb_e_2022

 #   Column                 Non-Null Count   Dtype  
---  ------                 --------------   -----  
 0   atb_year               286998 non-null  int64  
 1   core_metric_key        286812 non-null  object 
 2   core_metric_parameter  286998 non-null  object 
 3   core_metric_case       286998 non-null  object 
 4   crpyears               286998 non-null  object 
 5   technology             286998 non-null  object 
 6   technology_alias       285510 non-null  object 
 7   techdetail             286998 non-null  object 
 8   display_name           285510 non-null  object 
 9   default                285510 non-null  float64
 10  scenario               286998 non-null  object 
 11  core_metric_variable   286998 non-null  object 
 12  units                  286998 non-null  object 
 13  value                  286998 non-null  float64

Schema of atb_e_2024

 #   Column                 Non-Null Count   Dtype  
---  ------                 --------------   -----  
 0   atb_year               572232 non-null  int64  
 1   core_metric_key        572232 non-null  object 
 2   core_metric_parameter  572232 non-null  object 
 3   core_metric_case       572232 non-null  object 
 4   tax_credit_case        212388 non-null  object 
 5   crpyears               572232 non-null  object 
 6   technology             572232 non-null  object 
 7   technology_alias       572232 non-null  object 
 8   techdetail             572232 non-null  object 
 9   techdetail2            506772 non-null  object 
 10  resourcedetail         488328 non-null  object 
 11  display_name           572232 non-null  object 
 12  default                572232 non-null  int64  
 13  scale                  506772 non-null  object 
 14  maturity               506772 non-null  object 
 15  scenario               572232 non-null  object 
 16  core_metric_variable   572232 non-null  int64  
 17  units                  572232 non-null  object 
 18  value                  572232 non-null  float64

We consider a subset of such columns. This can be configured with the configuration from the config.yaml from config["nrel_atb"]["nrel_atb_columns_to_keep"]. For this pull request, the columns taken are

["atb_year", "core_metric_parameter", "core_metric_case", "core_metric_variable", "technology", "technology_alias", "display_name", "scenario", "units", "value"]

where in particular

• atb_year equals the year in the file name. For example atb_year = 2022 for atb_e_2022 or atb_year = 2024 for atb_e_2024
• core_metric_variable equals the year for which the cost assumption is made
• core_metric_parameter has various values. We consider

Column value	Unit
CAPEX	$/kW
CF
Fixed O&M	$/KW-yr
Variable O&M	$/MWh
Fuel	$/MWh
Additional OCC	$/KW
WACC Real

• scenario equals Moderate, Conservative, Advanced

outputs/US/costs_yyyy.csv values and schema

NREL/ATB column	outputs/US/costs_yyyy.csv column	Notes
display_name	technology
core_metric_parameter	parameter
units	unit
	source	taken from config["nrel_atb"]["nrel_atb_source_link"]
	further description	left blank
atb_year	currency_year
scenario	scenario
core_metric_case	financial_case

Changes

Changes in the workflow

The workflow has been updated as follows:

1. Step 1: rule compile_cost_assumptions generates outputs/costs_yyyy.csv files
2. Step 2: rule compile_cost_assumptions_nrel takes the outputs/costs_yyyy.csv files, reads-in the nrel/atb inputs, processes them and outputs a dedicated set of costs for the US in outputs/US/costs_yyyy.csv

The "high level" description of what compile_cost_assumptions_nrel.py does

The "high level" description of what the script does is:

• it loops through the years 2020, 2025, 2030, 2035, 2040, 2045, 2050
• for each year, it reads the corresponding outputs/cost_yyyy.csv file
• for each year, it reads the corresponding atb input file, filters the rows corresponding to the given year and extracts the necessary columns
• for each year, it normalizes the Fixed O&M by Additional OCC (for retrofits technologies) or CAPEX (for any other technology) and changes its unit from $/KW-yr to %-yr
• for each year, it performs the following technology renaming. NREL/ATB contains many more technologies than those listed in the table below. Those technologies are included anyway in the final outputs/US/costs_year.csv files but not renamed.

display_name	source NREL/ATB file	PyPSA technology name
Coal-new -> 2nd Gen Tech	atb_e_2022.parquet	coal
Coal-new	atb_e_2022.parquet, atb_e_2024.parquet	coal
NG F-Frame CT	atb_e_2022.parquet	CCGT
NG Combustion Turbine (F-Frame)	atb_e_2024.parquet	CCGT
Hydropower - NPD 1	atb_e_2022.parquet, atb_e_2024.parquet	hydro
Hydropower - NSD 1	atb_e_2022.parquet, atb_e_2024.parquet	ror
Pumped Storage Hydropower - National Class 1	atb_e_2022.parquet, atb_e_2024.parquet	PHS
Nuclear - Large	atb_e_2024.parquet	nuclear
Nuclear - AP1000	atb_e_2022.parquet	nuclear
Geothermal - Hydro / Flash	atb_e_2022.parquet, atb_e_2024.parquet	geothermal
Land-Based Wind - Class 1	atb_e_2022.parquet	onwind
Land-Based Wind - Class 1 - Technology 1	atb_e_2024.parquet	onwind
Offshore Wind - Class 1	atb_e_2022.parquet, atb_e_2024.parquet	offwind
Utility PV - Class 1	atb_e_2022.parquet, atb_e_2024.parquet	solar-utility
Commercial PV - Class 1	atb_e_2022.parquet, atb_e_2024.parquet	solar-rooftop
Utility-Scale Battery Storage - 6Hr	atb_e_2022.parquet, atb_e_2024.parquet	battery storage
Biopower	atb_e_2022.parquet	biomass
Biopower - Dedicated	atb_e_2022.parquet, atb_e_2024.parquet	biomass
CSP - Class 2	atb_e_2022.parquet, atb_e_2024.parquet	csp-tower

• for each year, it updates the cost values in the corresponding outputs/US/costs_year.csv file so that, the script
  • appends NREL/ATB technology entries not present in the original file outputs/costs_year.csv
  • leaves untouched technology entries not present in the NREL/ATB dataset
  • updates the technology entries present in both the NREL/ATB dataset and outputs/costs_year.csv, with NREL/ATB values
  • appends the values of the discount_rate from the input file discount_rates_usa.csv. This step is necessary because the NREL/ATB dataset provides "grouped names" instead of single technology names. For example, for the parameters CAPEX, CF, Fixed O&M we have the technology Land-Based Wind - Class 1, Land-Based Wind - Class 2 etc. Instead for WACC Real we have just Land-Based Wind
  • appends the values of the fuel from the input file fuel_costs_usa.csv. This step is necessary because the NREL/ATB dataset only presents fuel costs for nuclear and biomass. Oil and gas US-specific fuel costs are fetched from the World Bank's annual prices with projections up to 2030 (based on World Bank's 2026 estimations). Coal fuel cost is fetched from the EIA Annual Coal Report 2023.

Other noteworthy changes

Aligning the technology names of atb_e_2022 to the names of atb_e_2024

The technologies listed below have different name between atb_e_2022.parquet and atb_e_2024.parquet. Therefore the technologies on the left-hand side are renamed to the names on the right-hand side.

atb_e_2022 display_name	atb_e_2024 display_name
Land-Based Wind - Class 2	Land-Based Wind - Class 2 - Technology 1
Land-Based Wind - Class 3	Land-Based Wind - Class 3 - Technology 1
Land-Based Wind - Class 4	Land-Based Wind - Class 4 - Technology 1
Land-Based Wind - Class 5	Land-Based Wind - Class 5 - Technology 1
Land-Based Wind - Class 6	Land-Based Wind - Class 6 - Technology 1
Land-Based Wind - Class 7	Land-Based Wind - Class 7 - Technology 1
Land-Based Wind - Class 8	Land-Based Wind - Class 8 - Technology 2
Land-Based Wind - Class 9	Land-Based Wind - Class 9 - Technology 3
Land-Based Wind - Class 10	Land-Based Wind - Class 10 - Technology 4
NG F-Frame CC	NG 2-on-1 Combined Cycle (F-Frame)
NG H-Frame CC	NG 2-on-1 Combined Cycle (H-Frame)
NG combined cycle 95% CCS (F-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (F-Frame) 95% CCS
NG combined cycle 95% CCS (H-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (H-Frame) 95% CCS
NG combined cycle Max CCS (F-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (F-Frame) 97% CCS
NG combined cycle Max CCS (H-frame basis -> 2nd Gen Tech)	NG 2-on-1 Combined Cycle (H-Frame) 97% CCS
Coal-CCS-95% -> 2nd Gen Tech	Coal-95%-CCS
Coal-Max-CCS -> 2nd Gen Tech	Coal-99%-CCS
Coal-IGCC	Coal - IGCC
CSP - Class 7	CSP - Class 8
Nuclear - Small Modular Reactor	Nuclear - Small

Discarded technologies

The following technologies are present in the input file atb_e_2022.parquet. They are however not present in atb_e_2024.parquet. They are therefore discarded from the final cost output files. They are:

• Coal-CCS-95% -> Transformational Tech
• Coal-Max-CCS -> Transformational Tech
• Coal-new -> Transformational Tech
• NG combined cycle 95% CCS (F-frame basis -> Transformational Tech)
• NG combined cycle 95% CCS (H-frame basis -> Transformational Tech)
• NG combined cycle Max CCS (F-frame basis -> Transformational Tech)
• NG combined cycle Max CCS (H-frame basis -> Transformational Tech)

environment.yaml

We choose to take the the input datasets atb_e_2022 and atb_e_2024 in parquet format. This is because the corresponding csv files have a size which is significantly larger. This choice brings about the following addition to the environment.yaml file

  - pyarrow
  - fastparquet

unit tests

We added a test folder to include unit tests for the functions included in scripts/compile_cost_assumptions_nrel.py

inputs/manual_input.csv

New technologies have been added to manual_inputs.csv.

Checklist

• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Data source for new technologies is clearly stated.
• Newly introduced dependencies are added to environment.yaml (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the GPLv3 license.