Merge pull request #5 from KTH-dESA/minor-updates

README.md

@@ -1,6 +1,6 @@
 # Global Sensitivity Analysis of an Energy System Optimisation Model
 
-The snakemake workflow to conduct a global sensitivity analysis
+The snakemake workflow to conduct a global sensitivity analysis for an OSeMOSYS Model
 
 ## Getting Started
 
@@ -33,100 +33,87 @@ pip install -r requirements.txt
 
 ## Configure the workflow
 
-Update the paths in the `config/config.yaml` file for `datapackage:` and `model_file:` keys. These should both hold relative paths to the combined GUI model `datapackage.json` and osemosys model file (e.g. `osemosys_fast.txt`).
+### Configuration File: `config.yaml`
 
-Each of these paths can then point to the repositories outside of the gui_workflow. So deployment to an HPC will involve:
+The file `config/config.yaml` holds workflow configuration options. These options include the location of the model and accompanying data, solver, and sensitivity analysis parameters. Details on how to modify all configuration parameters are located in the `config.yaml` file. 
+
+Datapackage and model file paths can  point to the repositories outside of the gui_workflow. So deployment to an HPC will involve:
 
 1. Copying a release package of OSeMOSYS to the server and unzipping
 2. Cloning the gui_osemosys repository to the server
 3. Cloning this repository to the server
 4. Updating the config so that the paths point to the correct locations
 
-### Main configuration file
-
-A YAML file `config.yaml` must be placed in the config directory.
-
-```yaml
-# Populate the scenarios.csv file with a list of scenario names
-# and path (description optional) to the model datapackage
-datapackage: config/scenarios.csv
-
-# Tell the workflow which model results to plot
-results: config/results.csv
-
-# Filetype options: 'csv' or 'parquet' or 'feather'
-filetype: csv
-
-# Define the uncertain parameters used to define the Monte Carlo sample
-parameters: config/parameters.csv
-
-# Path to the OSeMOSYS model file
-model_file: ../osemosys/OSeMOSYS_GNU_MathProg/src/osemosys_fast.txt
-
-# Choose a solver, choices: 'cbc' or 'gurobi'
-solver: cbc
-
-# Sampling - how large should the sample be?
-replicates: 100
-```
-
 ### Parameters file: `parameters.csv`
 
-A CSV file containing the following structure should be placed in the config directory:
-
-```csv
-name,group,indexes,min_value,max_value,dist,interpolation_index,action
-CapitalCost,capex,"SIMPLICITY,NGCC",500,1100,unif,YEAR,interpolate
-DiscountRate,discountrate,"GLOBAL,NGCC",0.05,0.20,unif,None,fixed
-```
+The file `config/parameters.csv` holds information on what parameters to iterate when performing the sensitivity analysis. The information required is summarized below
 
 column_name | description
 :-- | :--
 name | the name of the OSeMOSYS parameter file into which the values should be written
-group | the group to which the parameter belonngs (groups of like names are moved together)
-indexes | a string of comma-separated entries matching the set elements for the parameter
-min_value | the minimum value that the parameter will be sampled
-max_value | the maximum value that the parameter will be sampled
+group | the group to which the parameter belongs (groups of like names are moved together)
+indexes | a string of comma-separated entries matching the OSeMOSYS set elements for the parameter
+min_value_base_year | the minimum value that the parameter will be sampled in the base year
+max_value_base_year | the maximum value that the parameter will be sampled in the base year
+min_value_end_year | the minimum value that the parameter will be sampled in the end year
+max_value_end_year | the maximum value that the parameter will be sampled in the end year
 dist | the probability distribution - currently, only 'unif' for uniform is supported
-interpolation_index | the index name over which the values will be interpolated
+interpolation_index | the index name over which the values will be interpolated. Only needed for 'interpolate' actions
 action | 'interpolate' will interpolate with a straight-line between the start and end years, where the sample value replaces the end year value; 'fixed' will replace all values in the interpolation index with the sampled value
 
-### Results file `results.csv`
-
-A CSV file containing the following structure should be placed in the config directory:
+An example of a correctly formatted CSV file is shown below:
 
 ```csv
-name
-ProductionByTechnologyAnnual
+name,group,indexes,min_value_base_year,max_value_base_year,min_value_end_year,max_value_end_year,dist,interpolation_index,action
+CapitalCost,capex,"SIMPLICITY,NGCC",500,600,1000,1100,unif,YEAR,interpolate
+CapitalCost,capex,"SIMPLICITY,NGOC",400,500,900,1000,unif,YEAR,interpolate
+DiscountRate,discountrate,"GLOBAL",0.04,0.05,0.15,0.20,unif,None,fixed
 ```
 
-This lists the result files which will be generated by the otoole results processing script.
+### Results file `results.csv`
 
-### Result aggregation file `agg_results.csv`
+The file `config/results.csv` holds information on what variables to run the sensitivity analysis over. The information required is summarized below. If a custom OSeMOSYS model is provided with new sets, the additional set can be added as a new column. 
 
-This CSV file allows users to specify parts of the OSeMOSYS result files to extract and aggregate,
-adding the model run as an index.
+column_name | description
+:-- | :--
+resultfile | Name of OSeMOSYS Result file 
+filename | Name of file to write results to 
+REGION | Model region 
+TECHNOLOGY | Model technology  
+FUEL | Model fuel 
+EMISSION | Model emission  
+YEAR | Model year
+
+An example of a correctly formatted CSV file is shown below. Note that if no index value is provided, the results will sum over all items in that set. For example, for the `AnnualEmissions` results, the sensitivity analysis will be run on the sum of all annual CO2 emissions in the region `SIMPLICITY` over the model horizon.  
 
 ```csv
-resultfile,indices,filename
-ProductionByTechnologyAnnual,"SIMPLICITY,NGCC,SEC_EL",electricity_from_gas
-ProductionByTechnologyAnnual,"SIMPLICITY,RIVER_2,WATIN",water_from_rivers
+resultfile,filename,REGION,TECHNOLOGY,FUEL,EMISSION,YEAR
+TotalDiscountedCost,DiscountedCost,"SIMPLICITY",,,,
+NewCapacity,NewCapacityHydro,"SIMPLICITY","HYDRO",,,
+NewCapacity,NewCapacityCCNG,"SIMPLICITY","CCNG",,,
+AnnualEmissions,Emissions,"SIMPLICITY",,,"CO2",
+TotalCapacityAnnual,TotalCapacityAnnualCCNGLastYear,"R1","CCNG",,,2070
 ```
 
 ### Scenarios file `scenarios.csv`
 
-A CSV file containing the following structure should be placed in the config directory:
+The file `config/scenarios.csv` is used to point the workflow to master models. Use each master models to define macro scenarios - e.g. forcing in and out a key technology. 
 
-```csv
-name,description,path
-0,"Interconnector Optimised",../gui_osemosys/combined_model/combined_datapackage/datapackage.json
-```
+Importantly, each of the master models is used as a base for the N replicates defined in `config.yaml. If you define 3 master models in this file, and N=100, then 300 model runs will be scheduled, but with the same 100 parameter values.
+
+column_name | description
+:-- | :--
+name | Index of the scenarion 
+description | Description of the scenario
+datapackage | path to otool datapackage
+config | path to otool config file 
 
-This file is used to point to master models. Importantly, each of the master models is used as
-a base for the N replicates defined in `config.yaml. If you define 3 master models in this file,
-and N=100, then 300 model runs will be scheduled, but with the same 100 parameter values.
+An example of a correctly formatted CSV file is shown below. 
 
-Use these master models to define macro scenarios - e.g. forcing in and out a key technology.
+```csv
+name,description,datapackage, config
+0,"Interconnector Optimised",config/scenarios/scenario_1/datapackage.json, config/scenarios/scenario_1/config.yaml
+```
 
 ## Running the workflow
 

workflow/rules/results.smk

@@ -8,7 +8,7 @@ def get_input(wildcards):
 
 def get_indices(wildcards):
     indices = RESULTS.set_index('filename').loc[wildcards.result_file].dropna().drop('resultfile').to_dict()
-    return {x:indices[x].split(',') for x in indices}
+    return {x:str(indices[x]).split(',') for x in indices}
 
 rule extract_results:
     input: 

workflow/scripts/create_modelrun.py

@@ -304,7 +304,7 @@ def main(
         model_params[name] = parameter
     model_params = modify_parameters(model_params, parameters, user_config)
     # WriteDatapackage(user_config=user_config).write(model_params, output_filepath, default_values)
-    WriteCsv(user_config=user_config, ).write(model_params, output_filepath, default_values)
+    WriteCsv(user_config=user_config).write(model_params, output_filepath, default_values)
 
 
 if __name__ == "__main__":

workflow/scripts/create_sample.py

@@ -40,7 +40,7 @@ def main(parameters: dict, sample_file: str, replicates: int):
 
     problem = utils.create_salib_problem(parameters)
 
-    sample = morris.sample(problem, N=50, optimal_trajectories=replicates,
+    sample = morris.sample(problem, N=100, optimal_trajectories=replicates,
                            local_optimization=True, seed=42)
 
     np.savetxt(sample_file, sample, delimiter=',')

workflow/scripts/extract_results.py

@@ -97,5 +97,8 @@ def main(input_files: List, output_file: str, indices: Tuple, config: Dict):
         output_file = Path(sys.argv[3]).stem
         indices = get_indices(parameters, output_file)
 
+    if 'YEAR' in indices:
+        indices['YEAR'] = [float(x) for x in indices['YEAR']]
+
     user_config = parse_yaml(yaml_config)
     main(input_files, output_file_path, indices, user_config)