time_order KeyError in storage modeling

[SongminYu]

Dear MESSAGE team, while trying to add storage in my model by following test_feature_storage. I received an error message while importing the attached df (Map_TimeOrder.xlsx) by:

scenario.add_set("time_order", df)

The error message is:

Traceback (most recent call last):
  File ".../miniconda3/envs/MESSAGE/lib/python3.10/site-packages/ixmp/core/scenario.py", line 289, in add_set
    self._backend("item_set_elements", "set", name, elements)
  File ".../miniconda3/envs/MESSAGE/lib/python3.10/site-packages/ixmp/core/timeseries.py", line 108, in _backend
    return self.platform._backend(self, method, *args, **kwargs)
  File ".../miniconda3/envs/MESSAGE/lib/python3.10/site-packages/ixmp/backend/base.py", line 55, in __call__
    return getattr(self, method)(obj, *args, **kwargs)
  File "/.../miniconda3/envs/MESSAGE/lib/python3.10/site-packages/ixmp/backend/jdbc.py", line 1086, in item_set_elements
    jobj = self._get_item(s, type, name)
  File ".../miniconda3/envs/MESSAGE/lib/python3.10/site-packages/ixmp/backend/jdbc.py", line 1217, in _get_item
    raise KeyError(name) from None
KeyError: 'time_order'

Question:

I am using message_ix v3.8.0 and it seems a developed feature to add storage and the time_order parameter. I don't know why I am seeing this error message. Or, is the attached df wrong?

Some side information:

After commenting out the single line of code importing thetime_order parameter and the df (all storage technologies and their parameters are imported, as well as the timeslices), I tried solving the model again. Interestingly, it worked. Part of the ACT results is attached here: Anhui.xlsx. As shown in the pivot sheet,

• The charger and discharger have same ACT in each timestep (season-hour).
• dam has constant ACT, which is 1/12 of the CAP --> I don't understand why it is 1/12. There are 4 seasons in a year and each season has 24 hours, so in total there are 96 season-hours.
• battery has zero ACT but the CAP is also zero.

[glatterf42]

Hi @SongminYu, thanks for raising this question.
The file you linked does not import any data from an excel file, so you must have adapted the code to import your time_order df. Please share the exact code you are running with us, as it can be close to impossible to troubleshoot such issues without knowing what exactly you were trying to run.

Apart from this, I must admit I don't fully understand if your excel sheet looks fine, maybe @behnam-zakeri (who developed this feature) can confirm. In the test file, time_order is added via scen.add_par("time_order", ["season", h], time_order[h], "-") and our documentation at least mentions that this parameter has the index sets lvl_temporal and time.
All parameters should have their index set data plus value and unit, as far as I know. But in this case, two index sets are mentioned, but three unnamed inputs are provided to add_par(). I'm not sure if "-" is supposed to be a value or a dimensionless unit and maybe time_order[h] serves as both the index set time and the value? Either way, if something is missing from your excel file, I suspect it's a unit column.
Our documentation for this feature really seems to be lacking, especially since the python code for this feature seems to only live in the test suite and lacks explanation or even type hints. In particular, using make_df in the add_par call above would already prove very useful.

Lastly, I'd like to note that we have a tutorial on seasonality, but this doesn't mention time_order at all.

[SongminYu]

Hi @glatterf42, thanks for your reply! OK, I added the relevant code as below. Basically, I developed a Scenario class that has message_ix.Scenario instance as an attribute. There are also some data importing and populating functions for setting up a MESSAGE scenario. I didn't copy all the code in the class but only the part relevant for importing this time_order parameter.

import ixmp
import message_ix


class Scenario:
    def __init__(self, config: "Config"):
        self.config = config
        self.mp = ixmp.Platform()
        self.scenario = message_ix.Scenario(
            mp=self.mp,
            model=self.config.model_name,
            scenario=self.config.scenario_name,
            version="new"
        )

    def load_xlsx(self, file_name: str, sheet_name: Optional[str] = None):
        if sheet_name is None:
            df = pd.read_excel(os.path.join(self.config.dir_input, f'{file_name}.xlsx'))
        else:
            df = pd.read_excel(os.path.join(self.config.dir_input, f'{file_name}.xlsx'), sheet_name=sheet_name)
        return df
    
    def add_map(self, map_name: str, file_name: str):
        df = self.load_xlsx(file_name)
        self.scenario.add_set(map_name, df)

When adding the time_order parameter, I was calling

scenario = Scenario(config=cfg)
scenario.add_map(map_name="time_order", file_name="Map_TimeOrder")

The weird thing is, I also have the following two lines importing map_temporal_hierarchy (Map_Hierarchy_Time.xlsx) and map_spatial_hierarchy (Map_Hierarchy_Spatial.xlsx), they both work fine.

scenario.add_map(map_name="map_temporal_hierarchy", file_name="Map_Hierarchy_Time")
scenario.add_map(map_name="map_spatial_hierarchy", file_name="Map_Hierarchy_Spatial")

In the test file, time_order is added via scen.add_par("time_order", ["season", h], time_order[h], "-") and our documentation at least mentions that this parameter has the index sets lvl_temporal and time.
All parameters should have their index set data plus value and unit, as far as I know. But in this case, two index sets are mentioned, but three unnamed inputs are provided to add_par(). I'm not sure if "-" is supposed to be a value or a dimensionless unit and maybe time_order[h] serves as both the index set time and the value? Either way, if something is missing from your excel file, I suspect it's a unit column.

Regarding this, yes, I have lvl_temporal and time columns. I tried adding the unit column, the error still exists. What I am not sure are:

• Is the value column name right?
• Given the fact that my model can be solved "successfully" if I just comment out this single line adding time_order parameter, but the results look wrong (charge = discharge for all hours) --> Anyone saw this weird results before? Is the mistake really caused by missing time_order or there might be other reasons?

Lastly, I'd like to note that we have a tutorial on seasonality, but this doesn't mention time_order at all.

Yes, this is because the storage feature is not used in the seasonality tutorial. In the central asia example, storage is used and one can find the time_order parameter dataframe in this file, the "time_order" sheet.

[behnam-zakeri]

@SongminYu and @glatterf42 thank you for the discussion.
time_order is a MESSAGEix parameter for defining the order of sub-annual time slices at a temporal level. For example, if you have 4 time slices defined at the level of "season", you can specify which one is the first, second, third, and fourth. The order should be given using an integer. This feature is only used in storage equations. So, when a model has sub-annual time slices but does not use storage, such as "Westeros_seasonality.ipynb", the user does not need to specify time_order. This parameter has 2 index sets, namely, lvl_temporal and time, value, and unit. The unit can be "-", because the order of time slices has no unit.

1. It seems you are adding the dataframe using Scenario.add_set(), which is not correct, because time_order is a parameter and you should use Scenario.add_par(). Also, your table lacks a column of "unit".
2. time_order exists in the data file of the Central Asia model, and you can find it alphabetically in the parameters. It looks like this:
   
   Please note that here time slices are named "1", "2", "3", ..., "12" as string, while their order is 1, 2, 3, ..., 12 as integer.
   I hope this help.
   @glatterf42 we can add some explanation in the doc to clarify this.

[glatterf42]

Thanks a lot @behnam-zakeri :)
One clarifying question: the names of the time slices are the content of the time column, while the value column contains their order, is that correct?

[SongminYu]

Thanks @behnam-zakeri! Now the parameter is successfully imported. I will look into the results in detail and see my storages are working properly. I also found the time_order parameter in the central asia model. Sorry for overlooking it. I will edit my previous reply and remove this.

[behnam-zakeri]

@glatterf42 yes, that is correct. @SongminYu happy to hear that you managed to import the data successfully.