STS groups : make group categories dynamic rather than fixed (#2007)

We make groups of **S**hort **T**erm **S**torage dynamic. 
This means that accepted groups are no longer fixed in the code.
Instead, any group can be defined by user.
It has consequences on the **Thematic Trimming**, which now accepts only
one item (namely **STS BY GROUP**) regarding STS groups output
variables.
Turning it to on prints all variables related to groups. 

Note : 
- We added a default group name (in case user doesn't specify a group
name for a cluster) : "**sts-def-group**". Feel free to amend it.
- A new test (a study) was created. Please find it
[here](https://github.com/AntaresSimulatorTeam/Antares_Simulator/files/14802896/ST-dynamic-groups.zip).
Look at its **user's notes** to know its intentions.
- upon a @sylvlecl suggestion, doc was updated accordingly

## Required
- [x] Merge
https://github.com/AntaresSimulatorTeam/Antares_Simulator_Tests_NR/pull/18/files,
generate new reference results

---------

Co-authored-by: Florian Omnès <[email protected]>

docs/CHANGELOG.md

@@ -1,5 +1,9 @@
 Antares Changelog
 =================
+Next version 
+--------------------
+## Improvements
+* STS groups are now "dynamic" : group names are no longer fixed by code, user is free to define these groups.
 
 9.0.0
 --------------------

docs/reference-guide/13-file-format.md

@@ -1,7 +1,8 @@
 # Study format changes
 This is a list of all recent changes that came with new Antares Simulator features. The main goal of this document is to lower the costs of changing existing interfaces, both GUI and scripts.
 ## v9.1.0
-### (Input) Hydro Maximum Generation/Pumping Power
+## Input 
+### Hydro Maximum Generation/Pumping Power
 * For time series ![Migration diagram](migration.png "Migration diagram"), for more details, see [this Python script](migration.py)
 * In the existing file **settings/scenariobuilder.dat**, under **<ruleset>** section following properties added: **hgp,<area>,<year> = <hgp-value>**
 
@@ -14,6 +15,38 @@ This implies that, inside one of the previous categories, the number of availabl
 
 * [Logic changes](17-v91.md)
 
+### Short term storage groups
+STS groups in input are now "dynamic" : group names are no longer fixed by code, user is free to define these groups.
+
+In the "thematic trimming" section, the new dynamic variable `STS by group` is now used to enable/disable all variables (level/injection/withdrawal) for all groups. The following variables are obsolete and must not be provided
+
+```
+PSP_open_injection
+PSP_open_withdrawal
+PSP_open_level
+...
+
+Other1_injection
+Other1_withdral
+Other1_level
+...
+Other5_injection
+Other5_withdral
+Other5_level
+```
+(3*9=27 variables in total)
+
+The default value for group is "OTHER1".
+
+## Output
+### Hydro maximum generation/pumping power
+In existing directory `ts-numbers`, add sub-directory `hgp` containing TS numbers for hydro max pumping/generation, for each area.
+
+### ST Storage
+- Output columns for ST storage are capitalized. For any STS group name my_group, 3 output columns are created : `MY_GROUP_INJECTION`, `MY_GROUP_WITHDRAWAL`, `MY_GROUP_LEVEL`.
+- If a group is empty, no column is produced.
+- There is now a variable number of columns in files values-XXX.txt, depending on the groups of ST storages provided by the user. Note that groups are case-insensitive, for example `battery`, `Battery` and `BATTERY` all represent the same group. If no ST storage exist for a given area, no variables associated to ST storages will be produced.
+
 ## v9.0.0
 ### Input
 ### Study version

src/libs/antares/study/CMakeLists.txt

@@ -105,8 +105,6 @@ set(SRC_STUDY_PART_SHORT_TERM_STORAGE
         include/antares/study/parts/short-term-storage/series.h
         include/antares/study/parts/short-term-storage/cluster.h
         parts/short-term-storage/cluster.cpp
-        include/antares/study/parts/short-term-storage/STStorageOutputCaptions.h
-        parts/short-term-storage/STStorageOutputCaptions.cpp
 )
 source_group("study\\part\\short-term-storage" FILES ${SRC_STUDY_PART_SHORT_TERM_SOTRAGE})
 

src/libs/antares/study/area/list.cpp

@@ -778,6 +778,7 @@ static bool AreaListLoadFromFolderSingleArea(Study& study,
     area.spreadSpilledEnergyCost = 0.;
 
     bool ret = true;
+    const auto studyVersion = study.header.version;
 
     // DSM, Reserves, D-1
     buffer.clear() << study.folderInput << SEP << "reserves" << SEP << area.id << ".txt";
@@ -879,12 +880,12 @@ static bool AreaListLoadFromFolderSingleArea(Study& study,
         if (!options.loadOnlyNeeded || !area.hydro.prepro) // Series
         {
             buffer.clear() << study.folderInput << SEP << "hydro" << SEP << "series";
-            ret = hydroSeries->loadGenerationTS(area.id, buffer, study.header.version) && ret;
+            ret = hydroSeries->loadGenerationTS(area.id, buffer, studyVersion) && ret;
 
             hydroSeries->EqualizeGenerationTSsizes(area, study.usedByTheSolver);
         }
 
-        if (study.header.version < StudyVersion(9,1))
+        if (studyVersion < StudyVersion(9,1))
         {
             buffer.clear() << study.folderInput << SEP << "hydro";
 
@@ -905,7 +906,7 @@ static bool AreaListLoadFromFolderSingleArea(Study& study,
         }
 
         hydroSeries->resizeTSinDeratedMode(
-            study.parameters.derated, study.header.version, study.usedByTheSolver);
+            study.parameters.derated, studyVersion, study.usedByTheSolver);
     }
 
     // Wind
@@ -940,7 +941,7 @@ static bool AreaListLoadFromFolderSingleArea(Study& study,
     }
 
     // Short term storage
-    if (study.header.version >= StudyVersion(8, 6))
+    if (studyVersion >= StudyVersion(8, 6))
     {
         buffer.clear() << study.folderInput << SEP << "st-storage" << SEP << "series"
             << SEP << area.id;
@@ -950,7 +951,7 @@ static bool AreaListLoadFromFolderSingleArea(Study& study,
     }
 
     // Renewable cluster list
-    if (study.header.version >= StudyVersion(8, 1))
+    if (studyVersion >= StudyVersion(8, 1))
     {
         buffer.clear() << study.folderInput << SEP << "renewables" << SEP << "series";
         ret = area.renewable.list.loadDataSeriesFromFolder(study, buffer) && ret;
@@ -1064,6 +1065,7 @@ bool AreaList::loadFromFolder(const StudyLoadOptions& options)
 {
     bool ret = true;
     Clob buffer;
+    auto studyVersion = pStudy.header.version;
 
     // Load the list of all available areas
     {
@@ -1102,7 +1104,7 @@ bool AreaList::loadFromFolder(const StudyLoadOptions& options)
     }
 
     // Short term storage data, specific to areas
-    if (pStudy.header.version >= StudyVersion(8, 6))
+    if (studyVersion >= StudyVersion(8, 6))
     {
         logs.info() << "Loading short term storage clusters...";
         buffer.clear() << pStudy.folderInput << SEP << "st-storage";
@@ -1123,7 +1125,7 @@ bool AreaList::loadFromFolder(const StudyLoadOptions& options)
     }
 
     // Renewable data, specific to areas
-    if (pStudy.header.version >= StudyVersion(8, 1))
+    if (studyVersion >= StudyVersion(8, 1))
     {
         // The cluster list must be loaded before the method
         // ensureDataIsInitialized is called

src/libs/antares/study/include/antares/study/parts/short-term-storage/properties.h

@@ -28,21 +28,6 @@
 
 namespace Antares::Data::ShortTermStorage
 {
-enum class Group
-{
-    PSP_open,
-    PSP_closed,
-    Pondage,
-    Battery,
-    Other1,
-    Other2,
-    Other3,
-    Other4,
-    Other5
-};
-
-unsigned int groupIndex(Group group);
-
 class Properties
 {
 public:
@@ -62,15 +47,14 @@ class Properties
     bool initialLevelOptim = false;
     /// Efficiency factor between 0 and 1
     double efficiencyFactor = 1;
-    /// Used to sort outputs
-    Group group = Group::Other1;
+    // Used to sort outputs
+    std::string groupName = "OTHER1";
     /// cluster name
     std::string name;
 
     /// Enabled ?
     bool enabled = true;
 
-    static const std::map<std::string, enum Group> ST_STORAGE_PROPERTY_GROUP_ENUM;
 private:
     static constexpr double initiallevelDefault = .5;
 };

src/libs/antares/study/parameters.cpp

@@ -33,6 +33,7 @@
 #include <antares/logs/logs.h>
 #include "antares/study/load-options.h"
 #include <climits>
+#include <boost/algorithm/string/case_conv.hpp>
 #include "antares/solver/variable/economy/all.h"
 
 #include <antares/exception/AssertionError.hpp>
@@ -854,6 +855,25 @@ static bool SGDIntLoadFamily_Playlist(Parameters& d,
     return false;
 }
 
+static bool deprecatedVariable(std::string var)
+{
+    static const std::vector<std::string> STSGroups_legacy
+      = {"psp_open_level",      "psp_closed_level",      "pondage_level",
+         "battery_level",       "other1_level",          "other2_level",
+         "other3_level",        "other4_level",          "other5_level",
+
+         "psp_open_injection",  "psp_closed_injection",  "pondage_injection",
+         "battery_injection",   "other1_injection",      "other2_injection",
+         "other3_injection",    "other4_injection",      "other5_injection",
+
+         "psp_open_withdrawal", "psp_closed_withdrawal", "pondage_withdrawal",
+         "battery_withdrawal",  "other1_withdrawal",     "other2_withdrawal",
+         "other3_withdrawal",   "other4_withdrawal",     "other5_withdrawal"};
+    boost::to_lower(var);
+    return std::find(STSGroups_legacy.begin(), STSGroups_legacy.end(), var)
+           != STSGroups_legacy.end();
+}
+
 static bool SGDIntLoadFamily_VariablesSelection(Parameters& d,
                                                 const String& key,
                                                 const String& value,
@@ -867,6 +887,12 @@ static bool SGDIntLoadFamily_VariablesSelection(Parameters& d,
     }
     if (key == "select_var +" || key == "select_var -")
     {
+        if (deprecatedVariable(value.to<std::string>()))
+        {
+            logs.warning() << "Output variable `" << original
+                           << "` no longer exists and has been ignored";
+            return true;
+        }
         // Check if the read output variable exists
         if (not d.variablesPrintInfo.exists(value.to<std::string>()))
         {

src/libs/antares/study/parts/short-term-storage/properties.cpp

@@ -21,50 +21,14 @@
 
 #include <antares/logs/logs.h>
 #include <stdexcept>
+#include <boost/algorithm/string/case_conv.hpp>
 
 #include "antares/study/parts/short-term-storage/properties.h"
 
 #define SEP Yuni::IO::Separator
 
 namespace Antares::Data::ShortTermStorage
 {
-const std::map<std::string, enum Group> Properties::ST_STORAGE_PROPERTY_GROUP_ENUM
-  = {{"PSP_open", Group::PSP_open},
-     {"PSP_closed", Group::PSP_closed},
-     {"Pondage", Group::Pondage},
-     {"Battery", Group::Battery},
-     {"Other1", Group::Other1},
-     {"Other2", Group::Other2},
-     {"Other3", Group::Other3},
-     {"Other4", Group::Other4},
-     {"Other5", Group::Other5}};
-
-unsigned int groupIndex(Group group)
-{
-    switch (group)
-    {
-    case Group::PSP_open:
-        return 0;
-    case Group::PSP_closed:
-        return 1;
-    case Group::Pondage:
-        return 2;
-    case Group::Battery:
-        return 3;
-    case Group::Other1:
-        return 4;
-    case Group::Other2:
-        return 5;
-    case Group::Other3:
-        return 6;
-    case Group::Other4:
-        return 7;
-    case Group::Other5:
-        return 8;
-    default:
-        throw std::invalid_argument("Group not recognized");
-    }
-}
 
 bool Properties::loadKey(const IniFile::Property* p)
 {
@@ -100,13 +64,9 @@ bool Properties::loadKey(const IniFile::Property* p)
 
     if (p->key == "group")
     {
-        if (auto it = Properties::ST_STORAGE_PROPERTY_GROUP_ENUM.find(p->value.c_str());
-            it != Properties::ST_STORAGE_PROPERTY_GROUP_ENUM.end())
-        {
-            this->group = it->second;
-            return true;
-        }
-        return false;
+        this->groupName = p->value.c_str();
+        boost::to_upper(this->groupName);
+        return true;
     }
 
     if (p->key == "enabled")
@@ -120,11 +80,7 @@ void Properties::save(IniFile& ini) const
     IniFile::Section* s = ini.addSection(this->name);
 
     s->add("name", this->name);
-
-    for (const auto& [key, value] : ST_STORAGE_PROPERTY_GROUP_ENUM)
-        if (value == this->group)
-            s->add("group", key);
-
+    s->add("group", this->groupName);
     s->add("reservoircapacity", this->reservoirCapacity);
     s->add("initiallevel", this->initialLevel);
     s->add("injectionnominalcapacity", this->injectionNominalCapacity);