[ANT-2299] Enable save/restore with xpress and enable by default (ProblemGeneration)

data_test/tests_lpnamer/tests_integer/test_lpnamer_Xpress/output/economy/reference_lp/structure.txt

@@ -1,2 +1,2 @@
                                             master                                 transmission_line         0
-                       problem-1-1--optim-nb-1.mps                                 transmission_line       840
+                       problem-1-1--optim-nb-1.svf                                 transmission_line       840

docs/changelog/CHANGELOG.md

@@ -4,12 +4,13 @@ Antares-Xpansion changelog
 v1.4.0 (12/2024)
 --------------------------------------------------------
 ### Features
-* Update Antares Simulator to  9.2-RC7
+* Update Antares Simulator to  9.2
 * Output file **criterions.txt** is renamed **LOLD.txt** as it gathers the loss of load for each area at each iteration of the algorithm
 * End to end coherence of solvers : if Xpress is selected, it is also used for Antares step (does not work yet with memory mode), if coin is selected, sirius is used in Antares
 * Support for in memory problems between Antares and problem generation
 * Remove local UI support [#972](https://github.com/AntaresSimulatorTeam/antares-xpansion/pull/972)
 * Properly handle Antares Simulator errors. Especially error -9 occurring when running out of memory where Xpansion tried to continue in an improper state.
+* When using xpress, problems generated by problem_generation are in a binary format to save space.
 
 ### For developers
 * Various improvement on dependency management through vcpkg.

docs/developer-guide/xpansion_steps/0-overview.md

@@ -20,8 +20,7 @@ This stage is responsible for building the investment optimization problem, usin
 
 Using the data on investment candidates specified in the `user/expansion/candidates.ini` file,
 Antares-Xpansion modifies the `.mps` problems provided by
-Antares-Simulator in the previous step to add information on the candidates. These modified problems, still written
- as `.mps` files, are the so-called _satellite problems_. They correspond to the supply-demand problem to be solved once the investment has been fixed to a given value (that changes during the Antares-Xpansion optimization).
+Antares-Simulator in the previous step to add information on the candidates. These modified problems are the so-called _satellite problems_. They correspond to the supply-demand problem to be solved once the investment has been fixed to a given value (that changes during the Antares-Xpansion optimization)
 
 This stage is also responsible for building the initial _master problem_
 used for Benders decomposition. If there are additional constraints linking the investment candidates (defined in the `additional-constraints` field of the `user/expansion/settings.ini` file), then they are added to the _master problem_.

docs/developer-guide/xpansion_steps/2-problem-modification.md

@@ -154,26 +154,26 @@ The _master problem_ is created from the list of candidates in the following way
 
 ### 5- Creation of a variable / column ID mapping file
 
-A file (`structure.txt`) is created, containing for each `.mps` file, the associated column ID for the candidates.
+A file (`structure.txt`) is created, containing for each satellite problem and the master, the associated column ID for the candidates.
 
 !!! Example
     ```
-                        master  battery                   0
-                        master  peak                      1
-                        master  pv                        2
-                        master  semibase                  3
-                        master  transmission_line         4
-    problem-1-1-20210713-163528  battery                5547
-    problem-1-1-20210713-163528  peak                   5545
-    problem-1-1-20210713-163528  pv                     5548
-    problem-1-1-20210713-163528  semibase               5546
-    problem-1-1-20210713-163528  transmission_line      5544
+                        master          battery                 0
+                        master          peak                    1
+                        master          pv                      2
+                        master          semibase                3
+                        master          transmission_line       4
+    problem-1-1-20210713-163528.mps     battery                 5547
+    problem-1-1-20210713-163528.mps     peak                    5545
+    problem-1-1-20210713-163528.mps     pv                      5548
+    problem-1-1-20210713-163528.mps     semibase                5546
+    problem-1-1-20210713-163528.mps     transmission_line       5544
     ```
 
     Here:
 
     - In `master.mps`, the candidate `battery` has column ID `0`,
-    - In `problem-1-1-20210713-163528`, the candidate `peak` has column ID `5545`.
+    - In `problem-1-1-20210713-163528` corresponding file `problem-1-1-20210713-163528.mps`, the candidate `peak` has column ID `5545`.
 
 !!! Note
     The column ID is no longer used by `benders`. This file must still be created so that `benders` knows the list of candidate for each problem.

src/cpp/benders/benders_core/BendersBase.cpp

@@ -686,8 +686,13 @@ double BendersBase::SubproblemWeight(int subproblem_count,
  *  \brief Get path to master problem mps file from options
  */
 std::filesystem::path BendersBase::get_master_path() const {
-  return std::filesystem::path(_options.INPUTROOT) /
-         (_options.MASTER_NAME + MPS_SUFFIX);
+  if (_options.PROBLEMS_FORMAT == ProblemsFormat::SAVED_FILE && _options.SOLVER_NAME == "XPRESS") {
+    return std::filesystem::path(_options.INPUTROOT) /
+           (_options.MASTER_NAME + SAVE_SUFFIX);
+  } else {
+    return std::filesystem::path(_options.INPUTROOT) /
+           (_options.MASTER_NAME + MPS_SUFFIX);
+  }
 }
 /*!
  *  \brief Get path to last mps file of master problem

src/cpp/benders/benders_core/CMakeLists.txt

@@ -31,22 +31,22 @@ target_sources(benders_core PRIVATE
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/BendersBase.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/BendersMathLogger.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/BendersStructsDatas.h
+        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CriterionComputation.h
+        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CriterionInputDataReader.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CustomVector.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CutsManagement.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/LastIterationPrinter.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/LastIterationReader.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/LastIterationWriter.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/MasterUpdate.h
+        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/ProblemFormat.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/SimulationOptions.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/StartUp.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/SubproblemWorker.h
+        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/VariablesGroup.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/Worker.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/WorkerMaster.h
         ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/common.h
-        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CriterionComputation.h
-        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CriterionInputDataReader.h
-        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/VariablesGroup.h
-        ${CMAKE_CURRENT_SOURCE_DIR}/include/antares-xpansion/benders/benders_core/CouplingMapGenerator.h
 )
 
 add_library(antaresXpansion::benders_core ALIAS benders_core)

src/cpp/benders/benders_core/SimulationOptions.cpp

@@ -19,6 +19,9 @@ Json::Value SimulationOptions::get_value_from_json(
   }
   return _input;
 }
+
+template <> inline ProblemsFormat Json::Value::as<ProblemsFormat>() const { return problemsFormatFromString(asString()); }
+
 /*!
  *  \brief Constructor of Benders Options
  *
@@ -133,6 +136,7 @@ BaseOptions SimulationOptions::get_base_options() const {
   result.SLAVE_WEIGHT = SLAVE_WEIGHT;
   result.MASTER_NAME = MASTER_NAME;
   result.STRUCTURE_FILE = STRUCTURE_FILE;
+  result.PROBLEMS_FORMAT = PROBLEMS_FORMAT;
   result.INPUTROOT = INPUTROOT;
   result.SOLVER_NAME = SOLVER_NAME;
   result.weights = _weights;

src/cpp/benders/benders_core/Worker.cpp

@@ -51,7 +51,7 @@ void Worker::init(VariableMap const &variable_map,
 
   _solver->set_threads(1);
   _solver->set_output_log_level(log_level);
-  _solver->read_prob_mps(path_to_mps);
+  read_prob(_solver.get(), path_to_mps);
 
   _name_to_id = variable_map;
 
@@ -142,3 +142,21 @@ void Worker::AddRows(std::vector<char> const &qrtype_p,
 
 int Worker::Getnrows() const { return _solver->get_nrows(); }
 int Worker::Getncols() const { return _solver->get_ncols(); }
+
+/**
+ * Have `problem` read the problem problem data from `path`
+ *
+ * Used to hold logic to select between mps/save/etc.
+ * @param problem
+ * @param path
+ */
+void Worker::read_prob(SolverAbstract * problem,
+                       const std::filesystem::path &path) const {
+  if (path.extension() == ".mps") {
+    problem->read_prob_mps(path);
+  } else if (path.extension() == ".svf") {//Brittle
+    problem->restore_prob(path);
+  } else {
+    throw LogUtils::XpansionError<std::runtime_error>("Unknown file extension for problem file: " + path.string(), LOGLOCATION);
+  }
+}

src/cpp/benders/benders_core/include/antares-xpansion/benders/benders_core/BendersStructsDatas.h

@@ -22,34 +22,33 @@ struct CriteriaCurrentIterationData {
  * struct that hold current Benders iteration
   */
 struct CurrentIterationData {
-  double subproblems_walltime;
-  double subproblems_cputime;
-  double subproblems_cumulative_cputime;
-  double timer_master;
-  double lb;
-  double ub;
-  double best_ub;
-  int deletedcut;
-  int it;
-  double iteration_time;
-  bool stop;
-  double overall_subpb_cost_under_approx;
+  double subproblems_walltime = 0;
+  double subproblems_cputime = 0;
+  double subproblems_cumulative_cputime = 0;
+  double timer_master = 0;
+  double lb = -1e20;
+  double ub = +1e20;
+  double best_ub = +1e20;
+  int it = 0;
+  double iteration_time = 0;
+  bool stop = false;
+  double overall_subpb_cost_under_approx = 0;
   std::vector<double> single_subpb_costs_under_approx;
   double subproblem_cost;
   double invest_cost;
-  int best_it;
+  int best_it = 0;
   Point x_in;
   Point x_out;
   Point x_cut;
   Point min_invest;
   Point max_invest;
-  int nsubproblem;
-  int master_status;
-  double benders_time;
-  StoppingCriterion stopping_criterion;
-  bool is_in_initial_relaxation;
-  int number_of_subproblem_solved;
-  int cumulative_number_of_subproblem_solved;
+  int nsubproblem = 0;
+  int master_status = SOLVER_STATUS::UNKNOWN;
+  double benders_time = 0;
+  StoppingCriterion stopping_criterion = StoppingCriterion::empty;
+  bool is_in_initial_relaxation = false;
+  int number_of_subproblem_solved = 0;
+  int cumulative_number_of_subproblem_solved = 0;
   int min_simplexiter;
   int max_simplexiter;
   // ugly

src/cpp/benders/benders_core/include/antares-xpansion/benders/benders_core/ProblemFormat.h

@@ -0,0 +1,29 @@
+#pragma once
+
+#include <stdexcept>
+#include <string>
+enum class ProblemsFormat { MPS_FILE, SAVED_FILE };
+
+inline ProblemsFormat problemsFormatFromString(const std::string &str) {
+  if (str == "MPS") {
+    return ProblemsFormat::MPS_FILE;
+  } else if (str == "SAVED") {
+    return ProblemsFormat::SAVED_FILE;
+  } else {
+    throw std::runtime_error("Unknown ProblemsFormat: " + str);
+  }
+}
+
+inline std::ostream &operator<<(std::ostream &stream, ProblemsFormat const &rhs) {
+  switch (rhs) {
+    case ProblemsFormat::MPS_FILE:
+      stream << "MPS_FILE";
+      break;
+    case ProblemsFormat::SAVED_FILE:
+      stream << "SAVED_FILE";
+      break;
+    default:
+      stream << "Unknown";
+  }
+  return stream;
+}

src/cpp/benders/benders_core/include/antares-xpansion/benders/benders_core/SimulationOptions.hxx

@@ -38,6 +38,9 @@ BENDERS_OPTIONS_MACRO(SLAVE_WEIGHT_VALUE, double, 1, asDouble())
 // Name of the master problem file, if different from 'master'
 BENDERS_OPTIONS_MACRO(MASTER_NAME, std::string, "master", asString())
 
+//Format of the problems
+BENDERS_OPTIONS_MACRO(PROBLEMS_FORMAT, ProblemsFormat, ProblemsFormat::MPS_FILE, as<ProblemsFormat>())
+
 // Number of slaves to use to solve the problem
 BENDERS_OPTIONS_MACRO(STRUCTURE_FILE, std::string, "structure.txt", asString())
 

src/cpp/benders/benders_core/include/antares-xpansion/benders/benders_core/Worker.h

@@ -70,4 +70,8 @@ class Worker {
   bool _is_master = false;
 
   Logger logger_;
+
+ private:
+  void read_prob(SolverAbstract * problem,
+                 const std::filesystem::path &path) const;
 };

src/cpp/benders/benders_core/include/antares-xpansion/benders/benders_core/common.h

@@ -23,6 +23,8 @@
 #include <tuple>
 #include <vector>
 
+#include "ProblemFormat.h"
+
 enum class MasterFormulation { INTEGER, RELAXED };
 enum class SOLVER { BENDERS, OUTER_LOOP, MERGE_MPS };
 
@@ -139,13 +141,15 @@ const std::string SUBPROBLEM_WEIGHT_CST_STR("CONSTANT");
 const std::string SUBPROBLEM_WEIGHT_UNIFORM_CST_STR("UNIFORM");
 const std::string WEIGHT_SUM_CST_STR("WEIGHT_SUM");
 const std::string MPS_SUFFIX = ".mps";
+const std::string SAVE_SUFFIX = ".svf";
 
 struct BaseOptions {
   std::string OUTPUTROOT;
   std::string INPUTROOT;
   std::string STRUCTURE_FILE;
   std::string LAST_ITERATION_JSON_FILE;
   std::string MASTER_NAME;
+  ProblemsFormat PROBLEMS_FORMAT = ProblemsFormat::MPS_FILE;
   std::string SOLVER_NAME;
   std::string SLAVE_WEIGHT;
   std::string AREA_FILE;

src/cpp/lpnamer/helper/ProblemGenerationLogger.cpp

@@ -121,7 +121,7 @@ const std::string& ProblemGenerationLogger::getContext() const {
   return context_;
 }
 
-ProblemGenerationLoggerSharedPointer BuildLogger(
+std::shared_ptr<ProblemGenerationLogger> BuildLogger(
     const std::filesystem::path& log_file_path, std::ostream& stream,
     const std::string& context) {
   auto logFile = std::make_shared<ProblemGenerationFileLogger>(log_file_path);

src/cpp/lpnamer/helper/include/antares-xpansion/lpnamer/helper/ProblemGenerationLogger.h

@@ -70,10 +70,6 @@ class ProblemGenerationOstreamLogger : public ProblemGenerationILogger {
   std::ostream& GetOstreamObject() override;
 };
 
-class ProblemGenerationLogger;
-using ProblemGenerationLoggerSharedPointer =
-    std::shared_ptr<ProblemGenerationLogger>;
-
 class ProblemGenerationLogger : public ILoggerXpansion {
  private:
   LogUtils::LOGLEVEL log_level_;
@@ -103,7 +99,7 @@ class ProblemGenerationLogger : public ILoggerXpansion {
 
   ProblemGenerationLogger& operator<<(std::ostream& (*f)(std::ostream&));
   ProblemGenerationLogger& operator<<(
-      const ProblemGenerationLoggerSharedPointer logger) {
+      const std::shared_ptr<ProblemGenerationLogger> logger) {
     return (*logger);
   }
   ProblemGenerationLogger& operator<<(const LogUtils::LOGLEVEL log_level);
@@ -125,7 +121,7 @@ ProblemGenerationLogger& ProblemGenerationLogger::operator<<(T const& t) {
   }
   return *this;
 }
-ProblemGenerationLoggerSharedPointer BuildLogger(
+std::shared_ptr<ProblemGenerationLogger> BuildLogger(
     const std::filesystem::path& log_file_path, std::ostream& stream,
     const std::string& context);
 

src/cpp/lpnamer/input_reader/AdditionalConstraintsReader.cpp

@@ -105,7 +105,7 @@ void AdditionalConstraintsReader::processEntryLine() {
 
 AdditionalConstraintsReader::AdditionalConstraintsReader(
     std::string const& constraints_file_path,
-    ProblemGenerationLog::ProblemGenerationLoggerSharedPointer logger)
+    std::shared_ptr<ProblemGenerationLog::ProblemGenerationLogger> logger)
     : logger_(logger) {
   std::ifstream file_l(constraints_file_path);
 

src/cpp/lpnamer/input_reader/CandidatesINIReader.cpp

@@ -12,7 +12,7 @@
 CandidatesINIReader::CandidatesINIReader(
     const std::filesystem::path &antaresIntercoFile,
     const std::filesystem::path &areaFile,
-    ProblemGenerationLog::ProblemGenerationLoggerSharedPointer logger)
+    std::shared_ptr<ProblemGenerationLog::ProblemGenerationLogger> logger)
     : logger_(logger) {
   _intercoFileData = ReadAntaresIntercoFile(antaresIntercoFile);
 

src/cpp/lpnamer/input_reader/GeneralDataReader.cpp

@@ -36,7 +36,7 @@ class IniReaderUtils {
 };
 GeneralDataIniReader::GeneralDataIniReader(
     const std::filesystem::path& file_path,
-    ProblemGenerationLog::ProblemGenerationLoggerSharedPointer logger)
+    std::shared_ptr<ProblemGenerationLog::ProblemGenerationLogger> logger)
     : logger_(logger) {
   if (file_path.empty() || !std::filesystem::exists(file_path)) {
     std::ostringstream msg;

src/cpp/lpnamer/input_reader/VariableFileReader.cpp

@@ -20,7 +20,7 @@ void updateMapColumn(const std::vector<ActiveLink>& links, int link_id,
 VariableFileReader::VariableFileReader(
     const std::filesystem::path& fileName, const std::vector<ActiveLink>& links,
     const VariableFileReadNameConfiguration& variable_name_config,
-    ProblemGenerationLog::ProblemGenerationLoggerSharedPointer logger)
+    std::shared_ptr<ProblemGenerationLog::ProblemGenerationLogger> logger)
     : logger_(logger) {
   std::string line;
   std::ifstream file(fileName.c_str());
@@ -38,7 +38,7 @@ VariableFileReader::VariableFileReader(
     std::istringstream& fileInIStringStream,
     const std::vector<ActiveLink>& links,
     const VariableFileReadNameConfiguration& variable_name_config,
-    ProblemGenerationLog::ProblemGenerationLoggerSharedPointer logger)
+    std::shared_ptr<ProblemGenerationLog::ProblemGenerationLogger> logger)
     : logger_(logger) {
   ReadVarsFromStream(fileInIStringStream, links, variable_name_config);
 }