Adding an option to load chronics episode by episode when using FromMultiEpisodeData

[EBoguslawski]

Problem description

Hello,

In my code, I create a deterministic environment from previously generated episodes (with a runner) thanks to the FromMultiEpisodeData class.

On my computer, creating such an environment with one year of data on the l2rpn_idf_2023 environment takes 2:36 minutes. It means that I would need around 41 minutes to load all the chronics (16 years).

You can do the test on your computer with the code below, which prints the duration of the environment creation (for one year).

Would it be possible to code an option to load the chronicles as you go along, rather than all at once at the creation?

How to reproduce

import os
import grid2op
from grid2op.Runner import Runner
from grid2op.Agent import RecoPowerlineAgent
from lightsim2grid import LightSimBackend
from grid2op.Chronics import FromMultiEpisodeData
from grid2op.Opponent import FromEpisodeDataOpponent
from grid2op.Episode import EpisodeData
import time

env_name = "l2rpn_idf_2023"
path_save = "./RecoPowerlineAgent_episodes"

## 1. Generating episodes (comment if it is already done)

env = grid2op.make(env_name, backend=LightSimBackend())
param = env.parameters
param.NO_OVERFLOW_DISCONNECTION = True
env.change_parameters(param)
env.reset()

# In our example, we evaluate only the first year
n_chronics = len(os.listdir(env.chronics_handler.path))
n_weeks = n_chronics // 16 # because there is 16 years in our dataset
episode_id = [k*16 for k in range(n_weeks)]
n_episode = len(episode_id)

runner = Runner(**env.get_params_for_runner(),
                agentClass=RecoPowerlineAgent)

res = runner.run(nb_episode=n_episode, pbar=False,
                        episode_id=episode_id, 
                        path_save=path_save,
                        )

## 2. Creating a new environment from generated episodes

li_episode = EpisodeData.list_episode(path_save)
# li_episode = [el for el in li_episode if el[1][-2:] in ["_0"]] # you can filter episodes of year 0 with this line if you evaluated other years

print("Env creation. Loading...")
start = time.time()
env_from_episodes = grid2op.make(env_name,
                   chronics_class=FromMultiEpisodeData,
                   data_feeding_kwargs={"li_ep_data": li_episode},
                   opponent_class=FromEpisodeDataOpponent,
                   backend = LightSimBackend(),
              )
end = time.time()
duration_in_min = (end - start) / 60
print(f"Env created ! It lasted {duration_in_min:.2f} minutes.")