Can't get attribute 'PlayableAction_l2rpn_case14_sandbox_l2rpn_case14_sandbox' on <module 'grid2op.Space.GridObjects'

[AvisP]

Environment

• Grid2op version: 1.9.3
• System: osx
• Ray version: 2.6.1

Hi, I was trying to simplify the training script of Grid2Op with RLLIB and wrote the following code based on the example specified here on ray website or custom environments. But I am getting the following error. Not sure if the issue is happening because I am not using an agent derived from the BaseAgent class or it is something else. Any guidance in resolving would be appreciated.

Exception has occurred: RaySystemError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
System error: Can't get attribute 'PlayableAction_l2rpn_case14_sandbox_l2rpn_case14_sandbox' on <module 'grid2op.Space.GridObjects' from '/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/grid2op/Space/GridObjects.py'>
traceback: Traceback (most recent call last):
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 387, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 268, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 223, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 211, in _deserialize_pickle5_data
    obj = pickle.loads(in_band, buffers=buffers)
AttributeError: Can't get attribute 'PlayableAction_l2rpn_case14_sandbox_l2rpn_case14_sandbox' on <module 'grid2op.Space.GridObjects' from '/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/grid2op/Space/GridObjects.py'>
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/worker.py", line 2495, in get
    raise value
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/utils/actor_manager.py", line 481, in __fetch_result
    result = ray.get(r)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/evaluation/worker_set.py", line 76, in handle_remote_call_result_errors
    raise r.get()
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/evaluation/worker_set.py", line 680, in foreach_worker
    handle_remote_call_result_errors(remote_results, self._ignore_worker_failures)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/evaluation/worker_set.py", line 267, in _get_spaces_from_remote_worker
    remote_spaces = self.foreach_worker(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/evaluation/worker_set.py", line 241, in _setup
    spaces = self._get_spaces_from_remote_worker()
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/evaluation/worker_set.py", line 157, in __init__
    self._setup(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 639, in setup
    self.workers = WorkerSet(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 169, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 517, in __init__
    super().__init__(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm_config.py", line 1068, in build
    return algo_class(
  File "/Users/paula/Desktop/Projects/RL Practice/RLLIB_Practice3/custom_RLLIB_Grid2op.py", line 69, in <module>
    algo = config.build()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
ray.exceptions.RaySystemError: System error: Can't get attribute 'PlayableAction_l2rpn_case14_sandbox_l2rpn_case14_sandbox' on <module 'grid2op.Space.GridObjects' from '/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/grid2op/Space/GridObjects.py'>
traceback: Traceback (most recent call last):
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 387, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 268, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 223, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/_private/serialization.py", line 211, in _deserialize_pickle5_data
    obj = pickle.loads(in_band, buffers=buffers)
AttributeError: Can't get attribute 'PlayableAction_l2rpn_case14_sandbox_l2rpn_case14_sandbox' on <module 'grid2op.Space.GridObjects' from '/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/grid2op/Space/GridObjects.py'>

The code that I am using is specified below

from Env_RLLIB import Env_RLLIB
import ray
from ray.rllib.algorithms.ppo import PPOConfig
import grid2op
from lightsim2grid.lightSimBackend import LightSimBackend
from grid2op.Reward import L2RPNReward
import re
import numpy as np

env_name = "l2rpn_case14_sandbox"
custom_path_dir = "/Users/paula/data_grid2op/"
obs_attr_to_keep = ["day_of_week", "hour_of_day", "minute_of_hour", "month", "rho", "line_status", 
    "load_p", "load_q", "load_v", "load_theta", "gen_p", "gen_q", "gen_v", "gen_theta", 
    "p_ex", "p_or", "q_ex", "q_or", "v_ex", "v_or", "theta_ex", "theta_or"]
act_attr_to_keep = ["set_bus"]
backend_class=LightSimBackend
reward_class=L2RPNReward
normalise_obs = True
normalise_act = False
tensorboard_enabled = True

if custom_path_dir is not None:
    env = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+env_name,
                    reward_class=reward_class,
                    backend=backend_class())
else:
    env = grid2op.make(env_name,
                    reward_class=reward_class,
                    backend=backend_class())

obs_space_kwargs={'divide':{"load_p": 100., "load_q": 100., "gen_p":100., "gen_q":100., "p_ex":100., "p_or":100., "q_ex":100., "q_or":100.,
                            "gen_v": np.array([142.0, 142.0, 22.0, 22.0, 13.2, 142.0], dtype=np.float32),
                            "load_v": np.array([142.0, 142.0, 142.0, 142.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0], dtype=np.float32),
                            "v_ex": np.array([142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 22.0, 22.0, 22.0, 22.0, 22.0,  22.0, 22.0, 
                                                22.0, 13.2,  22.0,  22.0,  13.2,  13.2], dtype=np.float32),
                            "v_or": np.array([142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 22.0, 22.0, 22.0, 22.0, 22.0,  22.0, 22.0, 
                                                22.0, 13.2,  22.0,  22.0,  13.2,  13.2], dtype=np.float32),
                            "gen_theta": 180.0/np.pi, "load_theta": 180.0/np.pi, "theta_ex": 180.0/np.pi, "theta_or": 180.0/np.pi}}

action_list = [env.action_space({})]
action_list.extend(env.action_space.get_all_unitary_line_set(env.action_space))
action_list = action_list[:40]
action_list.extend(env.action_space.get_all_unitary_topologies_set(env.action_space))
act_space_kwargs={'action_list':action_list}

env_config = {"env_name":env_name,
                "custom_path_dir":custom_path_dir,
                "reward_class":reward_class,
                "obs_attr_to_keep":obs_attr_to_keep,
                "act_attr_to_keep":act_attr_to_keep,
                "obs_space_kwargs":obs_space_kwargs,
                "act_space_kwargs":act_space_kwargs,
                "backend_class":backend_class,
                "reward_class":reward_class,
                "normalise_obs":normalise_obs,
                "normalise_act":normalise_act,
                "data_feeding_kwargs": {'filter_func': lambda x: re.match(".*0*", x) is not None}  #use one over 100 chronics to train (for speed)
}

ray.init()

config = PPOConfig().environment(Env_RLLIB, env_config=env_config)
algo = config.build()

for _ in range(3):
    print(algo.train())

algo.stop()
ray.shutdown()

The Env_RLLIB class is

import grid2op
from grid2op.gym_compat import BoxGymObsSpace, DiscreteActSpace, BoxGymActSpace
import gymnasium as gym
from grid2op.gym_compat import GymEnv
from grid2op.Chronics import MultifolderWithCache
from grid2op.Reward import BaseReward, L2RPNReward

class Env_RLLIB(gym.Env):
    def __init__(self, env_config):
        
        self.env_config = env_config
        if not "env_name" in self.env_config:
            raise RuntimeError("The configuration for RLLIB should provide the env name")

        nm_env = self.env_config["env_name"]
        del self.env_config["env_name"]

        custom_path_dir = self.env_config["custom_path_dir"]
        del self.env_config["custom_path_dir"]
        
        obs_attr_to_keep = None
        if "obs_attr_to_keep" in self.env_config:
            obs_attr_to_keep = self.env_config["obs_attr_to_keep"]
            del  self.env_config["obs_attr_to_keep"]
        act_attr_to_keep = None
        if "act_attr_to_keep" in self.env_config:  
            act_attr_to_keep = self.env_config["act_attr_to_keep"]
            del  self.env_config["act_attr_to_keep"]
        if "backend_class" in self.env_config:
            backend_kwargs = {}
            if "backend_kwargs" in self.env_config:
                backend_kwargs = self.env_config["backend_kwargs"]
                del self.env_config["backend_kwargs"]
            backend = self.env_config["backend_class"](**backend_kwargs)
            del  self.env_config["backend_class"]
        reward_class = L2RPNReward
        if "reward_class" in env_config:
            reward_class = env_config["reward_class"]
            del env_config["reward_class"]
        if "obs_space_kwargs" in self.env_config:
            obs_space_kwargs = self.env_config["obs_space_kwargs"]
            del self.env_config["obs_space_kwargs"]
        else:
            obs_space_kwargs = {}
        if "act_space_kwargs" in self.env_config:
            act_space_kwargs = self.env_config["act_space_kwargs"]
            del self.env_config["act_space_kwargs"]
        else:
            act_space_kwargs = {}
        if "normalise_obs" in self.env_config:
            normalise_obs = self.env_config["normalise_obs"]
            del self.env_config["normalise_obs"]
        else:
            normalise_obs=False

        if "normalise_act" in self.env_config:
            normalise_act = self.env_config["normalise_act"]
            del self.env_config["normalise_act"]
        else:
            normalise_act=False
        
        if "data_feeding_kwargs" in self.env_config:
            data_feeding_kwargs = self.env_config["data_feeding_kwargs"]
            print(data_feeding_kwargs)

        self.env_glop = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+nm_env,
                                    reward_class=reward_class,
                                    backend=backend,
                                    chronics_class=MultifolderWithCache,
                                    **env_config)
         # 1. create the grid2op environment
        self.env_glop.chronics_handler.real_data.set_filter(data_feeding_kwargs['filter_func'])
        self.env_glop.chronics_handler.real_data.reset()

        # 2. create the gym environment
        self.env_gym = GymEnv(self.env_glop)
        obs_gym, info = self.env_gym.reset(seed=None, options={})

        # 3. customize observation and action space
        if obs_attr_to_keep is not None:
            self.env_gym.observation_space.close()
            self.env_gym.observation_space =  BoxGymObsSpace(self.env_glop.observation_space,
                                                             attr_to_keep=obs_attr_to_keep,
                                                             **obs_space_kwargs)
        
        if act_attr_to_keep is not None:    
            self.env_gym.action_space.close()
            self.env_gym.action_space = DiscreteActSpace(self.env_glop.action_space,
                                                    #    attr_to_keep=act_attr_to_keep,
                                                        **act_space_kwargs)

        if normalise_obs:
            # if save_path is not None:
            #     with open(os.path.join(my_path, ".normalize_obs"), encoding="utf-8", 
            #             mode="w") as f:
            #         f.write("I have encoded the observation space !\n DO NOT MODIFY !")
            for attr_nm in obs_attr_to_keep:
                if (("divide" in obs_space_kwargs and attr_nm in obs_space_kwargs["divide"]) or 
                    ("subtract" in obs_space_kwargs and attr_nm in obs_space_kwargs["subtract"]) 
                ):
                    # attribute is scaled elsewhere
                    continue
                self.env_gym.observation_space.normalize_attr(attr_nm)

        if normalise_act:
            # if save_path is not None:
            #     with open(os.path.join(my_path, ".normalize_act"), encoding="utf-8", 
            #             mode="w") as f:
            #         f.write("I have encoded the action space !\n DO NOT MODIFY !")
            for attr_nm in act_attr_to_keep:
                if (("multiply" in act_space_kwargs and attr_nm in act_space_kwargs["multiply"]) or 
                    ("add" in act_space_kwargs and attr_nm in act_space_kwargs["add"]) 
                ):
                    # attribute is scaled elsewhere
                    continue
                self.env_gym.action_space.normalize_attr(attr_nm)

        # 4. specific to rllib
        self.action_space = self.env_gym.action_space
        self.observation_space = self.env_gym.observation_space

    def reset(self, *, seed=None, options=None):
        obs, info = self.env_gym.reset(seed=seed, options=options)
        return obs, info

    def step(self, action):
        obs, reward, terminated, truncated, info = self.env_gym.step(action)
        return obs, reward, terminated, truncated, info

[BDonnot]

Hi,

Please follow the readme (https://github.com/rte-france/Grid2Op#known-issues) or the doc https://grid2op.readthedocs.io/en/latest/gym.html#python-complains-about-pickle for this particular issue.

Problem is known (unfortunately) but i can't find a proper solution at the moment (but the experimental_read_from_local_dir seems to work correctly)

[AvisP]

Hi,

Thanks for letting me know about the issue and pointing me to the solution. I tried modifying the two functions based on the solution provided to the following but now I am getting a different error. Essentially instead of initializing the environment again within Env_RLLIB I tried to pass it as a configuration parameter from the script. The updated files are

from Env_RLLIB_mod4_custom_script import Env_RLLIB
import ray
from ray.rllib.algorithms.ppo import PPOConfig
import grid2op
from lightsim2grid.lightSimBackend import LightSimBackend
# from Rewards import CustomReward, CustomReward_AZPaper
from grid2op.Reward import L2RPNReward
import re
import numpy as np

env_name = "l2rpn_case14_sandbox"
custom_path_dir = "/Users/paula/data_grid2op/"
obs_attr_to_keep = ["day_of_week", "hour_of_day", "minute_of_hour", "month", "rho", "line_status", 
    "load_p", "load_q", "load_v", "load_theta", "gen_p", "gen_q", "gen_v", "gen_theta", 
    "p_ex", "p_or", "q_ex", "q_or", "v_ex", "v_or", "theta_ex", "theta_or"]
act_attr_to_keep = ["set_bus"]
backend_class=LightSimBackend
reward_class=L2RPNReward
normalise_obs = True
normalise_act = False
tensorboard_enabled = True

if custom_path_dir is not None:
    # env = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+env_name,
    env = grid2op.make(env_name,
                    reward_class=reward_class,
                    backend=backend_class())
else:
    env = grid2op.make(env_name,
                    reward_class=reward_class,
                    backend=backend_class())
    
env.generate_classes()
env = grid2op.make(env_name,
                    reward_class=reward_class,
                    backend=backend_class(),
                    experimental_read_from_local_dir=True)

obs_space_kwargs={'divide':{"load_p": 100., "load_q": 100., "gen_p":100., "gen_q":100., "p_ex":100., "p_or":100., "q_ex":100., "q_or":100.,
                            "gen_v": np.array([142.0, 142.0, 22.0, 22.0, 13.2, 142.0], dtype=np.float32),
                            "load_v": np.array([142.0, 142.0, 142.0, 142.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0, 22.0], dtype=np.float32),
                            "v_ex": np.array([142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 22.0, 22.0, 22.0, 22.0, 22.0,  22.0, 22.0, 
                                                22.0, 13.2,  22.0,  22.0,  13.2,  13.2], dtype=np.float32),
                            "v_or": np.array([142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 142.0, 22.0, 22.0, 22.0, 22.0, 22.0,  22.0, 22.0, 
                                                22.0, 13.2,  22.0,  22.0,  13.2,  13.2], dtype=np.float32),
                            "gen_theta": 180.0/np.pi, "load_theta": 180.0/np.pi, "theta_ex": 180.0/np.pi, "theta_or": 180.0/np.pi}}

action_list = [env.action_space({})]
action_list.extend(env.action_space.get_all_unitary_line_set(env.action_space))
action_list = action_list[:40]
# act_space_kwargs={'action_list':action_list[:40]}
# action_list = []
action_list.extend(env.action_space.get_all_unitary_topologies_set(env.action_space))
act_space_kwargs={'action_list':action_list}

env_config = {"actual_env":env,
                "env_name":env_name,
                "custom_path_dir":custom_path_dir,
                "reward_class":reward_class,
                "obs_attr_to_keep":obs_attr_to_keep,
                "act_attr_to_keep":act_attr_to_keep,
                "obs_space_kwargs":obs_space_kwargs,
                "act_space_kwargs":act_space_kwargs,
                "backend_class":backend_class,
                "reward_class":reward_class,
                "normalise_obs":normalise_obs,
                "normalise_act":normalise_act,
                "data_feeding_kwargs": {'filter_func': lambda x: re.match(".*0*", x) is not None}  #use one over 100 chronics to train (for speed)
}

ray.init()

config = PPOConfig().environment(Env_RLLIB, env_config=env_config)
algo = config.build()

for _ in range(3):
    print(algo.train())

algo.stop()
ray.shutdown()

And the Env_RLLIB class is

import grid2op
from grid2op.gym_compat import BoxGymObsSpace, DiscreteActSpace, BoxGymActSpace
import gymnasium as gym
from grid2op.gym_compat import GymEnv
from grid2op.Chronics import MultifolderWithCache
from grid2op.Reward import BaseReward, L2RPNReward
from lightsim2grid.lightSimBackend import LightSimBackend

class Env_RLLIB(gym.Env):
    def __init__(self, env_config):
        
        self.env_config = env_config
        if not "env_name" in self.env_config:
            raise RuntimeError("The configuration for RLLIB should provide the env name")

        nm_env = self.env_config["env_name"]
        del self.env_config["env_name"]

        custom_path_dir = self.env_config["custom_path_dir"]
        del self.env_config["custom_path_dir"]
        
        # if "custom_path_dir" in self.env_config["custom_path_dir"]:
        #     custom_path_dir = self.env_config["custom_path_dir"]
        #     del self.env_config["custom_path_dir"]
        # else:
        #     custom_path_dir = False
        # print(custom_path_dir)
        obs_attr_to_keep = None
        if "obs_attr_to_keep" in self.env_config:
            obs_attr_to_keep = self.env_config["obs_attr_to_keep"]
            del  self.env_config["obs_attr_to_keep"]
        act_attr_to_keep = None
        if "act_attr_to_keep" in self.env_config:  
            act_attr_to_keep = self.env_config["act_attr_to_keep"]
            del  self.env_config["act_attr_to_keep"]
        if "backend_class" in self.env_config:
            backend_kwargs = {}
            if "backend_kwargs" in self.env_config:
                backend_kwargs = self.env_config["backend_kwargs"]
                del self.env_config["backend_kwargs"]
            backend = self.env_config["backend_class"](**backend_kwargs)
            del  self.env_config["backend_class"]
        reward_class = L2RPNReward
        if "reward_class" in env_config:
            reward_class = env_config["reward_class"]
            del env_config["reward_class"]
        if "obs_space_kwargs" in self.env_config:
            obs_space_kwargs = self.env_config["obs_space_kwargs"]
            del self.env_config["obs_space_kwargs"]
        else:
            obs_space_kwargs = {}
        if "act_space_kwargs" in self.env_config:
            act_space_kwargs = self.env_config["act_space_kwargs"]
            del self.env_config["act_space_kwargs"]
        else:
            act_space_kwargs = {}
        if "normalise_obs" in self.env_config:
            normalise_obs = self.env_config["normalise_obs"]
            del self.env_config["normalise_obs"]
        else:
            normalise_obs=False

        if "normalise_act" in self.env_config:
            normalise_act = self.env_config["normalise_act"]
            del self.env_config["normalise_act"]
        else:
            normalise_act=False
        
        if "data_feeding_kwargs" in self.env_config:
            data_feeding_kwargs = self.env_config["data_feeding_kwargs"]
            print(data_feeding_kwargs)

        self.env_glop = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+nm_env,
                                    reward_class=reward_class,
                                    backend=LightSimBackend(),
                                    chronics_class=MultifolderWithCache,
                                    **env_config)
        
        # # self.env_glop.generate_classes()
        # self.env_glop = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+nm_env,
        #             reward_class=reward_class,
        #             backend=LightSimBackend(),
        #             chronics_class=MultifolderWithCache,
        #             experimental_read_from_local_dir=True,
        #             **env_config)
        #  1. create the grid2op environment
        # if custom_path_dir:
        #     self.env_glop = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+nm_env,
        #                             # reward_class=reward_class,
        #                             backend=backend,
        #                             chronics_class=MultifolderWithCache,
        #                             **env_config)
        # else:
        #     self.env_glop = grid2op.make(nm_env,
        #                             # reward_class=reward_class,
        #                             backend=backend,
        #                             chronics_class=MultifolderWithCache,
        #                             **env_config)
        # self.env_glop.chronics_handler.real_data.set_filter(data_feeding_kwargs['filter_func'])
        self.env_glop.chronics_handler.real_data.reset()

        # 2. create the gym environment
        self.env_gym = GymEnv(self.env_glop)
        # self.env_gym.observation_space.close()
        # self.env_gym.action_space.close()
        obs_gym, info = self.env_gym.reset(seed=None, options={})

        # 3. customize observation and action space
        if obs_attr_to_keep is not None:
            self.env_gym.observation_space.close()
            self.env_gym.observation_space =  BoxGymObsSpace(self.env_glop.observation_space,
                                                             attr_to_keep=obs_attr_to_keep,
                                                             **obs_space_kwargs)
        
        if act_attr_to_keep is not None:    
            self.env_gym.action_space.close()
            self.env_gym.action_space = DiscreteActSpace(self.env_glop.action_space,
                                                    #    attr_to_keep=act_attr_to_keep,
                                                        **act_space_kwargs)

        if normalise_obs:
            # if save_path is not None:
            #     with open(os.path.join(my_path, ".normalize_obs"), encoding="utf-8", 
            #             mode="w") as f:
            #         f.write("I have encoded the observation space !\n DO NOT MODIFY !")
            for attr_nm in obs_attr_to_keep:
                if (("divide" in obs_space_kwargs and attr_nm in obs_space_kwargs["divide"]) or 
                    ("subtract" in obs_space_kwargs and attr_nm in obs_space_kwargs["subtract"]) 
                ):
                    # attribute is scaled elsewhere
                    continue
                self.env_gym.observation_space.normalize_attr(attr_nm)

        if normalise_act:
            # if save_path is not None:
            #     with open(os.path.join(my_path, ".normalize_act"), encoding="utf-8", 
            #             mode="w") as f:
            #         f.write("I have encoded the action space !\n DO NOT MODIFY !")
            for attr_nm in act_attr_to_keep:
                if (("multiply" in act_space_kwargs and attr_nm in act_space_kwargs["multiply"]) or 
                    ("add" in act_space_kwargs and attr_nm in act_space_kwargs["add"]) 
                ):
                    # attribute is scaled elsewhere
                    continue
                self.env_gym.action_space.normalize_attr(attr_nm)

        # 4. specific to rllib
        self.action_space = self.env_gym.action_space
        self.observation_space = self.env_gym.observation_space

    def reset(self, *, seed=None, options=None):
        obs, info = self.env_gym.reset(seed=seed, options=options)
        return obs, info

    def step(self, action):
        obs, reward, terminated, truncated, info = self.env_gym.step(action)
        return obs, reward, terminated, truncated, info

The error message I am getting is

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/paula/Desktop/Projects/RL Practice/RLLIB_Practice3/custom_RLLIB_Grid2op.py", line 74, in <module>
    algo = config.build()
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm_config.py", line 1068, in build
    return algo_class(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 517, in __init__
    super().__init__(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/tune/trainable/trainable.py", line 169, in __init__
    self.setup(copy.deepcopy(self.config))
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/algorithms/algorithm.py", line 639, in setup
    self.workers = WorkerSet(
  File "/Users/paula/Desktop/Projects/venvs/L2RPN_RLLIB_2_3/lib/python3.10/site-packages/ray/rllib/evaluation/worker_set.py", line 179, in __init__
    raise e.args[0].args[2]
grid2op.Exceptions.Grid2OpException.Grid2OpException: Grid2OpException "Impossible to initialize the powergrid, the powerflow diverge at iteration 0. Available information are: {'disc_lines': array([-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1,
       -1, -1, -1], dtype=int32), 'is_illegal': False, 'is_ambiguous': False, 'is_dispatching_illegal': False, 'is_illegal_reco': False, 'reason_alarm_illegal': None, 'reason_alert_illegal': None, 'opponent_attack_line': None, 'opponent_attack_sub': None, 'opponent_attack_duration': 0, 'exception': [RuntimeError('DataLine::fillYbusBranch: the line with id 0 is connected (or side) to a disconnected bus while being connected')], 'rewards': {}}"

[AvisP]

Alternatively, do you have any updated training script of running Grid2Ops with latest version of RLLIB or any plans to do so in near future, then it would solve this issue of mine. Thanks!

[BDonnot]

Hello

Well i cannot update everything every time there is a problem with an external librarie unfortunately. I would really much have the time to do so however. There is no plan as of today to make things work with ray "in the near future" as far as I know.

Maybe if you look at the l2rpn-baselines package there are some stuff to help you get started. You also have some examples here https://github.com/rte-france/Grid2Op/tree/dev_multiagent/examples/multi_agents if you like (in a multi agents context but you get the idea)

I can spot a few stuff however: please consider doing exactly as the documentation said. You need to define the environment in a first script:

import grid2op
env_name = "l2rpn_case14_sandbox"
custom_path_dir = "/Users/paula/data_grid2op/"
backend_class=LightSimBackend
reward_class=L2RPNReward

if custom_path_dir is not None:
    # env = grid2op.MakeEnv.make_from_dataset_path(custom_path_dir+env_name,
    env = grid2op.make(env_name,
                    reward_class=reward_class,
                    backend=backend_class())
else:
    env = grid2op.make(env_name,
                    reward_class=reward_class,
                    backend=backend_class())
    
env.generate_classes()

Run it

And then you can load your classes without calling "the env.generate_classe()"

Also another tips to make your issues clearer :

1. use ```python instead of simply ``` when copy pasting your script so that python coloring gets applied by github
2. remove the commented code if I spend 1 minute at reading at commented code it's 1 less minute I have to help you debug
3. try to make your reproducible code as small as possible (pro tips: it's in making that that you'll find in 90% of the case a solution to your problem)

Your problem seems to be an issue with grid2op and not with ray (your environment does not converge). Have you tried to simply create an instance of Env_RLLIB like this:

env_rllib = Env_RLLIB(your_config)

And see the issue ? It's often 2 or 3 orders of magnitude easier and faster to debug without "multi processing" involved.

[AvisP]

Hi,

Thanks for updating the script on l2rpnbaselines repo for PPO_RLLIB. Based on this I made a new script and that one worked now. I tried developing the script I shared above based on an older version of that repo. I wasn't aware that it is possible to call the PPO class directly (or any rllib algorithm) and just passing the configuration makes it work. As you can see I was getting the configuration from PPOConfig and then trying to build the PPO agent, based on how it is shown in rllib examples. This was resulting in slow start up time for training even for small 14 bus networks and wasn't running for 36 bus at all. But after doing it the way as shown in the l2rpnbaselines now it seems to be working for 36 bus with limited hardware resource system of 32 GB. ( Yes I was trying to load the whole set of chronics and was hoping the MultiFolderCache class would take care of loading it by sections to memory somehow, but that wasn't the case).

Although I think the train script can be improved further as we are creating the environment twice. First one just to convert the environment observation and action space into gym format and then pass into the RLLIBAgent class where the environment is built again through rllib library. If I understand correctly this takes more memory for two environments and just making one call should reduce it but not sure how to implement it.

There was a minor issue and I had to add the following line self.env_glop.chronics_handler.reset() after https://github.com/rte-france/l2rpn-baselines/blob/ba346d347c85ac70cef6e4c73e3e60edc839490c/l2rpn_baselines/PPO_RLLIB/env_rllib.py#L103 to make it work.

Thanks for the other suggestions, I will incorporate them next time to make it easier for you to debug

[BDonnot]

Thanks, can you put these issues in the appropriate github repo please ? If it's here i'll 100% forget about them (and right now I don't have much time to work on l2rpn-baselines :-/ )

[AvisP]

Sure I have created this issue on the l2rpn repo. Yes I understand take a look when possible

[BDonnot]

Thanks a lot i'll give it a try as soon as possible (which might take a while unfortunately :-/)