GPU Based Offscreen Rendering

[hukz18]

How can I turn on gpu based offscreen rendering?
I use the follow code snippet to test offscreen rendering but their's no gpu activity on device 0:

import os
import time
from gym import spaces
import robosuite as suite

os.environ['MUJOCO_GL'] = 'egl'
if __name__ == '__main__':
    env = suite.make(
        env_name='Wipe',
        robots="Panda",
        has_renderer=False,
        has_offscreen_renderer=True,
        use_camera_obs=True,
        camera_heights=84,
        camera_widths=84,
        camera_names='sideview',
        render_gpu_device_id=0,
        )
    obs, done = env.reset(), False
    low, high = env.action_spec
    action_space = spaces.Box(low=low, high=high)
    steps, time_stamp = 0, time.time()
    while True:
        while not done:
            obs, reward, done, info = env.step(action_space.sample())
            steps += 1
        obs, done = env.reset(), False
        print(f'FPS: {steps / (time.time() - time_stamp)}')
        steps, time_stamp = 0, time.time()

Also, the rendering fps is the same (around 80 fpds) with or without the camera_heights and camera_widths specification. I think setting this values to (84, 84) could improve the render speed.

I've heard something about gpu rendering on mujoco_py here, however I've check that gpu rendering for mujoco_py is enabled for me. What maybe wrong here?

[hukz18]

Hi, is there any update on this issue? I'd like to provide more information needed to reproduce the problem.

[snasiriany]

Hi, I ran this snippet of code but I see activity under gpu0. Also, I didn't need the os.environ['MUJOCO_GL'] = 'egl' line for this to work. Which branch of robosuite are you on? Can you try with the latest commit on the master branch and let me know if you still see no activity under nvidia-smi?

[hukz18]

Thank you for your help! I double checked this code doesn't work for me on GPU.. I'm using robosuite 1.4.0 with mujoco 2.3.2 and mujoco_py 2.1.2.14.

One thing I noticed is that if I change the while True loop to a finite one like for _ in range(3), the program throw an error when exiting:

Exception ignored in: <function MjRenderContext.__del__ at 0x7f3d42cb5b40>
Traceback (most recent call last):
  File ".../site-packages/robosuite/utils/binding_utils.py", line 199, in __del__
  File ".../site-packages/robosuite/renderers/context/egl_context.py", line 149, in free
  File ".../site-packages/OpenGL/error.py", line 228, in glCheckError
OpenGL.error.GLError: GLError(
        err = 12289,
        baseOperation = eglMakeCurrent,
        cArguments = (
                <OpenGL._opaque.EGLDisplay_pointer object at 0x7f3d3a1d1d40>,
                <OpenGL._opaque.EGLSurface_pointer object at 0x7f3d42c54440>,
                <OpenGL._opaque.EGLSurface_pointer object at 0x7f3d42c54440>,
                <OpenGL._opaque.EGLContext_pointer object at 0x7f3d42efbb40>,
        ),
        result = 0
)
Exception ignored in: <function EGLGLContext.__del__ at 0x7f3d42cb5990>
Traceback (most recent call last):
  File ".../site-packages/robosuite/renderers/context/egl_context.py", line 155, in __del__
  File ".../site-packages/robosuite/renderers/context/egl_context.py", line 149, in free
  File ".../python3.10/site-packages/OpenGL/error.py", line 228, in glCheckError
OpenGL.error.GLError: GLError(
        err = 12289,
        baseOperation = eglMakeCurrent,
        cArguments = (
                <OpenGL._opaque.EGLDisplay_pointer object at 0x7f3d3a1d1d40>,
                <OpenGL._opaque.EGLSurface_pointer object at 0x7f3d42c54440>,
                <OpenGL._opaque.EGLSurface_pointer object at 0x7f3d42c54440>,
                <OpenGL._opaque.EGLContext_pointer object at 0x7f3d42efbb40>,
        ),
        result = 0
)

I've change the path of my conda environment to "..." above. I tried to verify my EGL installation using this repo, It shows the following message:

Loaded EGL 1.5 after reload.
GL_VENDOR=NVIDIA Corporation
GL_RENDERER=NVIDIA A40/PCIe/SSE2
GL_VERSION=4.6.0 NVIDIA 515.65.01
GL_SHADING_LANGUAGE_VERSION=4.60 NVIDIA

I reckon that's to say my EGL installation is successful, what might be wrong here?

[snasiriany]

regarding the error message you see when exiting: this is expected. we see this on our end too, but I think it's harmless.

I reckon that's to say my EGL installation is successful, what might be wrong here?

I'm not too sure in this case. Are you able to verify other programs successfully use the gpu? eg. if you run a pytorch job do you see activity for that job on nvidia-smi?

Also, have you set CUDA_VISIBLE_DEVICES or MUJOCO_EGL_DEVICE_ID to anything? you can check with for example echo $CUDA_VISIBLE_DEVICES

[hukz18]

Yes, sorry for the delay but after setting the MUJOCO_EGL_DEVICE_ID the GPU rendering seems to be enabled, the render speed increases to about 90 fps, if that's the expected speed then there seems no further issue.