MuJoCo compilation issues

[dennisushi]

Whenever I run the policy training script (eg bash scripts/train_policy.sh diffusion_image metaworld_push_image 0112 0 0 but also any combination) I get mujoco compilation errors. Have you encountered this?

Compiling /home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx because it changed.
[1/1] Cythonizing /home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx
performance hint: /home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx:67:5: Exception check on 'c_warning_callback' will always require the GIL to be acquired.
Possible solutions:
	1. Declare 'c_warning_callback' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on 'c_warning_callback' to allow an error code to be returned.
performance hint: /home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx:104:5: Exception check on 'c_error_callback' will always require the GIL to be acquired.
Possible solutions:
	1. Declare 'c_error_callback' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
	2. Use an 'int' return type on 'c_error_callback' to allow an error code to be returned.

Error compiling Cython file:
------------------------------------------------------------
...
    See c_warning_callback, which is the C wrapper to the user defined function
    '''
    global py_warning_callback
    global mju_user_warning
    py_warning_callback = warn
    mju_user_warning = c_warning_callback
                       ^
------------------------------------------------------------

/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx:92:23: Cannot assign type 'void (const char *) except * nogil' to 'void (*)(const char *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to the type of 'c_warning_callback'.

Error compiling Cython file:
------------------------------------------------------------
...
    See c_warning_callback, which is the C wrapper to the user defined function
    '''
    global py_error_callback
    global mju_user_error
    py_error_callback = err_callback
    mju_user_error = c_error_callback
                     ^
------------------------------------------------------------

/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx:127:21: Cannot assign type 'void (const char *) except * nogil' to 'void (*)(const char *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to the type of 'c_error_callback'.
Error executing job with overrides: ['task=metaworld_push_image', 'training.debug=False', 'training.seed=0', 'training.device=cuda:0', 'exp_name=metaworld_push_image-diffusion_image-0112', 'logging.mode=online', 'checkpoint.save_ckpt=False']
Traceback (most recent call last):
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 639, in _locate
    obj = getattr(obj, part)
AttributeError: module 'diffusion_policy_3d.env_runner' has no attribute 'metaworld_image_runner'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 645, in _locate
    obj = import_module(mod)
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/env_runner/metaworld_image_runner.py", line 6, in <module>
    from diffusion_policy_3d.env import MetaWorldEnv
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/env/__init__.py", line 2, in <module>
    from .adroit import AdroitEnv
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/env/adroit/__init__.py", line 1, in <module>
    from .adroit import AdroitEnv
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/env/adroit/adroit.py", line 12, in <module>
    import mj_envs
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/rrl-dependencies/mj_envs/mj_envs/__init__.py", line 1, in <module>
    import  mj_envs.hand_manipulation_suite 
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/rrl-dependencies/mj_envs/mj_envs/hand_manipulation_suite/__init__.py", line 2, in <module>
    from mjrl.envs.mujoco_env import MujocoEnv
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/rrl-dependencies/mjrl/mjrl/__init__.py", line 1, in <module>
    import mjrl.envs
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/rrl-dependencies/mjrl/mjrl/envs/__init__.py", line 31, in <module>
    from mjrl.envs.mujoco_env import MujocoEnv
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/rrl-dependencies/mjrl/mjrl/envs/mujoco_env.py", line 12, in <module>
    import mujoco_py
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/__init__.py", line 2, in <module>
    from mujoco_py.builder import cymj, ignore_mujoco_warnings, functions, MujocoException
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/builder.py", line 504, in <module>
    cymj = load_cython_ext(mujoco_path)
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/builder.py", line 110, in load_cython_ext
    cext_so_path = builder.build()
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/builder.py", line 226, in build
    built_so_file_path = self._build_impl()
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/builder.py", line 296, in _build_impl
    so_file_path = super()._build_impl()
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/builder.py", line 239, in _build_impl
    dist.ext_modules = cythonize([self.extension])
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1154, in cythonize
    cythonize_one(*args)
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 1321, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: /home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 134, in _resolve_target
    target = _locate(target)
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 653, in _locate
    raise ImportError(
ImportError: Error loading 'diffusion_policy_3d.env_runner.metaworld_image_runner.MetaworldImageRunner':
CompileError(None, '/home/dennisushi/repos/3dp/3D-Diffusion-Policy/third_party/mujoco-py-2.1.2.14/mujoco_py/cymj.pyx')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 45, in <module>
    main()
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "train.py", line 41, in main
    workspace.run()
  File "/home/dennisushi/repos/3dp/3D-Diffusion-Policy/3D-Diffusion-Policy/diffusion_policy_3d/workspace/train_diffusion_unet_hybrid_workspace.py", line 111, in run
    env_runner = hydra.utils.instantiate(
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 222, in instantiate
    return instantiate_node(
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 325, in instantiate_node
    _target_ = _resolve_target(node.get(_Keys.TARGET), full_key)
  File "/home/dennisushi/micromamba_envs/envs/diffusion_policy_v2/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 139, in _resolve_target
    raise InstantiationException(msg) from e
hydra.errors.InstantiationException: Error locating target 'diffusion_policy_3d.env_runner.metaworld_image_runner.MetaworldImageRunner', see chained exception above.
full_key: task.env_runner

[dennisushi]

Soloved by downgrading cython link

[YanjieZe]

Nice!