Issues in simulation

[Norweig1an]

Hi, thanks for your inspiring work!
During simulation, we encountered some issues:
The IP and ports were correctly set.
When we run python teleop.py robot=allegro_sim sim_env=True the scene in Meta Quest 3 froze after the hand model was loaded.
It seems that the server joint_angle_subscriber could not receive joint_angles (openteach/components/environment/allegro_hand_env.py, `take_action(self)').
We will appreciate your kind help.

[chiawenwang]

I also meet the same problem on Ubuntu20.04 !

[aadhithya14]

Thanks for using Openteach. Hi Could you print the screenshot of your terminal? Is the end effector position getting printed?

[chiawenwang]

Thanks for using Openteach. Hi Could you print the screenshot of your terminal? Is the end effector position getting printed?

Thank you for you replying. It just happens when I use the Bimanual, and it works on KinovaBot. Here is the content printed:

---

(openteach_isaac) wangjw@TRX40-AORUS-PRO:~/Open-Teach$ python3 teleop.py robot=allegro_sim sim_env=True

---

 Starting keypoint position transform component

---

---

 Starting vr detector component

---

Importing module 'gym_37' (/home/wangjw/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/wangjw/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
Importing module 'gym_37' (/home/wangjw/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/wangjw/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json

---

 Starting hand 2D plotter component

---

PyTorch version 1.13.1+cu117
Device count 1
/home/wangjw/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/wangjw/.cache/torch_extensions/py37_cu117 as PyTorch extensions root...
Emitting ninja build file /home/wangjw/.cache/torch_extensions/py37_cu117/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
tcp://192.168.31.176:10005
tcp://192.168.31.176:11005
Trying to make simulation
PyTorch version 1.13.1+cu117
Device count 1
/home/wangjw/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/wangjw/.cache/torch_extensions/py37_cu117 as PyTorch extensions root...
Not connected to PVD
+++ Using GPU PhysX
Emitting ninja build file /home/wangjw/.cache/torch_extensions/py37_cu117/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: disabled
Loading extension module gymtorch...

---

 Starting allegro hand sim operator component

---

/home/wangjw/.conda/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link Base link (index: 0) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/wangjw/.conda/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_3.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/wangjw/.conda/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_7.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/wangjw/.conda/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_11.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/wangjw/.conda/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_15.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))

---

 Starting calibration component

---

Calibration file already exists. Do you want to create a new one? Press y for Yes else press EnterSimulation created
Loading asset 'allegro_hand_description/urdf/model_only_hand.urdf' from '/home/wangjw/Open-Teach/openteach/components/environment/assets/urdf/'

And when I press Enter , it shows:

Stopping the calibrator.
Closing the subscriber socket in 192.168.31.176:8089.
THUMB BOUNDS IN THE OPERATOR: [array([0.07147029, 0.10102608]), array([0.06931292, 0.06902901]), array([-0.00888068, 0.08024863]), array([-0.0074921, 0.0941279]), [-0.029687233564084578, -0.060893850578688846]]

---

 Starting Allegro_Sim control component

---

Start controlling the robot hand using the Oculus Headset.
Applying retargeted angles

---

The image shows the hands are frozon after python3 teleop.py robot=allegro_sim sim_env=True

[Norweig1an]

Thanks for using Openteach. Hi Could you print the screenshot of your terminal? Is the end effector position getting printed?

python teleop.py robot=allegro_sim sim_env=True
Importing module 'gym_37' (/home/rhos/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/rhos/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
/home/rhos/miniconda3/envs/openteach_isaac/lib/python3.7/site-packages/torch/utils/cpp_extension.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
PyTorch version 1.7.1+cu110
Device count 1
/home/rhos/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/rhos/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/rhos/.cache/torch_extensions/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
tcp://192.168.0.188:10005
tcp://192.168.0.188:11005
Trying to make simulation 0 0 0 True 11.0 10de:2786
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: disabled
Simulation created
Loading asset 'allegro_hand_description/urdf/model_only_hand.urdf' from '/home/rhos/Desktop/Open-Teach/openteach/components/environment/assets/urdf/'
Loading Assets
Env suite is not None

---

 Starting Allegro_Sim environment component

---

Start controlling the Simulation hand using the Oculus Headset.

End Effector Position [ 1. 1.2 0. -0.7071068 -0.7071068 0.
0. ]

Then the terminal stops printing.

[aadhithya14]

Hi @chiawenwang , the allegro sim can be only used with the KinovaBot because we have only one robot hand in the simulation. We use the Bimanual APK for Bimanual gripper and Libero Simulation. So if you are able to use the allegro simulation with Kinova Bot you are all set and everything is perfect.

[aadhithya14]

Hi @Norweig1an , looks like your simulation is working perfectly. Your problem is a network related issue which some other users have faced and can be solved by trying to reinstall the APK on the Oculus and force clean the app data using sidequest if there is loss of network in between operation.

[chiawenwang]

Hi @chiawenwang, the allegro simulation can only be used with KinovaBot as we only have one robot in the simulation. We are using Bimanual APK for Bimanual robot and Libero simulation. So if you can use allegro simulation with Kinova Bot, then everything is ready and everything is perfect.

Thank you for your detailed reply! We are planning to use our own robot in LIBERO, do you have some instructions?

[aadhithya14]

In the config you have the option to change the suite, Link you can choosever whichever suite of tasks you want in libero. Also if you plan to add your own robot, which follows the axis coordinate system of libero you can directly use the current codebase for your robot.

[chiawenwang]

In the config you have the option to change the suite, Link you can choosever whichever suite of tasks you want in libero. Also if you plan to add your own robot, which follows the axis coordinate system of libero you can directly use the current codebase for your robot.

OK! Thank you so much! There is no problem for me now!

[Norweig1an]

HI @aadhithya14 , I apologize for the delayed response. We tested on different PCs and received the same outcome: we are still unable to transfer data from VR Quest to the server. We identify the issue in

line 171 VR/Franka-Bot-Unity/Assets/Scripts/Gesture Detection/GestureDetector.cs
client.sendFrame(RightHandDatastring);
and
line 75 openteach/utils/network.py
raw data =self.socket.recv()

Although it appears that the data has been successfully sent by VR, none of it can be received by the PC. We've tried reinstall the APK and clean the data, but neither of them worked. Do you have any idea?

[aadhithya14]

Are both quest and the machine under the same network provider? This is necessary because the APK expects it in that format.

[Norweig1an]

Thanks for your reply! We're sure we have set all devices under the same network. And now we have discovered that we can transfer 2D plotter but hand angel. We still don't know the reason.

[chenshuo20]

Hi, thank you for your help! @aadhithya14

I am experiencing the same issue as @Norweig1an. I tried reinstalling the APK and ensured that both the Quest and the machine are on the same network, but it still doesn't work. My server gets stuck after printing "End Effector Position [1. 1.2 0. -0.7071068 -0.7071068 0. 0.]" and the Quest also becomes unresponsive.

Do you have any advice on how to resolve this issue? Thank you!

[kopiotrek]

Hi, I am having the same issue!

[aadhithya14]

This could be because of network instabilities. Try clearing the cache of the apk and start it. Or just reinstall the APK.

[Bailey-24]

Hello,
I have some problem running in the simulation(I didn't have any real robot).
I wonder why the issac gym not show up and can't collect the data.

I enter the VR and put my own computer in to the IP address (because i have no real robot) and enter "Stream".

It works well. But the issac gym not show up. How to solve.
Here are the output in the terimintor.

(openteach_isaac) (base) pi@pi:~/Documents/Open-Teach$ python3 teleop.py robot=allegro_sim sim_env=True
***************************************************************
     Starting keypoint position transform component
***************************************************************
***************************************************************
     Starting vr detector component
***************************************************************
Importing module 'gym_37' (/home/pi/Documents/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/pi/Documents/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
Importing module 'gym_37' (/home/pi/Documents/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/pi/Documents/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
***************************************************************
     Starting hand 2D plotter component
***************************************************************
PyTorch version 1.13.1+cu117
PyTorch version 1.13.1+cu117
Device count 1
/home/pi/Documents/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/pi/.cache/torch_extensions/py37_cu117 as PyTorch extensions root...
Device count 1
/home/pi/Documents/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/pi/.cache/torch_extensions/py37_cu117 as PyTorch extensions root...
Emitting ninja build file /home/pi/.cache/torch_extensions/py37_cu117/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
tcp://192.168.1.4:10005
tcp://192.168.1.4:11005
Trying to make simulation
Loading extension module gymtorch...
***************************************************************
     Starting allegro hand sim operator component
***************************************************************
Not connected to PVD
+++ Using GPU PhysX
/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link Base link (index: 0) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
  warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_3.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
  warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_7.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
  warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: disabled
/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_11.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
  warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/ikpy/chain.py:60: UserWarning: Link joint_15.0_tip (index: 5) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive
  warnings.warn("Link {} (index: {}) is of type 'fixed' but set as active in the active_links_mask. In practice, this fixed link doesn't provide any transformation so is as it were inactive".format(link.name, link_index))
***************************************************************
     Starting calibration component
***************************************************************

Calibration file already exists. Do you want to create a new one? Press y for Yes else press Enter
Stopping the calibrator.
Closing the subscriber socket in 192.168.1.4:8089.
THUMB BOUNDS IN THE OPERATOR: [array([0.07147029, 0.10102608]), array([0.06931292, 0.06902901]), array([-0.00888068,  0.08024863]), array([-0.0074921,  0.0941279]), [-0.029687233564084578, -0.060893850578688846]]
***************************************************************
     Starting Allegro_Sim control component
***************************************************************
Start controlling the robot hand using the Oculus Headset.

Applying retargeted angles

there are no more information output.

And other question is Can't collect data in allegro sim. I ran python3 data_collect.py robot=allegro_sim sim_env=True demo_num=1
The simliar scene in the VR. just Green boarder and nothing in the left upper corner.

The output is

(openteach_isaac) (base) pi@pi:~/Documents/Open-Teach$ python3 data_collect.py robot=allegro_sim sim_env=True demo_num=1
Reaching correct function
***************************************************************
     Starting RGB stream: 10005 component
***************************************************************
tcp://192.168.1.4:10005
***************************************************************
     Starting Depth stream: 11005 component
***************************************************************
Starting to record RGB frames from port: 10005
tcp://192.168.1.4:11005
***************************************************************
     Starting allegro_joint_states component
***************************************************************
Checking if the keypoint port is active...
Starting to record keypoints to store in extracted_data/demonstration_1/allegro_joint_states.h5.
***************************************************************
     Starting allegro_commanded_joint_states component
***************************************************************
Checking if the keypoint port is active...
Starting to record keypoints to store in extracted_data/demonstration_1/allegro_commanded_joint_states.h5.
^CSaving data to extracted_data/demonstration_1/allegro_joint_states.h5
Saving data to extracted_data/demonstration_1/allegro_commanded_joint_states.h5
Number of datapoints recorded: 0.
Number of datapoints recorded: 0.
Process Process-2:
Data record frequency: 0.0.
Data record frequency: 0.0.
Compressing keypoint data...
Compressing keypoint data...
Traceback (most recent call last):
Closing the subscriber socket in 192.168.1.4:10005.
  File "data_collect.py", line 16, in <module>
    main()
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/main.py", line 99, in decorated_main
    config_name=config_name,
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/_internal/utils.py", line 401, in _run_hydra
Saving data to extracted_data/demonstration_1/cam_0_rgb_video.avi
Traceback (most recent call last):
Number of datapoints recorded: 0.
Data record frequency: 0.0.
    overrides=overrides,
Storing the final version of the video...
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/_internal/utils.py", line 458, in _run_app
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pi/Documents/Open-Teach/openteach/components/initializers.py", line 240, in _start_depth_component
    component.stream()
  File "/home/pi/Documents/Open-Teach/openteach/components/recorders/image.py", line 129, in stream
    if self.image_subscriber.recv_depth_image() is None:
  File "/home/pi/Documents/Open-Teach/openteach/utils/network.py", line 160, in recv_depth_image
    raw_data = self.socket.recv()
  File "_zmq.py", line 1156, in zmq.backend.cython._zmq.Socket.recv
  File "_zmq.py", line 1191, in zmq.backend.cython._zmq.Socket.recv
  File "_zmq.py", line 1278, in zmq.backend.cython._zmq._recv_copy
  File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
KeyboardInterrupt
    lambda: hydra.run(
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/_internal/utils.py", line 461, in <lambda>
    overrides=overrides,
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 127, in run
    configure_logging=with_log_configuration,
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "data_collect.py", line 13, in main
    process.join()
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
Stored the video in extracted_data/demonstration_1/cam_0_rgb_video.avi.
Stored the metadata in extracted_data/demonstration_1/cam_0_rgb_video.metadata.
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/home/pi/anaconda3/envs/openteach_isaac/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Saved keypoint data in extracted_data/demonstration_1/allegro_commanded_joint_states.h5.
Saved keypoint data in extracted_data/demonstration_1/allegro_joint_states.h5.

How to solve these two problem. Thank you.

[Bailey-24]

I reinstall dirver and then solve the problem 1.

and I run two command in two speart temrintor and then can collect data.

[sernostine]

May I ask what driver you reinstalled? i kinda met the same problem,i have the skeletons on left top corner but the Sim window doesn't show up. Should this Isaac Gym simulation window appear on the server/PC side or in VR?