Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Question] Stuck in headless mode #1924

Open
Elycyx opened this issue Feb 22, 2025 · 1 comment
Open

[Question] Stuck in headless mode #1924

Elycyx opened this issue Feb 22, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@Elycyx
Copy link

Elycyx commented Feb 22, 2025

I encountered an issue when using the data generation example in imitation learning. When running it in headless mode, the process gets stuck at a certain step without any further output in the terminal.

Additionally, when I try to train a BC policy using the following command: ./isaaclab.sh -p scripts/imitation_learning/robomimic/train.py --task Isaac-Stack-Cube-Franka-IK-Rel-v0 --algo bc --dataset ./datasets/generated_dataset.hdf5
It also gets stuck, seemingly due to headless mode. I am certain that I successfully trained the BC policy using the same process a few weeks ago, but I’m not sure why headless mode is causing issues this time.

[INFO] Using python from: /home/***/miniforge3/envs/isaaclab/bin/python                                                
[INFO][AppLauncher]: Loading experience file: /mnt/sda1/Isaac/IsaacLab/apps/isaaclab.python.headless.kit
Loading user config located at: '/mnt/sda1/Isaac/ov/pkg/isaac-sim-4.5.0/kit/data/Kit/Isaac-Sim/4.5/user.config.json'
[Info] [carb] Logging to file: /mnt/sda1/Isaac/ov/pkg/isaac-sim-4.5.0/kit/logs/Kit/Isaac-Sim/4.5/kit_20250222_165907.log
2025-02-22 08:59:07 [0ms] [Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
2025-02-22 08:59:08 [619ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) Graphics (RPL-S)
2025-02-22 08:59:08 [619ms] [Warning] [gpu.foundation.plugin] Skipping unsupported non-NVIDIA GPU: Intel(R) Graphics (RPL-S)

|---------------------------------------------------------------------------------------------|
| Driver Version: 560.35.03     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 4070 Laptop.. | Yes: 0 |     | 8188    MB | 10de      | 0          |
|     |                                  |        |     |            | 2860      | 6356a0df.. |
|     |                                  |        |     |            | 1         |            |
|---------------------------------------------------------------------------------------------|
| 1   | Intel(R) Graphics (RPL-S)        |        |     | 23848   MB | 8086      | 0          |
|     |                                  |        |     |            | a788      | 868088a7.. |
|     |                                  |        |     |            | 0         |            |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 6.8.0-52-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: Intel(R) Core(TM) i7-14700HX
| Cores: 20 | Logical Cores: 28
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 31798 | Free Memory: 23799
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2047
|---------------------------------------------------------------------------------------------|
2025-02-22 08:59:08 [885ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled.
2025-02-22 08:59:08 [885ms] [Warning] [gpu.foundation.plugin] Detected IOMMU is enabled. Running CUDA peer-to-peer bandwidth and latency validation.
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0 
     0 175.86 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0 
     0   1.76 

   CPU     0 
     0   1.35 
2025-02-22 08:59:08 [1,024ms] [Warning] [omni.usd_config.extension] Enable omni.materialx.libs extension to use MaterialX
2025-02-22 08:59:10 [2,242ms] [Warning] [omni.datastore] OmniHub is inaccessible
2025-02-22 08:59:10 [2,439ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.
@RandomOakForest
Copy link
Collaborator

Thank you for posting this. We will take a look. Could you fill in our Bug Report template in the issues? Or simply replace "[Question]" in the title with "[Bug Report]" and edit your text to fill in the following:

If you are submitting a bug report, please fill in the following details and use the tag [bug].

### Describe the bug

A clear and concise description of what the bug is.

### Steps to reproduce

Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful.

<!-- Please post terminal logs, minimal example to reproduce, or command to run under three backticks (```) to allow code formatting.

****Paste your error here****

For more information on this, check: https://www.markdownguide.org/extended-syntax/#fenced-code-blocks

-->

### System Info

Describe the characteristic of your environment:

<!-- Please complete the following description. -->
- Commit: [e.g. 8f3b9ca]
- Isaac Sim Version: [e.g. 2022.2.0, this can be obtained by `cat ${ISAACSIM_PATH}/VERSION`]
- OS: [e.g. Ubuntu 20.04]
- GPU: [e.g. RTX 2060 Super]
- CUDA: [e.g. 11.4]
- GPU Driver: [e.g. 470.82.01, this can be seen by using `nvidia-smi` command.]

### Additional context

Add any other context about the problem here.

### Checklist

- [ ] I have checked that there is no similar issue in the repo (**required**)
- [ ] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

### Acceptance Criteria

Add the criteria for which this task is considered **done**. If not known at issue creation time, you can add this once the issue is assigned.

- [ ] Criteria 1
- [ ] Criteria 2

Thank you.

@RandomOakForest RandomOakForest added the bug Something isn't working label Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants