Docker Discussion

[AndrejOrsula]

Hello,

I have some Docker-related topics for discussion after looking through the current Docker setup. Please, let me know what you think about these points and if I should create a PR to address some of them.

Installation instructions for Docker

In the current prerequisites, reader is only instructed to install something (Docker with nvidia-docker2) if their machine has an NVIDIA GPU. It does not really instruct any reader (with no NVIDIA GPU). Therefore, I think something more general about Docker being a requirement should be specified, as many participants might have never used Docker before.

Using Docker as a non-root user

The current scripts in this repository assume that a non-root user can manage Docker (e.g. docker run ... instead of sudo docker run ...). Although it is something I always do, adding ${USER} to the docker group is technically an extra post-installation step that participants might skip, which would result in errors trying to run the included helper scripts. Therefore, this requirement might also need to be specified.

Installation script for Docker

Not always, but I often include .docker/host/install_docker.bash script under my repositories just in case somebody wanted to use it. It makes sure wget is present, then installs Docker from https://get.docker.com (via wget ... | sh). If NVIDIA GPU is detected, then either nvidia-container-toolkit or nvidia-docker2 is installed via apt based on the version of Docker. Lastly, it prompts the user whether to add ${USER} to the docker group [Y/n].

Maybe a similar script could be added to simplify the setup for some participants? Of course, it would work only on specific systems (Debian/Ubuntu) and there might even be some exceptions there. It is not something that I use to configure my systems, so I have not tested it a lot and there might be some issues (e.g. NVIDIA GPU detector might not be 100% correct).

Pre-built Docker image

This might NOT be possible because of using user_id ARG in the current Dockerfile. However, it takes 6-7 minutes to build the Docker image (with NVIDIA) on a "decent" laptop with a stable Internet connection. And this could be much higher when participants try to do it on-site on various machines with a possibly sub-optimal network connection. Building the image on-site might be alright, but I believe the setup time could be significantly reduced if the Docker image was pre-built and pushed to Docker Hub (or something similar like ghcr.io).

Probably a weird and not recommended use-case, but participants wouldn't even need to clone the repository then and just bash -c "$(wget -qO - https://raw.githubusercontent.com/osrf/icra2023_ros2_gz_tutorial/main/docker/run.bash)" -- <args>. 😆

NVIDIA GPU ( --runtime nvidia OR --gpus all)

Partially related to #13.

The current instructions and the helper run.bash script use nvidia-docker2 with --runtime nvidia for participants with NVIDIA GPU. From my understanding, with Docker version >19.3, --gpus all is enough if the system has nvidia-container-toolkit/nvidia-container-runtime. Underneath, nvidia-docker2 actually depends on nvidia-container-runtime. Therefore, --gpus all should always work with Docker version >19.3 even if only nvidia-docker2 was explicitly installed (whereas --runtime nvidia won't work if only nvidia-container-toolkit/nvidia-container-runtime were installed).

Therefore, something like this condition could be added (L75-:L79 in my run.bash). As mentioned before, similar check is done in install_docker.bash.

if dpkg --compare-versions "$(docker version --format '{{.Server.Version}}')" gt "19.3"; then
  GPU_OPT="--gpus all"
else
  GPU_OPT="--runtime nvidia"
fi

Automatic detection of NVIDIA GPU

Similarly, I don't know how foolproof $(lshw -C display 2> /dev/null | grep vendor) =~ NVIDIA is at detecting NVIDIA GPU, but it could be used instead of explicitly passing --no-nvidia argument. It certainly does not detect if the NVIDIA driver is installed correctly, which is needed for either option to work. So maybe not...

For non-NVIDIA GPUs, I have seen something like --device=/dev/dri before but never tested it. Maybe it is not even required with --privileged option that is used right now?

Is xhost + necessary for GUI because of Wayland?

I noticed #11 and I don't use Wayland, so I won't go into depth. But I am wondering if xhost + is needed only for Wayland (because I see -e XAUTHORITY=$XAUTH and other options that usually work for me on their own).

Isolation of ROS 2 communication and Gazebo transport

As all participants might be on the same local network, maybe there is a need to partition/isolate their communication as a precaution. I am not exactly sure what is the proper way, but maybe something like this?

ROS_LOCALHOST_ONLY="1"
GZ_IP="127.0.0.1"
GZ_RELAY="127.0.0.1"

I remember having an issue with similar isolation because of disabled multicast on the loopback network interface, so something like ifconfig lo multicast might also be necessary?

Hardware and OS of participants

Now for the difficult point. Even with Docker, a lot of the setup might still be restricted to Linux (or even Ubuntu). The last time I tried to run Gazebo/Rviz under Docker on Windows 10 (Docker Desktop, not WSL 1/2), I couldn't get GUI working. Maybe it is different with WSL now, but I have not tried (and not sure how long it would take to set it up). Maybe MacOS works fine (x86_64/arm64)? But even on Ubuntu, something like broken NVIDIA drivers could prevent the GUI from working for some reason, or specific hardware could negatively impact rendering or something else. Are the participants recommended to bring some specific system, and do you have some insights/suggestions on how to tackle this?

[mabelzhang]

Thank you Andrej. The points you raise are very good because it tells us a different setup than the one we usually use in our team (nvidia-docker2 I basically copied from the SubT instructions). This is one of the things we were afraid of - that people come with any random setup and we're not prepared for them. I appreciate you taking the time to try things out and writing this up!

It does not really instruct any reader (with no NVIDIA GPU)

There is an --no-nvidia flag in the README if they don't have NVIDIA. But they still need some kind of graphics card for Gazebo (and maybe RViz), otherwise things might be slow with just CPU. I've also tried with an Intel integrated GPU and that works too. Does that flag address what you mean?

Docker being a requirement should be specified

So, Docker is optional. We are using that as the default because it's much easier than asking people to install ROS Humble and Gazebo Garden themselves - which they can do, but we don't think there's enough time and WiFi bandwidth for everyone in the 1.5 hours, hence Docker. These are the latest distributions, but they are not officially supported side by side (Humble comes with Fortress). I know, I know, Yeah... You're right, it would be good to add a sentence to clarify this is the intention.

setup time could be significantly reduced if the Docker image was pre-built and pushed to Docker Hub

Yes! We already have a DockerHub repo created (had to go through some logistics for that) here https://hub.docker.com/r/osrf/icra2023_ros2_gz_tutorial
We haven't pushed to it yet, waiting to finalize the Docker image.
I have to re-build the image to include the Fuel resources that people might need for the Gazebo splash screen, to make the WiFi bandwidth requirement smaller.
We are also bringing a couple dozen USB sticks (a LOT of administrative logistics is needed to make that happen, but we're going through that) with the Docker image to help speed up the process in case WiFi sucks at the conference (the tutorial is on the very first day, unfortunately that might be a reality...)

--gpus all

I'm not familiar with this flag, but we're reading here https://stackoverflow.com/questions/25185405/using-gpu-from-a-docker-container that "Please note, the flag --gpus all is used to assign all available gpus to the docker container."
That's a little worrying. Does it starve the host machine of GPUs?

Automatic detection of NVIDIA GPU

For this tutorial, since it's a one day thing, we do have to trade off what will be more guaranteed to work, versus what is correct. That is not what we usually do in ROS and Gazebo development, but for the 1.5 hours to run smoothly, we have to do this to save everybody's time.

I would opt for a foolproof way that works, which might be manual like with --no-nvidia, rather than doing something fancy to detect things.

But it would be great if you could add it to the docker/README.md documentation as a future work or troubleshoot section!

xhost +

Yeah we found this to work on Wayland (one of the organizer's setup). It is more permissive than we like, but Wayland is difficult to make work with Gazebo and Rviz, in the sense that we know people use different environment variables to make it work. We tried a couple, and decided it wasn't worth it to waste people's time on the day of to figure it out. So we opted to use a permissive way for it to work for everybody. I don't have Wayland, and this doesn't break anything for me. So it works for non-Wayland too.

As all participants might be on the same local network, maybe there is a need to partition/isolate their communication as a precaution

ROS_LOCALHOST_ONLY="1"
GZ_IP="127.0.0.1"
GZ_RELAY="127.0.0.1"

I remember having an issue with similar isolation because of disabled multicast on the loopback network interface, so something like ifconfig lo multicast might also be necessary?

Oof good point. @clalancette should we add network config to run.bash?
We are using --network=host in there, in order for non-NVIDIA displays to work. So... everybody is using the host network.

Are the participants recommended to bring some specific system, and do you have some insights/suggestions on how to tackle this?

That's our worry too. Let's add Ubuntu Jammy to the main README. We are planning on sending an email to the registered participants (with luck these are the people who show up) about system requirements. But it is much better to put it in the README. Would you mind adding a section in a PR? Probably under a new heading right after the Overview section (with quick access link at the top too) called Requirements.

Docker was our solution to platforms. Mac and Windows are experimental for Gazebo natively, so Docker is already a step better. If they have an Ubuntu VM, that is the best. Otherwise Mac and Windows will be hard.

Another way we're tackling this is to pair people up. That way at least they have 2 shots at having the right system.

If it happens on the day that many pairs cannot get the right setup, then I'll present the tutorial so that they can at least see what's supposed to happen. Otherwise our Plan A is to let people follow the tutorial at their own pace, and I'd be a floater helping out, nobody would be talking at people when they're trying to do a tutorial.

Please feel free to open PRs. Basically the rule of thumb we're leaning toward is, simplest and quickest solutions that make most people's machines work in 1.5 hours (without making it overly unsecure). Thank you!!

[AndrejOrsula]

It does not really instruct any reader (with no NVIDIA GPU)

There is an --no-nvidia flag in the README if they don't have NVIDIA. But they still need some kind of graphics card for Gazebo (and maybe RViz), otherwise things might be slow with just CPU. I've also tried with an Intel integrated GPU and that works too. Does that flag address what you mean?

Yes, this is good. What I meant was to make a slight edit to the instructions under docker/README.md#Prerequisites in order to make it more explicit that all participants that want to use Docker must install Docker on their machines - just to make it more clear for people with no prior Docker experience. For example:

- If your machine has an NVIDIA GPU, install Docker and NVIDIA Docker 2
- following instructions [here](https://github.com/osrf/subt/wiki/Docker%20Install).
+ Before starting, please ensure that Docker is installed and configured on your
+ system by following the instructions [here](https://github.com/osrf/subt/wiki/Docker%20Install).
+ If your machine has an NVIDIA GPU, make sure to also install NVIDIA Docker 2 following the same instructions.

This is also where "using Docker as a non-root user" could be mentioned.

I have to re-build the image to include the Fuel resources that people might need for ...

That's a great idea!

We are also bringing a couple dozen USB sticks with the Docker image

Nice!

On a side note, it would have been nice for users of Windows/MacOS if there was a bootable USB stick with Ubuntu (or another OS) that has preinstalled Docker, with the Docker image already loaded. I suppose the booted session wouldn't even need to be persistent for something like a tutorial (just boot from the USB stick, start the ROS 2/Gazebo application via Docker, go through the tutorial, shut down and unplug the USB). I have no idea if it has been done before, and there are probably thousands of pitfalls with drivers and other issues, but I kinda like the idea.

There's also the option of bringing a couple of USB sticks with installable OS for participants that feel adventurous and want to dual-boot Ubuntu alongside their existing OS. However, I suppose that would be too much of a liability in case something went wrong during the installation.

"Please note, the flag --gpus all is used to assign all available gpus to the docker container." That's a little worrying. Does it starve the host machine of GPUs?

I am not an expert on the topic, but I always just use --gpus all and haven't encountered problems. The host and Docker container(s) can use GPUs at the same time. Of course, if an application inside a running Docker container fills up VRAM, then there won't be any left for applications running under the host. But I don't think that's any different from an application under host using up all resources.

Would you mind adding a section in a PR? Probably under a new heading right after the Overview section (with quick access link at the top too) called Requirements.

I can take a look at it over the weekend/next week.

[mabelzhang]

Sounds good. Thanks for the thoughts!

Re bootable Ubuntu USB stick with Docker and images:
It would be nice, but we just don't have enough labor to make that happen. We have two people part-time running this tutorial, and it started as a side effort. So we don't have any official resources set aside for any of the work going into this tutorial.

Re USB sticks with installable OS:
Given that they only have 1.5 hours, we don't think spending time on the OS is going to be worth it, because then most of the time would be used to install, and there would be very little time left to do actual ROS/Gazebo. We spoke to people who run university-level classes, and for people who have never used Ubuntu or ROS, most of the time really ends up being installing Ubuntu. We didn't want this tutorial to end up like that.
We wanted to use the time as efficiently as possible, so that people aren't bored out of their minds by a 3-hour coding session (which we aren't doing) in each of AM and PM. Instead, we hope they get some interactions and networking with the breakout discussions, and then get a taste of ROS/Gazebo with some hands-on practice.
The hope is that if we pair people up, at least one person in each group will have Ubuntu. If half the room doesn't have Ubuntu, then we'll go with Plan B and present what they would have seen if they had Ubuntu, and the other half of the room with Ubuntu can follow the tutorial at their own pace.

It's tough! The toughest part about a ROS/Gazebo tutorial ends up not being about ROS/Gazebo at all, but about WiFi and OS! We find that we're spending half of the time working on Plan B's and logistics, on top of the actual technical content.

[AndrejOrsula]

Yes, that makes sense and I certainly agree. Both ideas related to USB sticks are not realistic within the scope.

I opened a couple of PRs (drafts) based on this discussion. However, I am not 100% sure about all of them.

• Update installation instructions for Docker #17
• Update run.bash to use --gpus all when possible #18
• Synchronize ROS 2 and Gazebo ENVs with the host (Default to isolated communication) #19

I noticed that you already added a mention about Ubuntu Jammy in #16. I don't see any other specific points from this discussion, but please let me know if I missed something.