ROS2 nodes can't communicate between Docker containers

[mmcza]

Is there an already existing issue for this?

• I have searched the existing issues

Expected behavior

ROS2 nodes can subscribe to ROS2 nodes running in another docker containers or on host

Current behavior

ROS2 nodes that run in a container are able to list topics created by ROS2 nodes run on host/another container, but are not able to subscribe to them. The same applies to host if a node is running inside a container.

Steps to reproduce

With a running Docker container from a ROS2 Humble image you can run any node or create a publisher like this:

ros2 topic pub /test std_msgs/msg/Bool

In another Docker container (not another terminal with the same image but a completely separate one) you should be able to see the topic after running

ros2 topic list

but

ros2 topic echo /test

will not return any messages

Fast DDS version/commit

ros-humble-fastrtps 2.6.8-1jammy.20240728.201342
ros-humble-fastrtps-cmake-module 2.2.2-2jammy.20240728.202329
ros-humble-rmw-fastrtps-cpp 6.2.7-1jammy.20240728.212513
ros-humble-rmw-fastrtps-shared-cpp 6.2.7-1jammy.20240728.211440
ros-humble-rosidl-typesupport-fastrtps-c 2.2.2-2jammy.20240728.210059
ros-humble-rosidl-typesupport-fastrtps-cpp 2.2.2-2jammy.20240728.205803

Platform/Architecture

Other. Please specify in Additional context section.

Transport layer

Default configuration, UDPv4 & SHM

Additional context

OS - Ubuntu 22.04
ROS2 Humble

I discovered the issue around a week ago, as 2 weeks ago everything was working properly. Additionally it was tested on another computers, where the same issue was experienced.

XML configuration file

No response

Relevant log output

No response

Network traffic capture

No response

[rsanchez15]

Hi @mmcza

I've just tested what you explained and it worked for me. Could you please send us the exact command you are using for running the Docker containers?

Could you also run the following commands and tell me what you see:

• Terminal 1

  docker run -it --rm ros:humble
  ros2 topic pub /test std_msgs/msg/Bool
  

• Terminal 2

  docker run -it --rm ros:humble
  ros2 topic list
  ros2 topic echo /test
  

[iandresolares]

Hello,
I am experiencing the same issue with ROS2 humble as well. In my case both containers are launched from different docker-compose.yaml files (although the same thing happens launching both containers from the same docker-compose.yaml file). Both containers are set to network_type host.

This works completely fine:

Hi @mmcza

I've just tested what you explained and it worked for me. Could you please send us the exact command you are using for running the Docker containers?

Could you also run the following commands and tell me what you see:

* Terminal 1
  ```
  docker run -it --rm ros:humble
  ros2 topic pub /test std_msgs/msg/Bool
  ```

* Terminal 2
  ```
  docker run -it --rm ros:humble
  ros2 topic list
  ros2 topic echo /test
  ```

[mmcza]

@rsanchez15 thanks for the response.

I run the test you mentioned, and you can see the results below. The right terminal was on host and the left one was in a Docker container.

To start the container I used a script that looks like this:

docker run -it --rm \
	--name=ros_humble \
	--shm-size=1g \
	--ulimit memlock=-1 \
	--env="DISPLAY" \
	--env="QT_X11_NO_MITSHM=1" \
	--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --privileged \
	--device=/dev/usb \
	--device=/dev/video0 \
	--gpus all \
	--env="XAUTHORITY=$XAUTH" \
	--volume="$XAUTH:$XAUTH" \
	--env="NVIDIA_VISIBLE_DEVICES=all" \
	--env="NVIDIA_DRIVER_CAPABILITIES=all" \
	--network=host \
	--ipc=host \
	--pid=host \
	ros-humble-image \
	bash

The Dockerfile I used was based on osrf/ros:humble-desktop (I also tested with ros:humble-ros-base-jammy).

Btw which OS are you using on the host? Because me and my coworker have the same issue, and we both have Ubuntu 22.04 installed and updated, but when we checked on a pc that wasn't updated lately there was no issue (also there was no issue on Ubuntu 20.04). And we found that after changing to CycloneDDS we were able normally communicate between containers.

[rsanchez15]

The problem is due to the SHM transport. When you launch a Docker with --net=host and --ipc=host configuration, the publisher and subscriber detect that they are on the same host and try to use SHM. This is due to how the calculation of SHM usage was done until now. In addition, SHM uses the /dev/shm directory, which in this case is being attempted to be shared by users who have different permissions: root in the case of Docker and $USER` in the case of the host. This causes communication to fail.

To fix this you have several quick solutions:

• Have the same user on Docker as on the host.
• Launch the ROS 2 application as root (equivalent to the above but without changing the Docker configuration).
• Use UDP instead of SHM. This is as simple as running the following before launching the ROS 2 application in the Docker.
  export FASTDDS_BUILTIN_TRANSPORTS=UDPv4
• Do not set --ipc=host --net=host when running the Docker container.

[iandresolares]

In my case setting UDP transport worked. However I wonder, since I am testing in two different docker containers, both of them are running all ros2 related commands as root, why does SHM transport does not work in this case?