[Snap] Give add-ons access to serial ports

[benfrancis]

May require the serial-port/raw-usb interface.

Currently the Zigbee add-on does not work...

STR:

• In the web interface navigate to Main Menu -> Settings -> Add-ons
• Click "+"
• Search for the Zigbee add-on
• Install the Zigbee add-on

Expected:

• Zigbee add-on finds XStick Zigbee dongle

Actual:

• Zigbee add-on installs successfully but dongle not found

Logs from /var/snap/webthings-gateway/current.webthings/log/run-app.log:

2024-08-20 19:06:03.296 INFO   : zigbee-adapter: Loading add-on zigbee-adapter from /root/snap/webthings-gateway/x1/.webthings/addons/zigbee-adapter
2024-08-20 19:06:03.879 INFO   : zigbee-adapter: DEBUG config = ''
2024-08-20 19:06:03.886 INFO   : zigbee-adapter: Probing serial ports
2024-08-20 19:06:04.081 INFO   : zigbee-adapter: Serial ports that were found:
2024-08-20 19:06:04.134 ERROR  : zigbee-adapter: Failed to start add-on zigbee-adapter: No Zigbee dongle found

[benfrancis]

The same will likely be true for the Z-Wave add-on too, since that also uses a serial connection.

In order to access serial ports, a snap needs access to the serial-port interface. We can request this interface in snapcraft.yaml, but it is usually not connected by default so would require a user to manually connect the interface on the command line. We can request an exception in the snap store but it may not be granted.

As I understand it on Ubuntu Core it also requires serial device paths to be hard coded in a gadget snap, unless using the experimental hotplug support, which is still marked as experimental and therefore can not be enabled by default. This may be a problem because the gateway supports multiple USB dongles.

As I understand it the existing Raspbian image supports hotplug USB devices.

[ogra1]

as long as the zigbee devices are USB dongles (vs. actual sub-d attached serial devices) it should be possible to simply turn on hotplug support of the UbuntuCore image via a gadget.yaml setting (or simply tell the user to turn it on manually when used out of a pre-made image) ... could you try turning on hotplug on your test device and see if the dongle can be found or if we need to dig deeper here ... (perhaps hardware-observe or system-observe interfaces might be needed alongside and probably udevadm as a bundled binary, we'd need more logs here but i think this could be a viable way forward.)

[benfrancis]

@ogra1 Thank you for the advice.

I enabled hotplug with a Digi XStick (Zigbee dongle) and Aeotec Z-Stick (Z-Wave dongle) plugged in and saw the following:

$ snap connections system
...
serial-port              -                                      :aeotecz-stickgen5zw0     -
serial-port              -                                      :xstick                   -

So that's a good start.

Oddly, when I disconnected the Aeotec Z-Wave dongle and plugged a Z-Wave dongle from Sigma Designs in its place, it was detected as aeotecz-stickgen5zw0 - so that's not ideal...

But the obvious question is if a serial-port plug needs to be connected to a slot or slots which may be dynamically added and removed at runtime and whose name may not be known in advance, how is that interface connected? The connections can't just be statically defined in a snapcraft.yaml, or even a gadget snap. The nature of the gateway's extensible add-ons system also means we may not even know in advance the vendor IDs of all devices that may need to be connected in the future. For our non-technical users it's also not acceptable to require a user to run commands on the command line, all user interaction needs to be through the gateway's web interface.

Looking at the source code of the Zigbee adapter add-on as a starting point, it appears to use the serialport npm package (and home-grown serial-prober package which builds on serialport to probe a serial port for certain vendor and product IDs). I have no idea what magic serialport uses on the back end, but it seems to support a lot of different architectures.

Presumably to get USB dongles to connect dynamically at runtime on Ubuntu Core we would need some additional logic which automatically connects the serial-port plug to sockets as they appear. Is that even possible?

FYI I'm already adding a system-observe plug in #3168. What were you thinking the hardware-observe interface and udevadm binary might be used for?

[ogra1]

udevadm could monitor the plug events so your app can be notified about a dongle being added/removed and act accordingly, hardware-observe was simply a guess like system-observe ...

For the monitoring you could then have a scripted daemon that operates around something like:

udevadm monitor --subsystem-match=usb --property --udev

[benfrancis]

@ogra1 Thank you for this suggestion.

The following is the output of udevadm when I plugged in an XStick Zigbee dongle:

$ udevadm monitor --subsystem-match=usb --property --udev
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing

UDEV  [1530.379429] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5 (usb)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5
SUBSYSTEM=usb
DEVNAME=/dev/bus/usb/001/004
DEVTYPE=usb_device
PRODUCT=403/6001/600
TYPE=0/0/0
BUSNUM=001
DEVNUM=004
SEQNUM=4086
USEC_INITIALIZED=1530369731
ID_BUS=usb
ID_MODEL=XStick
ID_MODEL_ENC=XStick
ID_MODEL_ID=6001
ID_SERIAL=Digi_XStick
ID_VENDOR=Digi
ID_VENDOR_ENC=Digi
ID_VENDOR_ID=0403
ID_REVISION=0600
ID_USB_MODEL=XStick
ID_USB_MODEL_ENC=XStick
ID_USB_MODEL_ID=6001
ID_USB_SERIAL=Digi_XStick
ID_USB_VENDOR=Digi
ID_USB_VENDOR_ENC=Digi
ID_USB_VENDOR_ID=0403
ID_USB_REVISION=0600
ID_USB_INTERFACES=:ffffff:
ID_VENDOR_FROM_DATABASE=Future Technology Devices International, Ltd
ID_MODEL_FROM_DATABASE=FT232 Serial (UART) IC
ID_PATH_WITH_USB_REVISION=pci-0000:00:14.0-usbv2-0:5
ID_PATH=pci-0000:00:14.0-usb-0:5
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5
ID_FOR_SEAT=usb-pci-0000_00_14_0-usb-0_5
MAJOR=189
MINOR=3
TAGS=:seat:
CURRENT_TAGS=:seat:

UDEV  [1530.424421] add      /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5:1.0 (usb)
ACTION=add
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5:1.0
SUBSYSTEM=usb
DEVTYPE=usb_interface
PRODUCT=403/6001/600
TYPE=0/0/0
INTERFACE=255/255/255
MODALIAS=usb:v0403p6001d0600dc00dsc00dp00icFFiscFFipFFin00
SEQNUM=4087
USEC_INITIALIZED=1530375340
ID_VENDOR_FROM_DATABASE=Future Technology Devices International, Ltd
ID_MODEL_FROM_DATABASE=FT232 Serial (UART) IC
ID_PATH_WITH_USB_REVISION=pci-0000:00:14.0-usbv2-0:5:1.0
ID_PATH=pci-0000:00:14.0-usb-0:5:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_1_0

UDEV  [1530.427739] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-5 (usb)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5
SUBSYSTEM=usb
DEVNAME=/dev/bus/usb/001/004
DEVTYPE=usb_device
DRIVER=usb
PRODUCT=403/6001/600
TYPE=0/0/0
BUSNUM=001
DEVNUM=004
SEQNUM=4088
USEC_INITIALIZED=1530369731
ID_BUS=usb
ID_MODEL=XStick
ID_MODEL_ENC=XStick
ID_MODEL_ID=6001
ID_SERIAL=Digi_XStick
ID_VENDOR=Digi
ID_VENDOR_ENC=Digi
ID_VENDOR_ID=0403
ID_REVISION=0600
ID_USB_MODEL=XStick
ID_USB_MODEL_ENC=XStick
ID_USB_MODEL_ID=6001
ID_USB_SERIAL=Digi_XStick
ID_USB_VENDOR=Digi
ID_USB_VENDOR_ENC=Digi
ID_USB_VENDOR_ID=0403
ID_USB_REVISION=0600
ID_USB_INTERFACES=:ffffff:
ID_VENDOR_FROM_DATABASE=Future Technology Devices International, Ltd
ID_MODEL_FROM_DATABASE=FT232 Serial (UART) IC
ID_PATH_WITH_USB_REVISION=pci-0000:00:14.0-usbv2-0:5
ID_PATH=pci-0000:00:14.0-usb-0:5
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5
ID_FOR_SEAT=usb-pci-0000_00_14_0-usb-0_5
MAJOR=189
MINOR=3
TAGS=:seat:
CURRENT_TAGS=:seat:

UDEV  [1530.444025] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5:1.0 (usb)
ACTION=bind
DEVPATH=/devices/pci0000:00/0000:00:14.0/usb1/1-5/1-5:1.0
SUBSYSTEM=usb
DEVTYPE=usb_interface
DRIVER=ftdi_sio
PRODUCT=403/6001/600
TYPE=0/0/0
INTERFACE=255/255/255
MODALIAS=usb:v0403p6001d0600dc00dsc00dp00icFFiscFFipFFin00
SEQNUM=4100
USEC_INITIALIZED=1530375340
ID_VENDOR_FROM_DATABASE=Future Technology Devices International, Ltd
ID_MODEL_FROM_DATABASE=FT232 Serial (UART) IC
ID_PATH_WITH_USB_REVISION=pci-0000:00:14.0-usbv2-0:5:1.0
ID_PATH=pci-0000:00:14.0-usb-0:5:1.0
ID_PATH_TAG=pci-0000_00_14_0-usb-0_5_1_0

I'm not completely clear on how you're suggesting I would get from this output to the gateway application being able to access the USB device.

As I understand it the hotplug support dynamically creates "slots" for the serial-port interface, but how would I get the gateway application to then automatically connect its serial-port plug to that slot when it appears? Does the user not need to manually run snap connect to do that? How would I achieve that programmatically?

Currently the WebThings Zigbee adapter add-on uses this home-grown serial-prober package (which uses the serialport npm package) to detect and connect to serial ports. It looks for devices with a path like /dev/cu.usbXXX. The WebThings Z-Wave adapter add-on does the same, but uses the serialport package directly.

I'm not sure whether once the serial-port interface is connected this existing code might "Just Work"? I'm thinking maybe not if the interface only gives you access to a specific serial port, it may not let you list them?

I also note that the DEVNAME output by udevadm is of the form /dev/bus/usb/001/004, which I understand describes the full physical USB topology with metadata about each device, rather than just looking like a plain serial port. I'm not sure how the gateway application would get from this to a useable serial port?

[benfrancis]

See also: https://forum.snapcraft.io/t/hotplug-support/10750/4?u=benfrancis

[benfrancis]

@ogra1 @dilyn-corner @lorenzo-medici Am I right in thinking that programmatically connecting a plug to a slot would require the snapd-control interface, which is considered super privileged and would therefore require a store assertion to even add a plug to the gateway snap in the public Snap Store, let alone have it autoconnect?

I've noticed that in the documentation for the serial-port interface it appears that it's possible to specify a serial-port plug which autoconnects to a specific device path or a triplet of usb-vendor, usb-product and path. Is that correct?

If I've understood correctly then this would require hard coding the the vendor and product IDs of every possible hardware dongle we might ever want to connect to in a long list of plugs in snapcraft.yaml, which doesn't scale at all but might be a way for us to get started.

When a triplet of usb-vendor, usb-product and path are provided in a plug in snapcraft.yaml, would I need to know the actual device path that has been assigned to a USB device, or does the path I provide get automatically symlinked to the real path of a device which matches the usb-vendor and usb-product IDs?

What happens if multiple dongles with the same usb-vendor and usb-product IDs are connected? That's not a common use case, but I believe it is something that WebThings Gateway currently supports.

[dilyn-corner]

require a store assertion to even add a plug to the gateway snap in the public Snap Store, let alone have it autoconnect?

You are correct, and additionally it is quite a challenge to get a snap with snapd-control in the Global Store, let alone autoconnecting.

Regarding specifically serial-port things, snapd has been hard at work adding support for hotplugging. The serial-port interface is currently the only interface which supports this (experimental) feature, but I've been using it on my own system for quite some time and it has proven quite useful.
It allows you to avoid hardcoding slots in the gadget for every potential serial device you can connect.
The documentation specifically mentions serial USB adapters and that is all I have tested with.

You can try it with snap set system experimental.hotplug=true.
You can checkout the documentation about this here.

[benfrancis]

@dilyn-corner Thank you for your reply, but @ogra1 already linked to the hotplug documentation. Please see my questions above.

In summary: I understand that the hotplug support automatically creates serial-port slots, but how do I get my snap to connect to them?