aiBrick - Machine Learning for Pybricks

[repkovsky]

Hi everyone,

as the future of Mindstorms and its Machine Learning module is unclear, while the future of Pybricks is surely bright, I decided to put some effort to write aiBrick - small AI/ML add-on module for Pybricks. To make it versatile, easy to use and develop, I assumed the AI/ML algorithms will be run in web browser with use of the microphone/camera from PC/smartphone, and the results of processing will be send to the Lego hub using the Bluetooth UART. On the hub's side, there is dedicated class for receiving the UART transmission and processing the results for further use with Pybricks apps. For now aiBrick works with AI/ML models trained in Teachable Machine, but I have some ideas to extend it with different modules which use AI/ML. I hope I will have some time to record demo showing example how to use aiBrick, for now there is some description and code example with working model in the project's repository https://github.com/repkovsky/aibrick. I am looking forward to your feedback!

[laurensvalk]

This is absolutely amazing! Well done!

I really love the simplicity of this setup, where the users can do all coding in the same environment.

[laurensvalk]

[repkovsky]

Thanks! One thing I want to ask is the possibility of hiding Bluetooth transmission. Now all communication is visible in Pybricks console, which is sometimes useful for debug, as all messages from/to aiBrick are send as a text. But for more complex models, for example object detection or pose detection, the volume of data which needs to be send (e.g. coordinates of the object) is much bigger for smooth operation. The Bluetooth transmission is very slow, even for the classification model from provided example, where the probability for only 6 classes is transmitted, the maximum update rate is about 10-15Hz. So it would be probably necessary to transmit the data in binary mode, where the information can be encoded much more efficiently. But then the console content will be a total mess, which is going to make debugging a bit harder. So is it or will it be possible to use Bluetooth "under the hood", as it works now for e.g. Bluetooth Remote?

[laurensvalk]

Yes, I suppose it depends on how far you want to take it. What you are describing might require some changes in Pybricks Code and perhaps the firmware, in order to have a hidden data channel for this kind of thing. (The remote case seems a little different, since Pybricks Code does not get involved there).

There are other cases where such a data channel could be very useful, such as:

• Live sensor data view
• A drawing canvas (draw shapes, text, etc), so your robot has a large "screen"
• Data logging and visualization
• A custom remote control with widgets

Taking it a step further, this type of application could possibly run in a Pybricks Code tab:

This is probably out of reach in the short term, but very cool to think about 😄

[repkovsky]

Very cool indeed! :) These features look great, I cannot wait to see them implemented :).