English| 简体中文

hobot_llm is a Node of Large Language Model (LLM) integrated into the Horizon RDK platform, allowing users to experience LLM at the edge. Currently, there are two ways to experience it: direct text chatting at the terminal and subscribing to text messages, then publishing the results in text format.

Before experiencing it, the following basic requirements need to be met:

• Confirm that the Horizon RDK is the 4GB RAM version.
• The Horizon RDK has been burned with the Ubuntu 20.04 system image provided by Horizon.
• Install transformers, using the command pip3 install transformers -i https://pypi.tuna.tsinghua.edu.cn/simple.
• Update hobot-dnn, using the command sudo apt update; sudo apt install -y hobot-dnn.

After starting the RDK X3, connect to the robot through SSH or VNC terminal, copy and run the following commands on the RDK system to complete the installation of the corresponding Nodes.

tros foxy:

sudo apt update
sudo apt install -y tros-hobot-llm

tros humble:

sudo apt update
sudo apt install -y tros-humble-hobot-llm

Before running the program, download the model files and unzip them using the following commands:

tros foxy:

# Download model files
wget http://sunrise.horizon.cc/llm-model/llm_model.tar.gz

# Unzip
sudo tar -xf llm_model.tar.gz -C /opt/tros/${TROS_DISTRO}/lib/hobot_llm/

tros humble:

# Download model files
wget http://sunrise.horizon.cc/llm-model/llm_model.tar.gz

# Unzip
source /opt/tros/humble/setup.bash ##export TROS_DISTRO=humble
sudo tar -xf llm_model.tar.gz -C /opt/tros/${TROS_DISTRO}/lib/hobot_llm/

At the same time, change the reserved memory size of BPU to 1.7GB. Refer to TODO for the setting method.

After rebooting, adjust the CPU's highest frequency to 1.5GHz and set the scheduling mode to performance, using the following commands:

sudo bash -c 'echo 1 > /sys/devices/system/cpu/cpufreq/boost'
sudo bash -c 'echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor'

Currently, two running programs are provided, hobot_llm_chat and hobot_llm. Among them, hobot_llm_chat provides a terminal interactive experience, allowing users to directly input text to experience the large model. The hobot_llm program subscribes to text messages of type std_msgs/msg/String, sends them to the large model for processing, and finally publishes the results in the form of std_msgs/msg/String. This program can be connected with other nodes, for example, to play the output text as speech.

tros foxy:

source /opt/tros/setup.bash

ros2 run hobot_llm hobot_llm_chat

tros humble:

source /opt/tros/humble/setup.bash

ros2 run hobot_llm hobot_llm_chat

After the program is launched, you can chat directly with the robot in the current terminal.

1. Start hobot_llm

   tros foxy:

   source /opt/tros/setup.bash
   
   ros2 run hobot_llm hobot_llm

   tros humble:

   source /opt/tros/humble/setup.bash
   
   ros2 run hobot_llm hobot_llm

2. Open a new terminal to subscribe to the output result topic

   tros foxy:

   source /opt/tros/setup.bash
   
   ros2 topic echo /text_result

   tros humble:

   source /opt/tros/humble/setup.bash
   
   ros2 topic echo /text_result

3. Open a new terminal to publish a message

   tros foxy:

   source /opt/tros/setup.bash
   
   ros2 topic pub --once /text_query std_msgs/msg/String "{data: ""What is the capital of China?""}"

   tros humble:

   source /opt/tros/humble/setup.bash
   
   ros2 topic pub --once /text_query std_msgs/msg/String "{data: ""What is the capital of China?""}"

The interface of hobot_llm program is as follows:

1. Model loading failure

   Confirm that the development board has 4GB of memory, and modify the reserved memory size of BPU to 1.7GB.

2. Output result is garbled

   Confirm that you have used the command sudo apt update; sudo apt install hobot-dnn to update hobot-dnn.

3. How to manually modify the BPU reserved memory to 1.7GB?

   Refer to the modification method in Setting ion_cam size in device tree, and change the values from 0x2a000000 to 0x6a400000 in the alloc-ranges and size attributes.