diff --git a/docs/conf.py b/docs/conf.py index 3fe4d7e..90af6da 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -32,6 +32,7 @@ 'sphinx.ext.mathjax', 'sphinxcontrib.youtube', "sphinx.ext.autosectionlabel", + 'sphinxcontrib.video', ] # True to prefix each section label with the name of the document it is in, followed by a colon. diff --git a/docs/operation.rst b/docs/operation.rst index 0d027ac..67b3437 100644 --- a/docs/operation.rst +++ b/docs/operation.rst @@ -10,3 +10,4 @@ Operation ./operation/data_collection.rst ./operation/training.rst ./operation/hugging_face.rst + ./operation/lerobot_guide.rst diff --git a/docs/operation/lerobot_guide.rst b/docs/operation/lerobot_guide.rst new file mode 100644 index 0000000..2eeddd6 --- /dev/null +++ b/docs/operation/lerobot_guide.rst @@ -0,0 +1,382 @@ +========================== +LeRobot X Aloha User Guide +========================== + +Virtual Environment Setup +========================= + +Containerization is crucial for running machine learning models to avoid dependency conflicts. +You can either use a Virtual Environment (venv) or Conda for this purpose. + +Using Virtual Environment (venv) +-------------------------------- + +#. Install the virtual environment package: + + .. code-block:: bash + + $ sudo apt-get install python3-venv + +#. Create a virtual environment: + + .. code-block:: bash + + $ python3 -m venv ~/lerobot # Creates a venv "lerobot" in the home directory + +#. Activate the virtual environment: + + .. code-block:: bash + + $ source ~/lerobot/bin/activate + +Using Conda +----------- + +#. Create a virtual environment: + + .. code-block:: bash + + $ conda create -n lerobot python=3.10 + +#. Activate the virtual environment: + + .. code-block:: bash + + $ conda activate lerobot + +.. note:: + + Use either `venv` or `Conda` based on your preference, but **do not** mix them to avoid dependency issues. + +Clone Repository +================ + +For users of Aloha Stationary: + +#. Clone the LeRobot repository: + + .. code-block:: bash + + $ cd ~ + $ git clone https://github.com/huggingface/lerobot.git + +Build and Install LeRobot Models +================================ + +#. Build and install the LeRobot models from source: + + .. code-block:: bash + + $ cd lerobot && pip install -e . + +Teleoperation +============= + +To teleoperate your robot, follow these steps: + +#. Find the serial numbers of your robot's arms and cameras as described in the following documentation: + + - Arm Symlink Setup: `Aloha Post Install Arm Setup `_ + - Camera Setup: `Aloha Post Install Camera Setup `_ + +#. Update the serial numbers in the configuration file: :file:`lerobot/common/configs/robot/aloha.yaml` + +#. Run the teleoperation script: + + .. code-block:: bash + + $ python lerobot/scripts/control_robot.py teleoperate \ + --robot-path lerobot/configs/robot/aloha.yaml + + You will see logs that include information such as delta time (dt), frequency, and read/write times for the robot arms. + +#. You can control the teleoperation frequency using the :guilabel:`--fps` argument. For example, to set it to 30 FPS: + + .. code-block:: bash + + $ python lerobot/scripts/control_robot.py teleoperate \ + --robot-path lerobot/configs/robot/aloha.yaml --fps 30 + +Customizing Teleoperation with Hydra +------------------------------------- + +You can override the default YAML configurations dynamically using Hydra syntax. +For example, to change the USB ports of the leader and follower arms: + +.. code-block:: bash + + $ python lerobot/scripts/control_robot.py teleoperate \ + --robot-path lerobot/configs/robot/aloha.yaml \ + --robot-overrides \ + leader_arms.main.port=/dev/tty.usbmodem575E0031751 \ + follower_arms.main.port=/dev/tty.usbmodem575E0032081 + + +If you don't have any cameras connected, you can exclude them using Hydra's syntax: + +.. code-block:: bash + + $ python lerobot/scripts/control_robot.py teleoperate \ + --robot-path lerobot/configs/robot/aloha.yaml \ + --robot-overrides '~cameras' + +Recording Data Episodes +======================= + +The system supports episode-based data collection, where episodes are time-bounded sequences of robot actions. + +#. Control the recording flow with these arguments: + + - :guilabel:`--warmup-time-s`: Number of seconds for device warmup (default: 10s) + - :guilabel:`--episode-time-s`: Number of seconds per episode (default: 60s) + - :guilabel:`--reset-time-s`: Time for resetting after each episode (default: 60s) + - :guilabel:`--num-episodes`: Number of episodes to record (default: 50) + + Example: + + .. code-block:: bash + + $ python lerobot/scripts/control_robot.py record \ + --robot-path lerobot/configs/robot/aloha.yaml \ + --fps 30 \ + --root data \ + --repo-id ${HF_USER}/aloha_test \ + --tags tutorial \ + --warmup-time-s 5 \ + --episode-time-s 30 \ + --reset-time-s 30 \ + --num-episodes 2 + +.. note:: + + #. The :guilabel:`--num-episodes` defines the total number of episodes to be collected. + Therefore it will check the existing output directories for any previously recorded episodes and will start recording from the last recorded episode. + + #. The recorded data is pushed to hugging face hub by default you can set this false by using :guilabel:`--push_to_hub 0`. + + +.. note:: + + #. To push your dataset to Hugging Face's Hub, log in with a write-access token: + + .. code-block:: bash + + $ huggingface-cli login --token ${HUGGINGFACE_TOKEN} --add-to-git-credential + + #. Set your Hugging Face username as a variable for ease: + + .. code-block:: bash + + $ HF_USER=$(huggingface-cli whoami | head -n 1) + +Visualizing Datasets +==================== + +.. video:: ../videos/visualize_dataset.mp4 + :width: 1280 + :height: 720 + + +To visualize all the episodes recorded in your dataset, run: + +.. code-block:: bash + + $ python lerobot/scripts/visualize_dataset_html.py \ + --root data \ + --repo-id ${HF_USER}/aloha_test + +To visualize a single dataset episode from the Hugging Face Hub: + +.. code-block:: bash + + $ python lerobot/scripts/visualize_dataset.py \ + --repo-id ${HF_USER}/aloha_static_block_pickup \ + --episode-index 0 + +To visualize a single dataset episode stored locally: + +.. code-block:: bash + + $ DATA_DIR='./my_local_data_dir' python lerobot/scripts/visualize_dataset.py \ + --repo-id TrossenRoboticsCommunity/aloha_static_block_pickup \ + --episode-index 0 + +Replay Recorded Episodes +======================== + +Replaying episodes allows you to test the repeatability of the robot's actions. +To replay the first episode of your recorded dataset: + +.. code-block:: bash + + $ python lerobot/scripts/control_robot.py replay \ + --robot-path lerobot/configs/robot/aloha.yaml \ + --fps 30 \ + --root data \ + --repo-id ${HF_USER}/aloha_test \ + --episode 0 + +.. tip:: + + Use different :guilabel:`--fps` values to adjust the frequency of the robot actions. + + + +Training +======== + +To train a policy for controlling your robot, use the following command: + +.. code-block:: bash + + $ DATA_DIR=data python lerobot/scripts/train.py \ + dataset_repo_id=${HF_USER}/aloha_test \ + policy=act_aloha_real \ + env=aloha_real \ + hydra.run.dir=outputs/train/act_aloha_test \ + hydra.job.name=act_aloha_test \ + device=cuda \ + wandb.enable=true + +.. note:: + + The arguments are explained below: + + #. We provided the dataset with :guilabel:`dataset_repo_id=${HF_USER}/aloha_test`. + #. The policy is specified with :guilabel:`policy=act_aloha_real`. + This configuration is loaded from :file:`lerobot/configs/policy/act_aloha_real.yaml`. + #. The environment is set with :guilabel:`env=aloha_real`. + This configuration is loaded from :file:`lerobot/configs/env/aloha_real.yaml`. + #. The device is set to :guilabel:`cuda` to utilize an NVIDIA GPU for training. + #. :guilabel:`wandb.enable=true` is used for visualizing training plots via [Weights and Biases](https://docs.wandb.ai/quickstart). + Ensure you are logged in by running `wandb login`. + +Upload Policy Checkpoints +========================= + +Once training is complete, upload the latest checkpoint with: + +.. code-block:: bash + + $ huggingface-cli upload ${HF_USER}/act_aloha_test \ + outputs/train/act_aloha_test/checkpoints/last/pretrained_model + +To upload intermediate checkpoints: + +.. code-block:: bash + + $ CKPT=010000 + $ huggingface-cli upload ${HF_USER}/act_aloha_test_${CKPT} \ + outputs/train/act_aloha_test/checkpoints/${CKPT}/pretrained_model + +Evaluation +========== + +To control your robot with the trained policy and record evaluation episodes: + +.. code-block:: bash + + $ python lerobot/scripts/control_robot.py record \ + --robot-path lerobot/configs/robot/aloha.yaml \ + --fps 30 \ + --root data \ + --repo-id ${HF_USER}/eval_aloha_test \ + --tags tutorial eval \ + --warmup-time-s 5 \ + --episode-time-s 30 \ + --reset-time-s 30 \ + --num-episodes 10 \ + -p outputs/train/act_aloha_test/checkpoints/last/pretrained_model + + +This command is similar to the one used for recording training datasets, with a couple of key changes: + +#. The :guilabel:`-p` argument is now included, which specifies the path to your policy checkpoint (e.g., :guilabel:`-p outputs/train/eval_aloha_test/checkpoints/last/pretrained_model`). + You can also refer to the model repository on Hugging Face if you have uploaded a model checkpoint there (e.g., :guilabel:`-p ${HF_USER}/act_aloha_test`). + +#. The dataset name begins with :guilabel:`eval`, reflecting that you are running inference (e.g., :guilabel:`--repo-id ${HF_USER}/eval_aloha_test`). + + +You can visualize the evaluation dataset afterward using: + +.. code-block:: bash + + $ python lerobot/scripts/visualize_dataset.py \ + --root data \ + --repo-id ${HF_USER}/eval_aloha_test + + +Trossen Robotics Community +========================== + +Pretrained Models +----------------- + +You can download pretrained models from the Trossen Robotics Community on Hugging Face and use them for evaluation purposes. +To run evaluation on the pretrained models, use the following command: + +.. code-block:: bash + + python lerobot/scripts/control_robot.py record \ + --robot-path lerobot/configs/robot/aloha.yaml \ + --fps 30 \ + --root data \ + --repo-id ${HF_USER}/eval_aloha_test \ + --tags tutorial eval \ + --warmup-time-s 5 \ + --episode-time-s 30 \ + --reset-time-s 30 \ + --num-episodes 10 \ + -p ${HF_USER}/act_aloha_test + + +Datasets for Training and Augmentation +-------------------------------------- + +Datasets can also be downloaded from the Trossen Robotics Community on Hugging Face for further training or data augmentation. +These datasets can be used with your preferred network architectures. +Instructions for downloading and using these datasets can be found at the following link: + +`Dataset Download and Upload Instructions `_ + +`Trossen Robotics Community `_ + + + +Troubleshooting +=============== + +.. warning:: + If you encounter issues, follow these troubleshooting steps: + +#. **OpenCV Installation Issues (Linux)** + + If you encounter OpenCV installation issues, uninstall it via :guilabel:`pip` and reinstall using Conda: + + .. code-block:: bash + + pip uninstall opencv-python + conda install -c conda-forge opencv=4.10.0 + +#. **FFmpeg Encoding Error (`unknown encoder libsvtav1`)** + + Install FFmpeg with :guilabel:`libsvtav1` support via Conda-Forge or Homebrew: + + .. code-block:: bash + + $ conda install -c conda-forge ffmpeg + + Or: + + .. code-block:: bash + + $ brew install ffmpeg + +#. **Arrow Keys Not Working During Data Recording (Linux)** + + Ensure that the :guilabel:`$DISPLAY` environment variable is set correctly. + + +#. Checkout LeRobot Documentation for further help and details. + + `LeRobot Github `_ diff --git a/docs/videos/visualize_dataset.mp4 b/docs/videos/visualize_dataset.mp4 new file mode 100644 index 0000000..42ce8b5 Binary files /dev/null and b/docs/videos/visualize_dataset.mp4 differ