FPGA Implementation of Visual Odometry using High-Level Synthesis

This repository contains the related work for the NTHU 11110EE 390000 / 11120EE 391000 Special Topic on Implementation (I)/(II) courses, specifically from Team A288.

The contributors are as follows:

李承澔
林奕杰
鄧文瑜
林妤諠
郭朝恩

Most research and implementations in the field of VO focus on specific algorithms such as Stereo Matching, Feature Extraction, Feature Tracking (Matching), and Motion Estimation. In this project, we propose that these four stages of the VO algorithm can be implemented and executed on the Xilinx Alveo™ U50 FPGA board, thereby exploring the possibility of accelerating these algorithms in comparison to traditional CPU-based approaches.

Abstract Development Flow and System Architecture

To begin, we selected a Python OpenCV tutorial on VO as a template and re-implemented it in C/C++ form with HLS synthesis accessibility. We removed all external library dependencies and eliminated CPU architecture-specific coding styles such as dynamic memory allocation and double-level pointers.

The VO algorithm is divided into four sub-algorithms: Stereo Matching, Feature Extraction, Feature Tracking, and Motion Estimation.

Each sub-algorithm corresponds to a kernel.

In order to properly verify the functionality of each kernel, we included four compile modes to run the VO program. In each compile mode, the corresponding kernel function runs on the programmable logic (PL) side of the FPGA, while the other three functions run on the processing system (PS) side (the host). This ensures that each individual kernel behaves correctly and allows us to compare the execution time between the FPGA and the CPU.

Toolchain and Prerequisites

The project has been tested in the following environment:

CPU: 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz
RAM: 49112592kB DDR4
OS: Ubuntu 20.04.4 LTS
FPGA: Xilinx Alveo™ U50 FPGA
Vitis Suite Version: 2022.1
OpenCV Version: 4.4.0

Prerequisites:

Ubuntu 20.04 (or a higher version certified with Vitis)
Xilinx Vitis Suite 2022.1 (or a higher version)
Xilinx® Runtime (XRT)
CMake 3.5+
OpenCV-4.4.0 x86 library (or a higher version certified with Vitis Vision Library)
libOpenCL.so
libpng library (optional)

Prepare the Environment

1. Xilinx Vitis Suite

Install necessary dependencies before Vitis installation:

$ sudo apt install libtinfo5 libncurses5 -y

And follow the offical installation guide.

2. Xilinx® Runtime (XRT)

Install XRT environment and set up variables.

Installing Xilinx Runtime and Platforms

Go to Alveo U50 Package File Downloads
Download and install them in order
Xilinx Runtime Deployment Target Platform Development Target Platform
Set up variables

Set up XILINX_VITIS and XILINX_VIVADO variables
```
$ source <Vitis_install_path>/Vitis/2022.2/settings64.sh 
```
Set up XILINX_XRT for data center platforms
```
$ source /opt/xilinx/xrt/setup.sh  
```

3. CMake

Install the latest CMake (Make sure cmake version>3.5 before proceeding).

Ubuntu cmake 編譯安裝 How to Install CMake on Ubuntu 20.04 LTS

Go to CMake File Downloads
Download the source distribution cmake-<latest-version>.tar.gz

*Remove previous CMake (If you have installed it before)
Remove cmake and cmake-qt-gui
```
$ sudo apt purge cmake
```
Remove cmake and other dependencies
```
$ sudo apt autoremove cmake
```

Install it
Extract it

$ tar -zxvf cmake-<latest-version>.tar.gz

Move to the extracted folder

$ cd cmake-<latest-version>

Run the following commands to compile and install

$ ./bootstrap
$ make
$ sudo make install

Check the installed CMace version

$ cmake --version

4. OpenCV-4.4.0 x86 library

Libraries should not be builded at shared folder!

Install OpenCV-Python in Ubuntu

Building OpenCV from source
Required build dependencies

# CMake to configure the installation, GCC for compilation, Python-devel and Numpy for building Python bindings etc.
$ sudo apt-get install cmake
$ sudo apt-get install gcc g++

# to support python3:
$ sudo apt-get install python3-dev python3-numpy

# GTK support for GUI features, Camera support (v4l), Media Support (ffmpeg, gstreamer) etc
$ sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
$ sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev

# to support gtk3:
$ sudo apt-get install libgtk-3-dev

We skip installation of 2 dependencies below: sudo apt-get install python-dev python-numpy sudo apt-get install libgtk2.0-dev

And install optional dependencies

$ sudo apt-get install libpng-dev
$ sudo apt-get install libjpeg-dev
$ sudo apt-get install libopenexr-dev
$ sudo apt-get install libtiff-dev
$ sudo apt-get install libwebp-dev

Download OpenCV-4.4.0 Repository
Download whole repository into a folder named as opencv.

Open a terminal window and navigate to the downloaded opencv folder. Create a new "build" folder and navigate to it.
```
$ mkdir build
$ cd build
```

Configuring and Installing
Configuration of OpenCV library build (executed from build folder)

$ cmake ../

OpenCV defaults assume Release build type and installation path is /usr/local.

You should see these lines in your CMake output (they mean that Python is properly found):

--   Python 3:
--     Interpreter:                 /usr/bin/python3.4 (ver 3.4.3)
--     Libraries:                   /usr/lib/x86_64-linux-gnu/libpython3.4m.so (ver 3.4.3)
--     numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.8.2)
--     packages path:               lib/python3.4/dist-packages

Now build the files

$ make
$ sudo make install

All files are installed in /usr/local/ folder. Open a terminal and try import cv2.

import cv2 as cv
print(cv.__version__)

5. libOpenCL.so

OpenCL Installable Client Driver Loader

Ubuntu
On Ubuntu the ICD library is packaged with the distribution. Install the following packages:

$ sudo apt-get install ocl-icd-libopencl1
$ sudo apt-get install opencl-headers
$ sudo apt-get install ocl-icd-opencl-dev

6. libpng library (optional)

Official Repository Installation Guide How to install libpng-1.6.37.tar.xz in ubuntu 20.04?

Download libpng Repository
Download whole repository into a temporary folder, or type GitHub CLI command under a temporary folder
```
$ gh repo clone glennrp/libpng
```

Configuring and Installing

$ cd libpng
$ ./autogen.sh
$ ./configure --prefix=/usr/local/libpng
$ make check
$ sudo make install

Appendix

< path-to-opencv-lib-folder > = /usr/local/lib < path-to-opencv-include-folder > = /usr/local/include/opencv4 < path-to-platform-directory >/< platform >.xpfm = /opt/xilinx/platforms/xilinx_u50_gen3x16_xdma_5_202210_1/xilinx_u50_gen3x16_xdma_5_202210_1.xpfm

Directory Structure of This Repository

 ${ThisRepository}
   |- SGBM
   |   |- c_src
   |   |- hardware
   |   \- program
   |
   |- feature-extraction
   |   |- c_src
   |   |- hardware
   |   \- program
   |
   |- feature-tracking
   |   |- c_src
   |   |- hardware
   |   \- program
   |
   |- motion-estimation
   |   |- c_src
   |   |- hardware
   |   \- program
   |
   |- host
   |   |- host-src
   |   |   |- C_FeatureExtraction  
   |   |   |- C_FeatureTracking
   |   |   |- C_MotionEstimation
   |   |   |- C_StereoMatching
   |   |   |- BasicFunction.cpp
   |   |   |- BasicFunction.hpp
   |   |   |- Host.cpp
   |   |   |- Host.hpp
   |   |   |- Matrix.h
   |   |   |- Parameter.hpp
   |   |   \- libpng.tar.gz
   |   |
   |   \- output.zip
   \- dataset.zip (Linked below)

Please download the dataset.zip from drive and unzip.

Quick Start (Build From CLI Script)

Build Bitstream from Each Kernel Function

Please reference the README.md in each kernel function(SGBM, feature-extraction, feature-tracking, motion-estimation) to build binary.

Setup Host Program Behavier

You can modify the behavior of the host program in Parameters.hpp. There are 3 sections that you can modify:

Include Section

In this section, you need to ensure that the header file of the pure C code is correct.
Macro Section
- You can complie the program in pure C code, only one kernel in HLS code, or all kernels in HLS code using 6 provided macros.
- You can print the program's progress on terminal with _INFO_
- You can write the output of the function to a .txt file with _PRINT_

Parameter Section

In this section, you are required to specify the path to the output folder on your computer while following the directory structure outlined below:

 ${ProjectFolder}
   |- ${HostFolder}
   |   |- src
   |   |   |- C_FeatureExtraction  
   |   |   |- C_FeatureTracking
   |   |   |- C_MotionEstimation
   |   \   \- C_StereoMatching
   |- output  
   |   |- C_FeatureExtraction  
   |   |- C_FeatureTracking
   |   |- C_MotionEstimation
   |   |- C_StereoMatching
   |   |- K_FeatureExtraction
   |   |- K_FeatureTracking
   |   |- K_MotionEstimation
   |   \- K_StereoMatching
   |- dataset
   |   |- poses
   \   \- sequences

Quick Start (Build From GUI)

Setup Vitis

Create an empty application project using xilinx_u50 as the platform.
Import the host code to the src folder under <ProjectName>[x86] and kernel code to the src folder under <ProjectName>_kernels in the Explorer panel.
Add kernel functions in the <ProjectName>_kernels.prj project editor.
Right-click <ProjectName>.prj and select Properties. Add the following includes and libraries in the Properties:
- C/C++ Build > Settings > Tool Setting > GCC Host Compiler(x86_64) > Includes
  
  Add the path <path-to-opencv-include-folder> to **Include paths(-l)*.
  
  If you have installed libpng library, please add the path <path-to-libpng-folder-where-you-installed> to Include paths(-l).
  Alternatively, if you choose to use the extracted libpng library, please add the path <path-to-libpng-folder-where-you-unzipped> to Include paths(-l)
- C/C++ Build > Settings > Tool Setting > GCC Host Linker(x86_64) > Libraries
  
  Add the path <path-to-opencv-lib-folder> to Library search path(-L).
  
  If you have installed libpng library, please add the path <path-to-libpng-folder-where-you-installed> to Include paths(-l).
  Alternatively, if you choose to use the extracted libpng library, please add the path <path-to-libpng-folder-where-you-unzipped> to Include paths(-l).
  
  Then add the following libraries:
  
  opencv_videoio opencv_imgcodecs opencv_core opencv_imgproc opencv_features2d opencv_flann opencv_video opencv_calib3d opencv_highgui png
  
  to Libraries(-l).
Click the Run button (green button at the top panel) to add configurations
- Double-click System Project Debug and click the generated SystemDebugger
- Edit Program Arguments and provide arguments as below:
```
<path-to-leftimage-folder> <path-to-rightimage-folder> <path-to-calib-file> <number-of-frames> <platform-vendor> <device-name>

e.g. 
/home/chngh/Desktop/Host/dataset/sequences/00/image_0/ /home/chngh/Desktop/Host/dataset/sequences/00/image_1/ /home/chngh/Desktop/Host/dataset/sequences/00/calib.txt 12 Xilinx xilinx_u50_gen3x16_xdma_5_202210_1
```
  Enable Automatically add binary container(s) to arguments
- Edit Configuration and enable OpenCL trace
Note:
- If you run the pure C code, please delete <platform-vendor> <device-name> arguments, and disable Automatically add binary container(s) to aruments
- When you run the HLS code host program, you can choose to load the .xclbin binary into FPGA directly, rather than regenerating .xclbin binary. If you want to load binary directly, please disable Automatically add binary container(s) to aruments, and add the <path-to-{KERNEL}.xclbin> as last argument in Program Arguments .

Setup Host Program Behavier

You can modify the behavior of the host program in Parameters.hpp. There are 3 sections that you can modify:

Include Section

In this section, you need to ensure that the header file of the pure C code is correct.
Macro Section
- You can complie the program in pure C code, only one kernel in HLS code, or all kernels in HLS code using 6 provided macros.
- You can print the program's progress on terminal with _INFO_
- You can write the output of the function to a .txt file with _PRINT_

Parameter Section

In this section, you need to verify that the name of the main function in your kernel is correct. Additionally, you are required to specify the path to the output folder on your computer while following the directory structure outlined below:

 ${ProjectFolder}
   |- ${HostFolder}
   |   |- src
   |   |   |- C_FeatureExtraction  
   |   |   |- C_FeatureTracking
   |   |   |- C_MotionEstimation
   |   \   \- C_StereoMatching
   |- output  
   |   |- C_FeatureExtraction  
   |   |- C_FeatureTracking
   |   |- C_MotionEstimation
   |   |- C_StereoMatching
   |   |- K_FeatureExtraction
   |   |- K_FeatureTracking
   |   |- K_MotionEstimation
   |   \- K_StereoMatching
   |- dataset
   |   |- poses
   \   \- sequences

Result

The output trajectory should be write to OUTPUT_FOLDER_PATH in parameter.hpp. And here's demo.

demo.mp4

trajectory of 2000 iterations generated with ground truth, self-contain C code, FPGA kernel.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

FPGA Implementation of Visual Odometry using High-Level Synthesis

Abstract Development Flow and System Architecture

Toolchain and Prerequisites

Prepare the Environment

1. Xilinx Vitis Suite

2. Xilinx® Runtime (XRT)

3. CMake

4. OpenCV-4.4.0 x86 library

5. libOpenCL.so

6. libpng library (optional)

Directory Structure of This Repository

Quick Start (Build From CLI Script)

Build Bitstream from Each Kernel Function

Setup Host Program Behavier

Quick Start (Build From GUI)

Setup Vitis

Setup Host Program Behavier

Result

Files

README.md

Latest commit

History

README.md

File metadata and controls

FPGA Implementation of Visual Odometry using High-Level Synthesis

Abstract Development Flow and System Architecture

Toolchain and Prerequisites

Prepare the Environment

1. Xilinx Vitis Suite

2. Xilinx® Runtime (XRT)

3. CMake

4. OpenCV-4.4.0 x86 library

5. libOpenCL.so

6. libpng library (optional)

Directory Structure of This Repository

Quick Start (Build From CLI Script)

Build Bitstream from Each Kernel Function

Setup Host Program Behavier

Quick Start (Build From GUI)

Setup Vitis

Setup Host Program Behavier

Result