improved linux and windows build scripts, updated readme (#348)

* improve linux build scripts * update jeston build script * update build scripts to run .bashrc from home * added --build-doc flag in linux scripts * update windows build scripts * remove sudo from cloud build scripts * update readme with build commands and documentation generation commands * Update build_dependencies_linux_cuda.sh * Update README.md * update windows build scripts to build with correct build type
Apra-Labs · Jul 24, 2024 · 62d7a07 · 62d7a07
1 parent 401b9eb
commit 62d7a07
Show file tree

Hide file tree

Showing 10 changed files with 155 additions and 94 deletions.
diff --git a/README.md b/README.md
@@ -7,7 +7,7 @@ A pipeline framework for developing video and image processing applications. Sup
 Learn more about ApraPipes here https://apra-labs.github.io/ApraPipes.
 
 ## Build status
-Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson Boards (Jetpack 4.4) and Windows (11) x64 Visual Studio 2017 Community.
+Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson Boards (Jetpack 4.4) and Windows (11) x64 Visual Studio 2019 Community.
 |OS|Version|With Cuda|Tests|Status|
 |--|-------|---------|------|------|
 |Windows|2019|No|[![Test Results](https://gist.githubusercontent.com/kumaakh/f80af234a4aabedc69af3ee197f66944/raw/badge_Windows.svg)](https://gist.githubusercontent.com/kumaakh/f80af234a4aabedc69af3ee197f66944/raw/badge_Windows.svg)|[![CI-Win-NoCUDA](https://github.com/Apra-Labs/ApraPipes/actions/workflows/CI-Win-NoCUDA.yml/badge.svg)](https://github.com/Apra-Labs/ApraPipes/actions/workflows/CI-Win-NoCUDA.yml)|
@@ -40,22 +40,24 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
 <details>
   <summary>Requirements</summary>
 
+  ###  Prerequisites
+  
+  ### Visual Studio
+  * Install Visual Studio 2019 Community 
+    * Install Desktop development C++
+    * .NET Desktop development
+    * Universal Windows Development Platform
+
   ### Cuda
   * Create an account on developer.nvidia.com if you're not already a member. Note : Otherwise the next step will show HTTP 404/403 error.
   * Windows 10/11 : [Cuda Toolkit 10.2](https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Windows&target_arch=x86_64&target_version=10&target_type=exelocal)  or  [CUDA Toolkit 11.8](https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Windows).
 
   ### Cudnn
   * Download [Cudnn](https://developer.nvidia.com/rdp/cudnn-archive#a-collapse765-102) and extract files where cuda is installed. Note: Please be aware that this process requires some effort. Here are the necessary steps:
-   * Download the correct zip file matching your cuda version. _Do not download the exe/installer/deb package._
-   * Windows: 
-     * Download [this file](https://developer.nvidia.com/compute/cudnn/secure/8.3.2/local_installers/10.2/cudnn-windows-x86_64-8.3.2.44_cuda10.2-archive.zip).
-
-  ###  Prerequisites 
-  * Install Visual Studio 2019 Community 
-    * Install Desktop development C++
-    * .NET Desktop development
-    * Universal Windows Development Platform
-  
+    * Download the correct zip file matching your cuda version. _Do not download the exe/installer/deb package._
+    * Windows: 
+      * Download [this file](https://developer.nvidia.com/compute/cudnn/secure/8.3.2/local_installers/10.2/cudnn-windows-x86_64-8.3.2.44_cuda10.2-archive.zip).
+    
   * Clone with submodules and LFS. 
     ```
     git clone --recursive https://github.com/Apra-Labs/ApraPipes.git
@@ -66,9 +68,9 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
 <details>
   <summary>Build</summary>
 
-  ### Build Without Cuda
   Open PowerShell as an administrator and execute the following commands
 
+  ### Build Without Cuda
   If your windows system does not have an NVIDIA GPU use this script
   ```
   build_windows_no_cuda.bat
@@ -77,7 +79,10 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
   ```
   build_windows_cuda.bat
   ```
-
+  ### To Build With Documentation
+  ```
+  build_windows_cuda.bat --build-doc
+  ```
 </details>
 
 <details>
@@ -114,6 +119,8 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
 <details>
   <summary>Requirements</summary>
   
+  ### Prerequisites
+
   ### Cuda
   * Create an account on developer.nvidia.com if you're not already a member. Note : Otherwise the next step will show HTTP 404/403 error.
   * Ubuntu 18.04/20.04:   
@@ -125,7 +132,8 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
   * Linux:
       * Download [this file](https://developer.nvidia.com/compute/cudnn/secure/8.3.2/local_installers/10.2/cudnn-linux-x86_64-8.3.2.44_cuda10.2-archive.tar.xz)
 
-  ### Prerequisites
+  * Clone with submodules and LFS.
+    ```
     git clone --recursive https://github.com/Apra-Labs/ApraPipes.git
     ```
 
@@ -141,11 +149,15 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
   ### Build Without Cuda
   If your windows system does not have an NVIDIA GPU use this script
   ```
-  sudo ./build_linux_no_cuda.sh
+  ./build_linux_no_cuda.sh
   ```
   ### Build With Cuda
   ```
-  sudo ./build_linux_cuda.sh
+  ./build_linux_cuda.sh
+  ```
+  ### To Build With Documentation
+  ```
+  ./build_linux_cuda.sh --build-doc
   ```
 
   Build can take ~2 hours depending on the machine configuration.
@@ -200,11 +212,14 @@ Aprapipes is automatically built and tested on Ubuntu (18.04 and 20.04), Jetson
   ```
   chmod +x build_jetson.sh
   ```
-  * ApraPipes builds CUDA version on Jerson Boads.
+  * ApraPipes builds CUDA version on Jetson Boards.
   ```
-  sudo ./build_jetson.sh
+  ./build_jetson.sh
+  ```
+  * To Build With Documentation
+  ```
+  ./build_jetson.sh --build-doc
   ```
-
   Build can take ~12 hours on Jetson Nano. 
   Note: Jetson build can also be done using Ubuntu 18.04 x86_64 Laptop via cross compilation.
 </details>
@@ -290,13 +305,10 @@ This build will be fairly fast (~10 mins) as entire vcpkg cache comes down with
 git submodule update --init --recursive
 ```
 ## Update Documentation
- After making changes to the documentation located in the /docs/source folder, it's essential to regenerate the documentation by following the provided steps. Once regenerated, commit the new content to ensure the latest documentation is up-to-date.
+To update documentation, refer to Documentation Guidelines in the [Contribution-Guidelines](https://github.com/Apra-Labs/ApraPipes/wiki/Contribution-Guidelines).
 
 ### To regenerate documentation
+Run,
+```
+./build_documentation.sh
 ```
-To build docs
-apt-install get python-sphinx 
-pip install sphinx-rtd-theme
-cd docs
-make html
-```
diff --git a/build_jetson.sh b/build_jetson.sh
@@ -1,4 +1,4 @@
-apt-get install clang-format
+sudo apt-get install clang-format
 clang-format -style=llvm -dump-config > .clang-format
 if ! command -v pip &> /dev/null; then
     # If pip is not available, download and install pip
@@ -9,10 +9,44 @@ pip install pre-commit
 pre-commit install
 
 chmod +x build_scripts/build_dependencies_jetson_cuda.sh
-./build_scripts/build_dependencies_jetson_cuda.sh
+sudo ./build_scripts/build_dependencies_jetson_cuda.sh
 
-chmod +x build_documentation.sh
-./build_documentation.sh
+if nvcc --version; then
+  USER_NAME=$(whoami)
+  TARGET_USER="$USER_NAME"
+  TARGET_HOME=$(eval echo ~$TARGET_USER)
+
+  # Append lines to the target user's ~/.bashrc
+  if ! grep -qxF 'export VCPKG_FORCE_SYSTEM_BINARIES=1' $TARGET_HOME/.bashrc; then
+    echo 'export VCPKG_FORCE_SYSTEM_BINARIES=1' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
+    echo "VCPKG_FORCE_SYSTEM_BINARIES flag added in .bashrc"
+  else
+    echo "VCPKG_FORCE_SYSTEM_BINARIES flag already exists in .bashrc"
+  fi
+
+  if ! grep -qxF 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' $TARGET_HOME/.bashrc; then
+    echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
+    echo "CUDA Binary Path added to .bashrc"
+  else
+    echo "CUDA Binary Path already exists in .bashrc"
+  fi
+
+  if ! grep -qxF 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' $TARGET_HOME/.bashrc; then
+    echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
+    echo "CUDA Library Path added to .bashrc"
+  else
+    echo "CUDA Library Path already exists in .bashrc"
+  fi
+
+  echo "Appended paths to ~/.bashrc and saved changes."
+  source ~/.bashrc
+  echo "Reloaded ~/.bashrc"
+fi
+
+if [[ $1 == "--build-doc" ]]; then
+    chmod +x build_documentation.sh
+    ./build_documentation.sh
+fi
 
 cd vcpkg
 ./bootstrap-vcpkg.sh

diff --git a/build_linux_cuda.sh b/build_linux_cuda.sh
@@ -1,4 +1,4 @@
-apt-get install clang-format
+sudo apt-get install clang-format
 clang-format -style=llvm -dump-config > .clang-format
 if ! command -v pip &> /dev/null; then
     # If pip is not available, download and install pip
@@ -11,8 +11,55 @@ pre-commit install
 chmod +x build_scripts/build_dependencies_linux_cuda.sh
 sudo ./build_scripts/build_dependencies_linux_cuda.sh
 
-chmod +x build_documentation.sh
-./build_documentation.sh
+if ! sudo nvcc --version &>/dev/null; then
+  USER_NAME=$(whoami)
+  cudnn_archives="/home/$USER_NAME/Downloads/cudnn-*.tar.xz"
+
+  for archive in $cudnn_archives; do
+    if [ -e "$archive" ]; then
+      extracted_folder="/home/$USER_NAME/Downloads/$(basename "$archive" .tar.xz)"
+      if [ ! -d "$extracted_folder" ]; then
+        echo "Extracting $archive..."
+        tar xf "$archive" -C "/home/$USER_NAME/Downloads/"
+      else
+        echo "Archive already extracted: $extracted_folder"
+      fi
+    fi
+  done
+
+  echo "Copying files..."
+  sudo cp -r /home/$USER_NAME/Downloads/cudnn-*/include/* /usr/local/cuda/include/
+  sudo cp -r /home/$USER_NAME/Downloads/cudnn-*/lib/* /usr/local/cuda/lib64/
+
+  TARGET_USER="$USER_NAME"
+  TARGET_HOME=$(eval echo ~$TARGET_USER)
+
+  # Append lines to the target user's ~/.bashrc
+  if ! grep -qxF 'export PATH=/usr/local/cuda/bin:${PATH}' $TARGET_HOME/.bashrc; then
+    echo 'export PATH=/usr/local/cuda/bin:${PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
+    echo "CUDA Binary Path added to .bashrc"
+  else
+    echo "CUDA Binary Path already exists in .bashrc"
+  fi
+
+  if ! grep -qxF 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}' $TARGET_HOME/.bashrc; then
+    echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
+    echo "CUDA Library Path added to .bashrc"
+  else
+    echo "CUDA Library Path already exists in .bashrc"
+  fi
+
+  # Reload .bashrc
+  source ~/.bashrc
+
+  echo "Appended line to ~/.bashrc and saved changes."
+  echo "Reloaded ~/.bashrc"
+fi
+
+if [[ $1 == "--build-doc" ]]; then
+    chmod +x build_documentation.sh
+    ./build_documentation.sh
+fi
 
 cd vcpkg
 ./bootstrap-vcpkg.sh

diff --git a/build_linux_no_cuda.sh b/build_linux_no_cuda.sh
@@ -1,4 +1,4 @@
-apt-get install clang-format
+sudo apt-get install clang-format
 clang-format -style=llvm -dump-config > .clang-format
 if ! command -v pip &> /dev/null; then
     # If pip is not available, download and install pip
@@ -9,13 +9,15 @@ pip install pre-commit
 pre-commit install
 
 chmod +x build_scripts/build_dependencies_linux_no_cuda.sh
-./build_scripts/build_dependencies_linux_no_cuda.sh
+sudo ./build_scripts/build_dependencies_linux_no_cuda.sh
 
 chmod +x base/fix-vcpkg-json.sh
 ./base/fix-vcpkg-json.sh true false false
 
-chmod +x build_documentation.sh
-./build_documentation.sh
+if [[ $1 == "--build-doc" ]]; then
+    chmod +x build_documentation.sh
+    ./build_documentation.sh
+fi
 
 cd vcpkg
 ./bootstrap-vcpkg.sh

diff --git a/build_scripts/build_dependencies_jetson_cuda.sh b/build_scripts/build_dependencies_jetson_cuda.sh
@@ -30,15 +30,4 @@ if [ ! -d "/usr/local/cuda/include" ] || [ ! -d "/usr/local/cuda/lib64" ]; then
   exit 1
 fi
 
-if  nvcc --version; then
-  TARGET_USER="$SUDO_USER"
-  TARGET_HOME=$(eval echo ~$TARGET_USER)
-  echo 'export VCPKG_FORCE_SYSTEM_BINARIES=1' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
-  echo 'export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
-  echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
-  echo "Appended paths to ~/.bashrc and saved changes."
-  source ~/.bashrc
-  echo "Reloaded ~/.bashrc"
-fi
-
 echo "Dependencies verified and installed successfully."
diff --git a/build_scripts/build_dependencies_linux_cuda.sh b/build_scripts/build_dependencies_linux_cuda.sh
@@ -47,33 +47,4 @@ if [ ! -d "/usr/local/cuda/include" ] || [ ! -d "/usr/local/cuda/lib64" ]; then
     exit 1
 fi
 
-if ! nvcc --version &>/dev/null; then
-  userName=$(whoami)
-  cudnn_archives="/home/$userName/Downloads/cudnn-*.tar.xz"
-
-  for archive in $cudnn_archives; do
-    if [ -e "$archive" ]; then
-      echo "Extracting $archive..."
-      tar xf "$archive" -C /home/$userName/Downloads/
-    fi
-  done
-
-  echo "Copying files..."
-  cp -r /home/$userName/Downloads/cudnn-*/include/* /usr/local/cuda/include/
-  cp -r /home/$userName/Downloads/cudnn-*/lib/* /usr/local/cuda/lib64/
-
-  TARGET_USER="$SUDO_USER"
-  TARGET_HOME=$(eval echo ~$TARGET_USER)
-
-  # Append lines to the target user's ~/.bashrc
-  echo 'export PATH=/usr/local/cuda/bin:${PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
-  echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH}' | sudo -u $TARGET_USER tee -a $TARGET_HOME/.bashrc
-
-  # Reload .bashrc
-  source $TARGET_HOME/.bashrc
-
-  echo "Appended line to ~/.bashrc and saved changes."
-  echo "Reloaded ~/.bashrc"
-fi
-
-echo "Dependencies verified and installed successfully."
+echo "Dependencies verified and installed successfully."
diff --git a/build_scripts/build_dependencies_linux_no_cuda.sh b/build_scripts/build_dependencies_linux_no_cuda.sh
@@ -19,8 +19,8 @@ done
 # If there are missing dependencies, install them
 if [ "${#missing_dependencies[@]}" -gt 0 ]; then
   echo "Installing missing dependencies..."
-   apt-get update -qq
-   apt-get -y install "${missing_dependencies[@]}"
+  apt-get update -qq
+  apt-get -y install "${missing_dependencies[@]}"
 fi
 
 # Install Meson if not present

diff --git a/build_scripts/build_dependencies_windows_cuda.ps1 b/build_scripts/build_dependencies_windows_cuda.ps1
@@ -22,10 +22,14 @@ if (-not (Test-Path "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\$cudaVer
 } else {
     $userName = $env:UserName
     $zipFilePath = "C:\Users\$userName\Downloads\cudnn-*.zip"
-
-    Write-Host "Extracting zip file..."
-
-    Expand-Archive -Path $zipFilePath -DestinationPath C:\Users\$userName\Downloads\ -Force
+    $extractedPath = "C:\Users\$userName\Downloads\cudnn-*\"
+
+    if (-not (Test-Path $extractedPath)) {
+        Write-Host "Extracting zip file..."
+        Expand-Archive -Path $zipFilePath -DestinationPath "C:\Users\$userName\Downloads\" -Force
+    } else {
+        Write-Host "Already extracted files found."
+    }
 
     Write-Host "Copying files..."
     Copy-Item -Path "C:\Users\$userName\Downloads\cudnn-*\include\*.h" -Destination "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\$cudaVersion\include\" -Recurse