Docs - Updates

ROCm · Apr 9, 2024 · dcfbe48 · dcfbe48
1 parent 4170d1a
commit dcfbe48
Show file tree

Hide file tree

Showing 36 changed files with 134 additions and 3,495 deletions.
diff --git a/docs/README.md b/docs/README.md
@@ -1,3 +1,5 @@
+<p align="center"><img width="70%" src="data/rocAL_logo.png" /></p>
+
 # rocAL User Guide
 
 Today’s deep learning applications require loading and pre-processing data efficiently to achieve high processing throughput.  This requires creating efficient processing pipelines fully utilizing the underlying hardware capabilities. Some examples are load and decode data, do a variety of augmentations, color-format conversions, etc.
@@ -7,12 +9,12 @@ AMD ROCm Augmentation Library (rocAL) is designed to efficiently do such process
 These pipelines are programmable by the user using both C++ and Python APIs.
 
 ## User Guide Chapters
-* [Chapter 1 - Overview](user_guide/ch1.md)
-* [Chapter 2 - Architecture Components](user_guide/ch2.md)
-* [Chapter 3 - Installation](user_guide/ch3.md)
-* [Chapter 4 - Using with Python API](user_guide/ch4.md)
-* [Chapter 5 - Framework Integration](user_guide/ch5.md)
-* [Chapter 6 - Using with C++ API](user_guide/ch6.md)
+- [Chapter 1 - Overview](user_guide/ch1.md)
+- [Chapter 2 - Architecture Components](user_guide/ch2.md)
+- [Chapter 3 - Installation](user_guide/ch3.md)
+- [Chapter 4 - Using with Python API](user_guide/ch4.md)
+- [Chapter 5 - Framework Integration](user_guide/ch5.md)
+- [Chapter 6 - Using with C++ API](user_guide/ch6.md)
 
 ## Key Components of rocAL
 
@@ -33,6 +35,7 @@ Refer [rocAL build instructions](https://github.com/ROCm/rocAL#build-instruction
 ## rocAL Python
 
 *   rocAL Python package has been created using Pybind11 which enables data transfer between rocAL C++ API and Python API.
+*   Module imports are made similar to other data loaders like NVidia's DALI.
 *   rocal_pybind package has both PyTorch and TensorFlow framework support.
 *   Various reader format support including FileReader, COCOReader, and TFRecordReader.
 *   example folder contains sample implementations for each reader variation as well as sample training script for PyTorch
@@ -81,27 +84,25 @@ amd.rocal.types are enums exported from C++ API to python. Some examples include
 *  From the above classes, any hybrid iterator pipeline can be created by adding augmentations.
 *  see example [PyTorch Simple Example](./examples). Requires PyTorch.
 
-### installing rocAL python plugin (Python 3.9+)
+### installing rocAL python plugin (Python 3.6)
 
 *  Build and install RPP
-*  Build and install MIVisionX
-*  Build and install [rocAL](https://github.com/ROCm/rocAL/)
+*  Build and install MIVisionX which installs rocAL c++ lib
+*  Go to [rocal_pybind](https://github.com/ROCm/rocAL/tree/master/rocAL_pybind) folder
+*  sudo ./run.sh
 
 ### Steps to run MLPerf Resnet50 classification training with rocAL on a system with MI50+ and ROCm
 
 * Step 1: Ensure you have downloaded ILSVRC2012_img_val.tar (6.3GB) and ILSVRC2012_img_train.tar (138 GB) files and unzip into train and val folders
-* Step 2: Build [MIVisionX Pytorch docker](../docker/README.md)
+* Step 2: Build [MIVisionX Pytorch docker](https://github.com/ROCm/rocAL/blob/master/docker/README.md)
 * Step 3: Install rocAL python_pybind plugin as described above
 * Step 4: Clone [MLPerf](https://github.com/rrawther/MLPerf-mGPU) branch and checkout mlperf-v1.1-rocal branch
-
-``` bash
+```
 git clone -b mlperf-v1.1-rocal https://github.com/rrawther/MLPerf-mGPU
 ```
-
 * Step 5: Modify RN50_AMP_LARS_8GPUS_NCHW.sh or RN50_AMP_LARS_8GPUS_NHWC.sh to reflect correct path for imagenet directory
 * Step 8: Run RN50_AMP_LARS_8GPUS_NCHC.sh or RN50_AMP_LARS_8GPUS_NHWC.sh
-
-``` bash
+```
 ./RN50_AMP_LARS_8GPUS_NCHW.sh 
 (or)
 ./RN50_AMP_LARS_8GPUS_NHWC.sh
@@ -111,11 +112,9 @@ git clone -b mlperf-v1.1-rocal https://github.com/rrawther/MLPerf-mGPU
 
 * Refer to the [docker](https://github.com/ROCm/MIVisionX#docker) page for prerequisites and information on building the docker
 * Step 1: Run the docker image*
-
-``` bash
+````
 sudo docker run -it -v <Path-To-Data-HostSystem>:/data -v /<Path-to-GitRepo>:/dockerx -w /dockerx --privileged --device=/dev/kfd --device=/dev/dri --group-add video --shm-size=4g --ipc="host" --network=host <docker-name>
-```
-
+````
   * Optional: Map localhost directory on the docker image
     * option to map the localhost directory with imagenet dataset folder to be accessed on the docker image.
     * usage: -v {LOCAL_HOST_DIRECTORY_PATH}:{DOCKER_DIRECTORY_PATH}
diff --git a/docs/data/amd-epyc-black-resize.png b/docs/data/amd-epyc-black-resize.png
diff --git a/docs/data/rocAL_logo.png b/docs/data/rocAL_logo.png
diff --git a/docs/data/rocm-black-resize.png b/docs/data/rocm-black-resize.png
diff --git a/docs/doxygen/Doxyfile b/docs/doxygen/Doxyfile
@@ -951,16 +951,7 @@ INPUT                  = ../../README.md \
                         ../../rocAL/include/api/rocal_api_meta_data.h \
                         ../../rocAL/include/api/rocal_api_parameters.h \
                         ../../rocAL/include/api/rocal_api_types.h \
-                        ../../rocAL/include/api/rocal_api.h \
-                        ../../rocAL_pybind/amd/rocal/decoders.py \
-                        ../../rocAL_pybind/amd/rocal/fn.py \
-                        ../../rocAL_pybind/amd/rocal/pipeline.py \
-                        ../../rocAL_pybind/amd/rocal/random.py \
-                        ../../rocAL_pybind/amd/rocal/readers.py \
-                        ../../rocAL_pybind/amd/rocal/types.py \
-                        ../../rocAL_pybind/amd/rocal/plugin/generic.py \
-                        ../../rocAL_pybind/amd/rocal/plugin/pytorch.py \
-                        ../../rocAL_pybind/amd/rocal/plugin/tf.py
+                        ../../rocAL/include/api/rocal_api.h
 
 
 # This tag can be used to specify the character encoding of the source files

diff --git a/docs/examples.md b/docs/examples.md
diff --git a/docs/examples.rst b/docs/examples.rst
@@ -12,6 +12,4 @@ Use the links below to see more examples:
 
 * `Image Processing <https://github.com/ROCm/rocAL/tree/master/docs/examples/image_processing>`_ 
 * `Pytorch <https://github.com/ROCm/rocAL/tree/master/docs/examples/pytorch>`_ 
-* `Tensorflow <https://github.com/ROCm/rocAL/tree/master/docs/examples/tf>`_ 
-* `Jupyter Notebooks <https://github.com/ROCm/rocAL/tree/master/docs/examples/notebooks>`_ 
 
diff --git a/docs/examples/image_processing/decoder.py b/docs/examples/image_processing/decoder.py
@@ -0,0 +1,61 @@
+
+import sys
+from amd.rocal.pipeline import pipeline_def
+from amd.rocal.plugin.generic import ROCALClassificationIterator
+import amd.rocal.fn as fn
+import amd.rocal.types as types
+import matplotlib.gridspec as gridspec
+import matplotlib.pyplot as plt
+import cupy as cp
+
+seed = 1549361629
+image_dir = "../../../../data/images/AMD-tinyDataSet/"
+batch_size = 4
+gpu_id = 0
+
+def show_images(image_batch, device):
+    columns = 4
+    rows = (batch_size + 1) // (columns)
+    #fig = plt.figure(figsize = (32,(32 // columns) * rows))
+    gs = gridspec.GridSpec(rows, columns)
+    for j in range(rows*columns):
+        #print('\n Display image: ', j)
+        plt.subplot(gs[j])
+        img = image_batch[j]
+        plt.axis("off")
+        if device == "cpu":
+            plt.imshow(img)
+        else:
+            plt.imshow(cp.asnumpy(img))
+    plt.show()
+
+
+def show_pipeline_output(pipe, device):
+    pipe.build()
+    data_loader = ROCALClassificationIterator(pipe, device)
+    images = next(iter(data_loader))
+    show_images(images[0], device)
+
+@pipeline_def(seed=seed)
+def image_decoder_pipeline(device="cpu", path=image_dir):
+    jpegs, labels = fn.readers.file(file_root=path, shard_id=0, num_shards=1, random_shuffle=False)
+    images = fn.decoders.image(jpegs, file_root=path, device=device, output_type=types.RGB, shard_id=0, num_shards=1, random_shuffle=False)
+    return fn.resize(images, device=device, resize_x=300, resize_y=300)
+
+def main():
+    print ('Optional arguments: <cpu/gpu image_folder>')
+    bs = batch_size
+    rocal_device = "cpu"
+    img_folder = image_dir
+    if  len(sys.argv) > 1:
+      if(sys.argv[1] == "gpu"):
+          rocal_device = "gpu"
+    if  len(sys.argv) > 2:
+      img_folder = sys.argv[2]
+
+    pipe = image_decoder_pipeline(batch_size=bs, num_threads=1, device_id=gpu_id, rocal_cpu=True, tensor_layout=types.NHWC,
+                                  reverse_channels=True, mean = [0, 0, 0], std=[255, 255, 255], device=rocal_device, path=img_folder)
+    show_pipeline_output(pipe, device=rocal_device)
+
+if __name__ == '__main__':
+    main()
diff --git a/...examples/notebooks/decoder_examples.ipynb → ...s/image_processing/decoder_examples.ipynb b/...examples/notebooks/decoder_examples.ipynb → ...s/image_processing/decoder_examples.ipynb
@@ -24,7 +24,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -38,7 +38,7 @@
     "%matplotlib inline\n",
     "\n",
     "seed = 1549361629\n",
-    "image_dir = \"../../../data/images/AMD-tinyDataSet/\"\n",
+    "image_dir = \"../../../../data/images/AMD-tinyDataSet/\"\n",
     "batch_size = 4\n",
     "gpu_id = 0\n",
     "\n",
@@ -61,7 +61,7 @@
     "    pipe.build()\n",
     "    data_loader = ROCALClassificationIterator(pipe, device, device_id)\n",
     "    images = next(iter(data_loader))\n",
-    "    show_images(images[0][0], device)\n"
+    "    show_images(images[0], device)\n"
    ]
   },
   {
@@ -82,9 +82,9 @@
    "source": [
     "@pipeline_def(seed=seed)\n",
     "def image_decoder_pipeline(device=\"cpu\"):\n",
-    "    jpegs, labels = fn.readers.file(file_root=image_dir)\n",
+    "    jpegs, labels = fn.readers.file(file_root=image_dir, shard_id=0, num_shards=1, random_shuffle=False)\n",
     "    images = fn.decoders.image(jpegs, file_root=image_dir, device=device, output_type=types.RGB, shard_id=0, num_shards=1, random_shuffle=False)\n",
-    "    return fn.resize(images, device=device, resize_width=300, resize_height=300)\n",
+    "    return fn.resize(images, device=device, resize_x=300, resize_y=300)\n",
     "\n",
     "pipe = image_decoder_pipeline(batch_size=batch_size, num_threads=1, device_id=gpu_id, rocal_cpu=True, tensor_layout=types.NHWC, \n",
     "                            reverse_channels=True, mean = [0, 0, 0], std=[255,255,255], device=\"cpu\")\n",
@@ -109,13 +109,12 @@
    "source": [
     "@pipeline_def(seed=seed)\n",
     "def image_decoder_random_crop_pipeline(device=\"cpu\"):\n",
-    "    jpegs, labels = fn.readers.file(file_root=image_dir)\n",
+    "    jpegs, labels = fn.readers.file(file_root=image_dir, shard_id=0, num_shards=1, random_shuffle=False)\n",
     "    images = fn.decoders.image_slice(jpegs, file_root=image_dir, \n",
+    "                                     device=device,\n",
     "                                     output_type=types.RGB,\n",
-    "                                     shard_id = 0,\n",
-    "                                     num_shards = 1,\n",
     "                                     random_shuffle=True)\n",
-    "    return fn.resize(images, device=device, resize_width=300, resize_height=300)\n",
+    "    return fn.resize(images, device=device, resize_x=300, resize_y=300)\n",
     "    \n",
     "pipe = image_decoder_random_crop_pipeline(batch_size=batch_size, num_threads=1, device_id=gpu_id, rocal_cpu=True, tensor_layout=types.NHWC, \n",
     "                                          reverse_channels=True, mean=[0,0,0], std = [255,255,255], device=\"cpu\")\n",
@@ -185,7 +184,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.10.6"
   },
   "vscode": {
    "interpreter": {

diff --git a/docs/examples/image_processing/inference_pipeline.py b/docs/examples/image_processing/inference_pipeline.py
@@ -31,7 +31,7 @@
 
 
 seed = 1549361629
-image_dir = "../../../data/images/AMD-tinyDataSet/"
+image_dir = "../../../../data/images/AMD-tinyDataSet/"
 batch_size = 4
 gpu_id = 0