build tensorflow_rocm

[xuhuisheng]

clone source

git clone https://github.com/ROCmSoftwarePlatform/tensorflow-upstream

switch r2.9-rocm-enhanced branch

git checkout r2.9-rocm-enhanced

install bazel
https://github.com/bazelbuild/bazel

run build script

./build_rocm_python3

[LamEnder]

hmm, I also want to make sure it can run an inference test successfully with the expected result.

Did you have your own build with the test results available?

[xuhuisheng]

here is my tf-2.8 on gfx803
https://github.com/xuhuisheng/rocm-gfx803/releases/download/rocm500/tensorflow_rocm-2.8.0-cp38-cp38-linux_x86_64.whl

mnist is fine

[LamEnder]

Thank you, now I can confirm that tensorflow will still work with gfx803 with latest 5.1.3 toolchain.

But isn't that tensorflow binary was built using ROCm 5.0.0 toolchain?

[xuhuisheng]

Yeah, since it costs about 3 hours for me to build tf from sources, I usually build one binary for one tf version. Even rocm had upgrade to new version, like 5.1.3, the tf whl could run properly, only need export LD_LIBRARY_PATH=/opt/rocm

[LamEnder]

ok @xuhuisheng, here's my output for the mnist test in tensorflow:

Python 3.8.13 (default, Jun 19 2022, 02:41:46) 
[GCC 12.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> mnist = tf.keras.datasets.mnist
>>> 
>>> (x_train, y_train),(x_test, y_test) = mnist.load_data()
>>> x_train, x_test = x_train / 255.0, x_test / 255.0
>>> 
>>> model = tf.keras.models.Sequential([
...   tf.keras.layers.Flatten(input_shape=(28, 28)),
...   tf.keras.layers.Dense(128, activation='relu'),
...   tf.keras.layers.Dropout(0.2),
...   tf.keras.layers.Dense(10, activation='softmax')
... ])
2022-06-19 09:53:06.982471: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.337685: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.337791: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.338246: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-06-19 09:53:24.338964: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.339116: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.339188: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.359442: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.359553: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.359632: I tensorflow/stream_executor/rocm/rocm_gpu_executor.cc:832] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-06-19 09:53:24.359690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3596 MB memory:  -> device: 0, name: AMD Radeon RX Graphics, pci bus id: 0000:01:00.0
>>> 
>>> model.compile(optimizer='adam',
...               loss='sparse_categorical_crossentropy',
...               metrics=['accuracy'])
>>> 
>>> model.fit(x_train, y_train, epochs=5)
2022-06-19 09:53:37.583711: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 188160000 exceeds 10% of free system memory.
2022-06-19 09:53:38.139488: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 188160000 exceeds 10% of free system memory.
2022-06-19 09:53:38.367044: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2022-06-19 09:53:38.372238: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2022-06-19 09:53:38.376331: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
Epoch 1/5
2022-06-19 09:53:38.756101: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
1875/1875 [==============================] - 9s 3ms/step - loss: 0.2980 - accuracy: 0.9134 
Epoch 2/5
1875/1875 [==============================] - 6s 3ms/step - loss: 0.1459 - accuracy: 0.9566
Epoch 3/5
1875/1875 [==============================] - 6s 3ms/step - loss: 0.1088 - accuracy: 0.9668
Epoch 4/5
1875/1875 [==============================] - 6s 3ms/step - loss: 0.0884 - accuracy: 0.9729
Epoch 5/5
1875/1875 [==============================] - 6s 3ms/step - loss: 0.0780 - accuracy: 0.9756
<keras.callbacks.History object at 0x7feb8247c2e0>
>>> model.evaluate(x_test, y_test)
2022-06-19 09:54:12.656082: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2022-06-19 09:54:12.662765: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2022-06-19 09:54:12.667800: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
2022-06-19 09:54:12.805647: I tensorflow/core/common_runtime/gpu_fusion_pass.cc:507] ROCm Fusion is enabled.
313/313 [==============================] - 1s 3ms/step - loss: 0.0736 - accuracy: 0.9792
[0.07360884547233582, 0.979200005531311]
>>> 

Do you think NUMA node read returning -1 from SysFS is fine?
I tried to search for this but I only got the results for CUDA tensorflow.