Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

[DanielYeh]

If you face this problem

2020-01-16 00:25:55.881381: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR2020-01-16 00:25:55.901231: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR2020-01-16 00:25:55.901300: W tensorflow/core/common_runtime/base_collective_executor.cc:216] BaseCollectiveExecutor::StartAbort Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

And you already make sure that you have been installed CUDA & Cudnn package correctly, the problems are maybe caused by Tensorflow.
Please refer to this

Solution:

Add this code at the beginning of the training code

for tf 1.xx
import tensorflow as tfconfig = tf.ConfigProto()

config.gpu_options.allow_growth = True

session = tf.InteractiveSession(config=config)

for tf 2.xx
import tensorflow as tf config = tf.compat.v1.ConfigProto()

config.gpu_options.allow_growth = True

session = tf.compat.v1.InteractiveSession(config=config)