No checkpoints found and Node: MaxBytesInUse = MaxBytesInUse

[macjr-cmd]

Hello,
I did not trained the model but I used the pretrained model from setup.py (mini-batches and clusters have been generated).
First I had a wavedata issue which I solved by following the instructions in the related project (https://github.com/kujason/avod). Then copied the missing files into my TLNET folder and the error was gone. Then I had the next issue (below) when I run either evaluation or inference. I followed the instructions as in Issue #132 (kujason/avod#132) by copying files from avod/models and pasting them in a new directory: avod/data/outputs/pyramid_cars_with_aug_example/checkpoints and the config file in avod/data/outputs/pyramid_cars_with_aug_example. I also uninstalled and reinstalled tensorflow. pip install tensorflow==1.3.0 and pip install tensorflow-gpu==1.3.0. I still haven't escaped the issue below.

I keep having this issue:
(Tlnetconda) emmanuel@karim:/media/emmanuel/Daten/TLNet$ python avod/experiments/run_evaluation.py --pipeline_config=avod/configs/pyramid_cars_with_aug_example.config --data_split='val' --device=0
/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:458: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:459: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:460: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:461: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:462: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:465: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2021-05-19 15:55:35.612134: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2021-05-19 15:55:35.612159: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2021-05-19 15:55:35.612166: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2021-05-19 15:55:35.612183: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2021-05-19 15:55:35.612192: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
No checkpoints found
INFO:tensorflow:Restoring parameters from /pyramid_cars_with_aug_example
Traceback (most recent call last):
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
return fn(*args)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1297, in _run_fn
self._extend_graph()
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1358, in _extend_graph
self._session, graph_def.SerializeToString(), status)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/contextlib.py", line 89, in exit
next(self.gen)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'MaxBytesInUse' with these attrs. Registered devices: [CPU], Registered kernels:
device='GPU'

 [[Node: MaxBytesInUse = MaxBytesInUse[]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "avod/experiments/run_evaluation.py", line 130, in
tf.app.run()
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "avod/experiments/run_evaluation.py", line 126, in main
evaluate(model_config, eval_config, dataset_config)
File "avod/experiments/run_evaluation.py", line 85, in evaluate
model_evaluator.run_latest_checkpoints()
File "/media/emmanuel/Daten/TLNet/avod/core/evaluator.py", line 384, in run_latest_checkpoints
self.run_checkpoint_once(checkpoint_to_restore)
File "/media/emmanuel/Daten/TLNet/avod/core/evaluator.py", line 144, in run_checkpoint_once
self._saver.restore(self._sess, checkpoint_to_restore)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'MaxBytesInUse' with these attrs. Registered devices: [CPU], Registered kernels:
device='GPU'

 [[Node: MaxBytesInUse = MaxBytesInUse[]()]]

Caused by op 'MaxBytesInUse', defined at:
File "avod/experiments/run_evaluation.py", line 130, in
tf.app.run()
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "avod/experiments/run_evaluation.py", line 126, in main
evaluate(model_config, eval_config, dataset_config)
File "avod/experiments/run_evaluation.py", line 80, in evaluate
eval_config)
File "/media/emmanuel/Daten/TLNet/avod/core/evaluator.py", line 133, in init
tf.contrib.memory_stats.MaxBytesInUse())
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/contrib/memory_stats/python/ops/memory_stats_ops.py", line 36, in MaxBytesInUse
return gen_memory_stats_ops.max_bytes_in_use()
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/contrib/memory_stats/ops/gen_memory_stats_ops.py", line 41, in max_bytes_in_use
result = _op_def_lib.apply_op("MaxBytesInUse", name=name)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/emmanuel/anaconda3/envs/Tlnetconda/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): No OpKernel was registered to support Op 'MaxBytesInUse' with these attrs. Registered devices: [CPU], Registered kernels:
device='GPU'

 [[Node: MaxBytesInUse = MaxBytesInUse[]()]]

The two main recurring errors are: No checkpoints found and Node: MaxBytesInUse = MaxBytesInUse. Sorry for the long text and thanks for your time!