Calculating training speeds

[michaelbornholdt]

"model": {
            "name": "efficientnet",
            "crop_generator": "sampled_crop_generator",
            "metrics": ["accuracy", "top_k"],
            "epochs": 10,
            "initialization":"ImageNet",
            "params": {
                "learning_rate": 0.005,
                "batch_size": 32,
                "conv_blocks": 0,
                "feature_dim": 256,
                "pooling": "avg"
            },
 "validation": {
            "frequency": 2,
            "top_k": 5,
            "batch_size": 32,
            "frame": "val",
            "sample_first_crops": true
        }

Output

Epoch 00001: LearningRateScheduler setting learning rate to 0.001.
2021-07-22 11:02:41.058623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-07-22 11:02:44.932000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
     1/171641 [..............................] - ETA: 2337:41:53 - loss: 6.7107 - acc: 0.0000e+00 - top_5: 0.0000e+00COMET INFO: Ignoring automatic log_metric('batch_batch') because 'keras:batch_batch' is in COMET_LOGGING_METRICS_IGNORE
COMET INFO: Ignoring automatic log_metric('batch_size') because 'keras:batch_size' is in COMET_LOGGING_METRICS_IGNORE
  9913/171641 [>.............................] - ETA: 7:12:14 - loss: 6.0701 - acc: 0.0171 - top_5: 0.0646/home/ubuntu/dp/DeepProfiler/deepprofiler/imaging/cropping.py:48: RuntimeWarning:

invalid value encountered in true_divide

108305/171641 [=================>............] - ETA: 2:42:55 - loss: nan - acc: 0.0029 - top_5: 0.00592021-07-22 15:41:05,745 - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
122029/171641 [====================>.........] - ETA: 2:07:29 - loss: nan - acc: 0.0027 - top_5: 0.00522021-07-22 16:16:04,505 - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
132062/171641 [======================>.......] - ETA: 1:41:48 - loss: nan - acc: 0.0026 - top_5: 0.00482021-07-22 16:42:11,980 - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
132149/171641 [======================>.......] - ETA: 1:41:35 - loss: nan - acc: 0.0026 - top_5: 0.00482021-07-22 16:42:26,080 - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
132267/171641 [======================>.......] - ETA: 1:41:17 - loss: nan - acc: 0.0026 - top_5: 0.00482021-07-22 16:42:44,112 - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
171641/171641 [==============================] - 26822s 156ms/step - loss: nan - acc: 0.0024 - top_5: 0.0037
Epoch 2/10

Epoch 00002: LearningRateScheduler setting learning rate to 0.00125.
171641/171641 [==============================] - 26525s 155ms/step - loss: nan - acc: 0.0015 - top_5: 0.0000e+00 - val_loss: nan - val_acc: 0.0018 - val_top_5: 0.0000e+00
Epoch 3/10

Epoch 00003: LearningRateScheduler setting learning rate to 0.0016666666666666668.
171641/171641 [==============================] - 26190s 153ms/step - loss: nan - acc: 0.0015 - top_5: 0.0000e+00 - val_loss: nan - val_acc: 0.0018 - val_top_5: 0.0000e+00
Epoch 4/10

Epoch 00004: LearningRateScheduler setting learning rate to 0.0025.
 18718/171641 [==>...........................] - ETA: 6:33:07 - loss: nan - acc: 0.0014 - top_5: 0.0000e+002021-07-23 09:56:18,385 - WARNING - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
 18811/171641 [==>...........................] - ETA: 6:32:51 - loss: nan - acc: 0.0014 - top_5: 0.0000e+002021-07-23 09:56:32,497 - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='www.comet.ml', port=443): Read timed out. (read timeout=10)",)': /clientlib/status-report/update
 23932/171641 [===>..........................] - ETA: 6:16:03 - loss: nan - acc: 0.0014 - top_5: 0.0000e+00