Question

[nadongjin]

hello ~
I modified your code a little to randomly generate the mask.
Is there any problem with this code in your view?
#=====================================================
if random_cutout_mask_area > 0:
print(">>>> random_cutout_mask_area provided:", random_cutout_mask_area)
mask_height = img_shape[0] * 2 // 5
mask_func = lambda images, labels: (
tf.cond(
tf.random.uniform(()) < random_cutout_mask_area,
lambda: tf.concat([images[:68, :], tf.zeros_like(images[68:, :]) + 128], axis=0),
# lambda: tf.concat([images[:, :-mask_height], tf.zeros_like(images[:, -mask_height:]) + 128], axis=1),
lambda: images,
),
labels,
)
ds = ds.map(mask_func, num_parallel_calls=AUTOTUNE)

ds = ds.batch(batch_size, drop_remainder=True)

#=====================================================
-> result

And another strange thing happens to me.
In the upgraded version of "Tensorflow 2.8,
Loss is normally reduced as before, but this area of accuracy improves accuracy very slowly.
My tensorflow Version : 2.6

[leondgarse]

• No issue from my sight. Just maybe not a big difference applying this on an entire batch after bs.batch or before. But I didn't take tests on this, so it all depends on your own results.
• Which optimizer are you using? If AdamW, accuracy was not raising in some of my tests, even after first epoch. Just notice if the accuracy is raising or not in the first ~4000 batchs. For my training test on MS1MV3 with 93431 calsses, 1e-4 accuracy on batch 1000 is pretty normal.

[nadongjin]

Thank you for your answer.
I use nvidia-titan rtx 24G (2-GPU).
So far, I have never succeeded by learning with ADAM optimizer.
Even if you run the same epoch as sgd and sgdw, adam is too slow to improve.

[leondgarse]

Just made 2 checkins:

• move random_cutout_mask_area ahead of random_process_func adds more randomness to mask area cooperating with random_status.
• random_height for random_cutout_mask_area applies random height in range [0.55, 0.7].

ds, steps = data.prepare_dataset('/datasets/faces_casia_112x112_folders/', random_status=100, random_cutout_mask_area=0.5, batch_size=64)
data.show_batch_sample(ds)

[nadongjin]

As expected, you are the best.
It would be better if there was one slight correction.(This is just what I think)
How about not overlapping the mask and cutout?
If I'm unlucky, I think the two will overlap and cause a face that can't be fully recognized.

Wouldn't it affect my learning because of too hard image?
Personally, I think about making sure that only one of the masks and cutout is applied.

[leondgarse]

• OK, added here add random_cutout_or_cutout_mask. It's implemented by applying cutout separately from randaug. But currently it's commented off. You can enable it by using those 5 commented lines in data.py line 83 instead of the other 4 lines under them.

  # aa.available_ops = ["AutoContrast", "Equalize", "Color", "Contrast", "Brightness", "Sharpness"]
  # random_cutout = 1 / len(aa.available_ops)
  # self.process = lambda img: aa.distort(
  #     random_cutout_or_cutout_mask(tf.image.random_flip_left_right(img), img_shape, random_cutout_mask_area, random_cutout)
  # )

• IMO, image quality can be worse in real world, like someone with masks and glasses reflecting lights. Thus by default, I'm combining cutout with cutout_mask. May change it if testing show different results.

[nadongjin]

My Train Schedule

with tf.distribute.MirroredStrategy(devices=["/gpu:0", "/gpu:1"],cross_device_ops=tf.distribute .HierarchicalCopyAllReduce()).scope():
basic_model = efficientnet.EfficientNetV2S(input_shape=(112, 112, 3), num_classes=0, drop_connect_rate=0.2,
pretrained="imagenet", first_strides=1)
basic_model = models.buildin_models(basic_model, dropout=0.2, emb_shape=512, output_layer='F', bn_momentum=0.9,
bn_epsilon=1e-5, add_pointwise_conv=True, pointwise_conv_act="swish",
scale=True, use_bias=False)
basic_model = models.add_l2_regularizer_2_model(basic_model, weight_decay=5e-4, apply_to_batch_normal=False)

tt = train.Train(data_path, eval_paths=eval_paths,
                 save_path='20220110_sgd_effv2s.h5',
                 basic_model=basic_model, model=None, lr_base=0.1, lr_decay=0.5, lr_decay_steps=16, lr_min=1e-5,
                 batch_size=224, random_status=5,random_cutout_mask_area=0.3, eval_freq=1, output_weight_decay=1)

======================================================================================
1epoch : 12996/12996 [==============================] - 11971s 917ms/step - loss: 18.9451 - accuracy: 0.3554
2epoch : 12996/12996 [==============================] - 11935s 914ms/step - loss: 28.0782 - accuracy: 0.7461
3epoch : 12996/12996 [==============================] - 12082s 929ms/step - loss: 21.0604 - accuracy: 0.8394
4epoch : 12996/12996 [==============================] - 15514s 1s/step - loss: 19.3998 - accuracy: 0.8677
5epoch : 12996/12996 [==============================] - 13413s 1s/step - loss: 77.9566 - accuracy: 0.8745
6epoch : 12996/12996 [==============================] - 22763s 2s/step - loss: 34.2559 - accuracy: 0.9076
7epoch : 12996/12996 [==============================] - 28021s 2s/step - loss: 27.6001 - accuracy: 0.9238
8epoch : 12996/12996 [==============================] - 29280s 2s/step - loss: 24.8311 - accuracy: 0.9364

In my gpu learning model, learning time increases a lot from 6 epochs.
Is this normal?

[leondgarse]

• This happens to me when my system memory is limited, like when I ran training on a 64G total memory machine. Since I enlarged my system memory to 128G, also swap area to 64G, training time is rather stable for me.
• Another thing you may try is enbable XLA. Just start your ipython / jupyter-notebook / python script with TF_XLA_FLAGS="--tf_xla_auto_jit=2" python xxx.py.