Add AdaBelief optimizer [https://arxiv.org/abs/2010.07468]

[TimbusCalin]

Describe the feature and the current behavior/state.

Relevant information

• Are you willing to contribute it (yes/no): no
• Are you willing to maintain it going forward? (yes/no): no
• Is there a relevant academic paper? https://arxiv.org/abs/2010.07468
• Is there already an implementation in another framework? https://github.com/juntang-zhuang/Adabelief-Optimizer
• Was it part of tf.contrib? No

**Which API type would this fall under (layer, metric, optimizer, etc.): Optimizer/Layer

Who will benefit with this feature? Everybody; shows better results/more promising ones than the other optimizers

Any other info.

[AakashKumarNain]

Although Adabelief looks interesting but we will wait for some time before adding it here. Why? Many optimizers showed improvements over Adam in the past as well but they failed to deliver that performance when tested on a wide variety of datasets. Once we get more insights about the performance, we will start the process of adding it to addons with a PR

[TimbusCalin]

Yes, I agree with your statement.

[bhack]

Yes see a recent work like https://arxiv.org/abs/2007.01547

[bhack]

TF reference implementation at https://github.com/juntang-zhuang/Adabelief-Optimizer/blob/master/TensorFlow_Experiments/AdaBelief_tf.py

[dathudeptrai]

TF reference implementation at https://github.com/juntang-zhuang/Adabelief-Optimizer/blob/master/TensorFlow_Experiments/AdaBelief_tf.py

Seems that implementation missed many features, hope this optimizer included in tf_addons soon :D.

The adabelief-tf is a naive implementation in Tensorflow. It lacks many features such as decoupled weight decay, and is not extensively tested. Currently I don't have plans to improve it since I seldom use Tensorflow, please contact me if you want to collaborate and improve it.

[juntang-zhuang]

TF reference implementation at https://github.com/juntang-zhuang/Adabelief-Optimizer/blob/master/TensorFlow_Experiments/AdaBelief_tf.py

Seems that implementation missed many features, hope this optimizer included in tf_addons soon :D.

The adabelief-tf is a naive implementation in Tensorflow. It lacks many features such as decoupled weight decay, and is not extensively tested. Currently I don't have plans to improve it since I seldom use Tensorflow, please contact me if you want to collaborate and improve it.

Hi, we have updated adabelief-tf==0.1.0 based on tensorflow-addons, which is also available by pip, and supports decoupled weight decay and rectification, and supports Tensorflow>=2.0 and Keras.
See the source code for adabelief-tf==0.1.0 here:
https://github.com/juntang-zhuang/Adabelief-Optimizer/blob/update_0.1.0/pypi_packages/adabelief_tf0.1.0/adabelief_tf/AdaBelief_tf.py
See toy examples on text classification task and a word embedding task.

[bhack]

This Is the first AdaBelief citation with code:
https://github.com/yuanwei2019/EAdam-optimizer

[seanpmorgan]

TensorFlow Addons is transitioning to a minimal maintenance and release mode. New features will not be added to this repository. For more information, please see our public messaging on this decision:
TensorFlow Addons Wind Down

Please consider sending feature requests / contributions to other repositories in the TF community with a similar charters to TFA:
Keras
Keras-CV
Keras-NLP