[ViT] Vision Transformer (ViT) backbone, layers, and image classifier #1989
Conversation
| dtype=None, | ||
| **kwargs, | ||
| ): | ||
| data_format = standardize_data_format(data_format) |
There was a problem hiding this comment.
add comment === Layers ===
| dropout_rate=0.0, | ||
| attention_dropout=0.0, | ||
| layer_norm_epsilon=1e-6, | ||
| use_mha_bias=True, |
There was a problem hiding this comment.
these args are missing in get_config
|
Weights Transfer for 4 variants colab gist: https://colab.research.google.com/gist/sineeli/10a7884bef6114eade3b237b63d7f2bd/-keras-hub-vit-weights-transfer.ipynb |
| "image resolution of 384x384 " | ||
| ), | ||
| "params": 86090496, | ||
| "official_name": "ViT", |
There was a problem hiding this comment.
we no longer need the official name or model card, we've reduced what we show on Keras.io to make this simpler. Our Kaggle page will act as the new model card.
| ) | ||
|
|
||
|
|
||
| def convert_weights(keras_hub_model, hf_model): |
There was a problem hiding this comment.
Could we write this as an in library converter? Seems very doable and then we expose this to anyone wanting to convert a vit checkpoint.
There was a problem hiding this comment.
| @@ -137,7 +139,10 @@ def __init__( | |||
| # === Functional Model === | |||
| inputs = self.backbone.input | |||
| x = self.backbone(inputs) | |||
| x = self.pooler(x) | |||
| if pooling == "token": # used for Vision Transformer(ViT) | |||
There was a problem hiding this comment.
"token" feels like a bit a weird name here, especially when compared to "avg" or "max". Maybe "first"?
This PR introduces a Vision Transformer (ViT) implementation