Preparing transformers model to train it with our demo-data

[mxochicale]

Using transformers might add novelty to our work, so I suggest to looking and replicating the following examples of transformers for time-series that made use of tensorflow library:

• https://github.com/mxochicale/transformers-tutorials/blob/main/tutorials/time-series-classification/timeseries_classification_from_scratch.ipynb
• https://github.com/mxochicale/transformers-tutorials/blob/main/tutorials/time-series-classification/timeseries_transformer_classification.ipynb

Few suggested steps for this one:

1. Create branch
2. Create a new folder in scripts called transformer-model and put above notebooks inside
3. Remove mamba remove -n ai-surg-skills-VE -alland create a new mamba environment mamba env create -f ve.yml > using "tensorflow" package in line 22 the yml file (https://github.com/mxochicale/in2research2023/blob/main/dependencies/ve.yml#L21):

  - cudatoolkit #=11.*.*   #=11.*.*
  - tensorflow
  - pip

4. Run both notebooks and get unstarnging of the workflow in both notebooks as we will use them to train them with our data.

Thanks, --Miguel

[mxochicale]

2023-07-17 14:30:53.185652: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-07-17 14:30:53.185684: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Potential solutions:

• https://stackoverflow.com/questions/70967651/could-not-load-dynamic-library-libcudart-so-11-0
• Could not load dynamic library 'libcudart.so.11.0' tensorflow/tensorflow#45930
• https://discuss.tensorflow.org/t/could-not-load-dynamic-library-libcudart-so-11-0/15711/8

[mxochicale]

Hi @AbuAbdul1ah

See this tutorials to get better understanding for transformers https://github.com/mxochicale/transformers-tutorials/tree/main/references

Thanks, --Miguel

[AbuAbdul1ah]

Hi @mxochicale

Found this article to useful, it speaks about the advantage of transformer.
https://ieeexplore.ieee.org/abstract/document/9252123
https://github.com/linlei1214/SITS-BERT

[mxochicale]

Hi @AbuAbdul1ah
Have a look to this references that will help to put together or coming experiments with hyperarameter tuning:

• https://medium.com/mlearning-ai/hyperparameter-tuning-fixing-overfitting-in-neural-networks-b983b21d60bd
• https://medium.com/mlearning-ai/hyperparameter-tuning-fixing-high-bias-underfitting-in-neural-networks-5184ead3cbed