Add Keras 3 example for "Transformer model for MIDI music generation"

[johacks]

Hi,

This is my first contribution, so apologies in advance for mistakes made!

I saw a call for contribution with an example of midi generation using transformers. I adapted the referenced code to keras 3.

Here are some notes on the implementation:

• All MIDI datasets used by original code are inaccessible, so I switched to Maestro dataset.
• I implemented the relative global attention used in the paper, which results in lots of additional code over using existing multi-head attention. Maybe it would be better to use already implemented attention layer CachedMultiHeadAttention to compact code, but probably getting worse results.
• Tokenization used in the code needs lots of code and is even hosted on a different repo. I have currently forked their repo and published to pypi so it can be easily installed, but I'm not sure this is the most appropiate way to handle it. Maybe it would make more sense to have a MIDI tokenizer on keras_hub?
• Following advice of Keras contributing guide issue, I'm only including Python script before generating .md and .ipynb files.

Thanks!

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[fchollet]

Thanks for the PR. It looks great!

• Did you it works with all backends?
• Which backend is the fastest on the Colab GPU?
• Excluding text, how many lines of code do you end up with (are you able to run the rendering script)? You can use scripts/tutobooks.py:count_locs_in_file to count

[johacks]

@fchollet
Hi, thanks for the feedback, I have made corresponding changes according to comments on the code.

Regarding your other questions:

• Did you check it works with all backends?
  • I have now tried in in colab on all backends. I had to make some adjustments to the code:
    • In JAX it was too slow to concatenate multiple times for each generation, so I changed approach to fill a padded array with always the same size.
    • In torch there were some incompatibilities with negative slicing steps that have been fixed. It also seems that due to numerical stability issues, torch needs a lower learning rate. I adjusted the script to conditionally set a learning rate adjustment factor based on the backend.
• Which backend is the fastest on the Colab GPU?
  • I tried all backends with a T4 GPU. It takes about 11 minutes for data preprocessing. Training + generation time is about 40 minutes, 27 minutes, 25 minutes in Torch, JAX and Tensorflow respectively.
    • Tensorflow notebook
    • JAX notebook
    • Torch notebook
• Excluding text, how many lines of code do you end up with (are you able to run the rendering script)? You can use scripts/tutobooks.py:count_locs_in_file to count
  • I was at 460. I refactored and got to 350.
  • I can generate the rendering script (autogen.py add_example), as long as it is launched with root privileges. I have added generated files to the PR.

[fchollet]

Awesome -- thanks a lot for the contribution!