Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training is very slow #7

Open
kevaday opened this issue Jul 30, 2018 · 4 comments
Open

Training is very slow #7

kevaday opened this issue Jul 30, 2018 · 4 comments

Comments

@kevaday
Copy link

kevaday commented Jul 30, 2018

Hi Skuldur,

I am using your training script to train with my own midi files. I don't know why, but the training is going very slowly. It takes about 25 minutes to train only one epoch. I'm confused because i have a brand new GTX 1070 with drivers installed including geforce experience, cuda 9.0 (because tensorflow is incompatible with version 9.2 for some reason), and cudNN. With another training script (it's for generating text) it took only 1-2 minutes per epoch, but its model only had 1 LSTM so that might also be relatively slow.
How long did it take you to train with your midi files, with what GPU?
If you can help, it would be greatly appreciated.

Thanks,
Kevaday

@kevaday
Copy link
Author

kevaday commented Jul 30, 2018

Also, I tried using your text generating LSTM but it's also very slow on my GPU, it takes about 1 hour to do one epoch. Shouldn't my GPU be a lot faster at training?

@mehrzeller
Copy link

you can use CuDNNLSTM instead :
from keras.layers import CuDNNLSTM
instead of
from keras.layers import LSTM
don't forget to change all your LSTM statements in the source code

@kevaday
Copy link
Author

kevaday commented Sep 13, 2018

Thanks mehrzeller,
actually, it's a coincidence because I was just looking at the keras recurrent layers site and just saw that there's such thing
but thanks anyways

@victor-felicitas
Copy link

victor-felicitas commented Mar 15, 2021

For new Keras versions specifying CuDNNLSTM is obsolete.
Training is slow because with recurrent_dropout >0 CUDNN can not be used.

From Keras documentation:
"The requirements to use the cuDNN implementation are:
activation == tanh
recurrent_activation == sigmoid
recurrent_dropout == 0
unroll is False
use_bias is True
Inputs, if use masking, are strictly right-padded.
Eager execution is enabled in the outermost context."

I suggest changing the model by adding dropout layer instead, after each LSTM layer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants