-
Notifications
You must be signed in to change notification settings - Fork 140
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
It seems training will collapse when applying another Dataset #19
Comments
We also have experienced similar issues for some datasets. To the best of my knowledge this is caused by either
We've noticed that this happens when training on style datasets of especially complicated artists (those where local structure and texture is less prominent, but the painting composition and content is what matters the most) |
Thanks very much for your detailed reply. |
I am running into this same issue of the output being entirely black stylized images when I train using the art style dataset of 144 black and white paintings of different sizes. Used --image_size=256 due to limitations of my hw. Ran the training to 30000 iterations. Would really appreciate some help esp. on how to restart training from the point of corruption. Or other things that would be good to try. Thanks much. |
i've managed to solve collapsed black output problem by:
i also removed winrate-based training schedule for now (left just one G and one D pass, no accuracy calculation), but will check again later if it was an issue |
great! possible to share the modified files for me to quickly rerun the training on my art dataset to see if it might work please? |
@narayansundararajan123 well, i've quite refactored the whole code in a way that i'm more used to, so it's rather different from the original repo now - including names, vars, module structures, utility functions, etc. applied changes are kind of standard GAN tricks to 'slow down' or 'distract' discriminator when it's trained much faster than generator (which is the reason of collapsing - that's quite well seen on the D losses behaviour in tensorboard). and btw i also totally removed all accuracy calculation and winrate-based training schedule part, cause the model never converged with it (and perfectly did without). |
@narayansundararajan123 ok, let's try these quick updates for original code:
replace this
by this
if you use last 'fix' you can also comment out everything related to accuracy measurement/reporting. |
Thanks so much! Will try and let you know. |
Hi Tried training again using the art style dataset of 144 black and white paintings of different sizes. Used --image_size=256. Ran the training to 30000 iterations again with the new modifications. Still unfortunately running into the same issue of the output being entirely black stylized images. Would there be anything that I might have missed implementing other than the modifications from above or other suggestions on solving this issue? |
@narayansundararajan123 other changes were quite subtle (like tweaking loss weights for D and G separately), so i don't think they really matter. i also changed some technical ops (like loading data) for the ones i'm used to, but this was done for easier reading/maintaining, i doubt it could affect the result. |
Thanks. I also noticed beyond 210000 iterations when the model likely goes off, I am also getting RuntimeWarning: invalid value encountered in reduce when i run the inference and get black output images after stylization. I can also share the dataset if you could send me an email at [email protected]. |
haven't seen such warnings.. |
@eps696 Can you tell me what discriminator,transformer loss and feature loss weight you used because I cant get the GAN to converge. |
@andrew194 |
@eps696 Thanks! Did you also use 1 for the discriminator loss weight? |
didn't quite catch what you mean by 'use 1' |
Sorry I was referring to the optimizer |
multiple weights are applied to the losses before, no need for another multiplier.
|
I have trained the model using another content dataset with the given styles images of Monet, but it seems it will soon collapse and output entirely black stylized images.
I've tried discarding the image augmentation process and scipy.misc.imresize(), but it still can not work with this content dataset (150,000 jpg images with generally 1800+ pixels).
Would you please give me some tips or suggestions about this issue? Such as trying another learning rate/discriminator success rate.
Thanks for your time in advance.
The text was updated successfully, but these errors were encountered: