Add manual optimization to core task #1796
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the same way that is done for the speaker diarization task
…ages/pyannote-audio into fix-optimization-issue
Co-authored-by: Hervé BREDIN <[email protected]>
hbredin
reviewed
Nov 26, 2024
Comment on lines
+253
to
+255
| "clip_val": 5.0, | ||
| "clip_algorithm": "norm", | ||
| "accumulate_batches": 1, |
There was a problem hiding this comment.
Shouldn't we instead try to grab these values from trainer options directly?
Maybe trainer is exposed as an attribute of model?
There was a problem hiding this comment.
Yep, looks like you could use something like
self.model.trainer.{accumulate_grad_batches, gradient_clip_val, gradient_clip_algorithm}
There was a problem hiding this comment.
Passing accumulate_grad_batches, gradient_clip_val or gradient_clip_algorithm to the Trainer will raise a MisconfigurationException if we set automatic_optimization=False.
lightning_fabric.utilities.exceptions.MisconfigurationException: Automatic gradient clipping is not supported for manual optimization. Remove `Trainer(gradient_clip_val=5.0)` or switch to automatic optimization.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds:
manual_optimizationtopyannote.audio.core.Taskgradientattribute to pyannote's tasks, which is a dict of gradient arguments: gradient clipping (val and algo), and gradient batch accumulation. (needed bymanual_optimisation