Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Wait for #2803][Mixed Precision] Fix gradient clipping logic @open sesame 12/02 09:37 #2749

Merged
merged 1 commit into from
Dec 2, 2024
Merged

[Wait for #2803][Mixed Precision] Fix gradient clipping logic @open sesame 12/02 09:37 #2749

merged 1 commit into from
Dec 2, 2024

Conversation

DonghakPark
Copy link
Member

[Mixed Precision] Fix gradient clipping logic

update mixed precision - gradient clipping logic

  • when gradient clipping, gradient should unscale before calc l2norm

Resolves:

Self evaluation:

  1. Build test: [X]Passed [ ]Failed [ ]Skipped
  2. Run test: [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Donghak PARK [email protected]

@DonghakPark DonghakPark self-assigned this Oct 8, 2024
@taos-ci
Copy link

taos-ci commented Oct 8, 2024

📝 TAOS-CI Version: 1.5.20200925. Thank you for submitting PR #2749. Please a submit 1commit/1PR (one commit per one PR) policy to get comments quickly from reviewers. Your PR must pass all verificiation processes of cibot before starting a review process from reviewers. If you are new member to join this project, please read manuals in documentation folder and wiki page. In order to monitor a progress status of your PR in more detail, visit http://ci.nnstreamer.ai/.

@DonghakPark DonghakPark changed the title [Mixed Precision] Fix gradient clipping logic [Wait for #2663][Mixed Precision] Fix gradient clipping logic Oct 8, 2024
@taos-ci
Copy link

taos-ci commented Oct 8, 2024

:octocat: cibot: @DonghakPark, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2749-202410081648530.3276469707489-2d4e34793f8bee4d6bcada6961a4d3345e7e8d56/.

@DonghakPark DonghakPark mentioned this pull request Oct 30, 2024
Merged
@DonghakPark DonghakPark force-pushed the fp16_gradient_clipping_fix branch from 2d4e347 to cd67fd0 Compare November 25, 2024 01:18
@DonghakPark DonghakPark force-pushed the fp16_gradient_clipping_fix branch from cd67fd0 to 2a41677 Compare November 25, 2024 01:23
@taos-ci
Copy link

taos-ci commented Nov 25, 2024

:octocat: cibot: @DonghakPark, A builder checker could not be completed because one of the checkers is not completed. In order to find out a reason, please go to http://ci.nnstreamer.ai/nntrainer/ci/repo-workers/pr-checker/2749-202411251023080.00036811828613281-2a4167753060e51f0e04d20b8a66a6611ada5977/.

@DonghakPark DonghakPark changed the title [Wait for #2663][Mixed Precision] Fix gradient clipping logic [Wait for #2803][Mixed Precision] Fix gradient clipping logic Nov 25, 2024
taos-ci
taos-ci approved these changes Nov 25, 2024
View reviewed changes
Copy link

@taos-ci taos-ci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DonghakPark, 💯 All CI checkers are successfully verified. Thanks.

@DonghakPark DonghakPark force-pushed the fp16_gradient_clipping_fix branch 2 times, most recently from 10cae11 to fba7c09 Compare November 26, 2024 00:56
skykongkong8
skykongkong8 reviewed Nov 26, 2024
View reviewed changes
Copy link
Member

@skykongkong8 skykongkong8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this about gradient "clipping"..? or just loss scaling

@DonghakPark
Copy link
Member Author

Is this about gradient "clipping"..? or just loss scaling

This PR about "loss scaling" at gradient clipping logic in case Mixed Precision

taos-ci
taos-ci approved these changes Nov 26, 2024
View reviewed changes
Copy link

@taos-ci taos-ci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DonghakPark, 💯 All CI checkers are successfully verified. Thanks.

taos-ci
taos-ci approved these changes Nov 27, 2024
View reviewed changes
Copy link

@taos-ci taos-ci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DonghakPark, 💯 All CI checkers are successfully verified. Thanks.

jijoongmoon
jijoongmoon approved these changes Nov 28, 2024
View reviewed changes
Copy link
Collaborator

@jijoongmoon jijoongmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

djeong20
djeong20 approved these changes Nov 28, 2024
View reviewed changes
Copy link
Contributor

@djeong20 djeong20 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jijoongmoon jijoongmoon changed the title [Wait for #2803][Mixed Precision] Fix gradient clipping logic [Wait for #2803][Mixed Precision] Fix gradient clipping logic @open sesame 12/02 09:37 Dec 2, 2024
taos-ci
taos-ci approved these changes Dec 2, 2024
View reviewed changes
Copy link

@taos-ci taos-ci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DonghakPark, 💯 All CI checkers are successfully verified. Thanks.

@DonghakPark
[Mixed Precision] Fix gradient clipping logic
12ebb99 
update mixed precision - gradient clipping logic
- when gradient clipping, gradient should unscale before calc l2norm

**Self evaluation:**
1. Build test:	 [X]Passed [ ]Failed [ ]Skipped
2. Run test:	 [X]Passed [ ]Failed [ ]Skipped

Signed-off-by: Donghak PARK <[email protected]>
@DonghakPark DonghakPark force-pushed the fp16_gradient_clipping_fix branch from 30d3abc to 12ebb99 Compare December 2, 2024 01:47
taos-ci
taos-ci approved these changes Dec 2, 2024
View reviewed changes
Copy link

@taos-ci taos-ci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DonghakPark, 💯 All CI checkers are successfully verified. Thanks.

@jijoongmoon jijoongmoon merged commit 0ee82e0 into nnstreamer:main Dec 2, 2024
36 checks passed
@DonghakPark DonghakPark deleted the fp16_gradient_clipping_fix branch December 2, 2024 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skykongkong8 skykongkong8 skykongkong8 approved these changes

@djeong20 djeong20 djeong20 approved these changes

@taos-ci taos-ci taos-ci approved these changes

@jijoongmoon jijoongmoon jijoongmoon approved these changes

@myungjoo myungjoo Awaiting requested review from myungjoo myungjoo is a code owner

@again4you again4you Awaiting requested review from again4you again4you is a code owner

@jaeyun-jung jaeyun-jung Awaiting requested review from jaeyun-jung jaeyun-jung is a code owner

@leemgs leemgs Awaiting requested review from leemgs leemgs is a code owner

@wooksong wooksong Awaiting requested review from wooksong wooksong is a code owner

@helloahn helloahn Awaiting requested review from helloahn

@kparichay kparichay Awaiting requested review from kparichay

@gichan-jang gichan-jang Awaiting requested review from gichan-jang gichan-jang is a code owner

@anyj0527 anyj0527 Awaiting requested review from anyj0527 anyj0527 is a code owner

@zhoonit zhoonit Awaiting requested review from zhoonit

@lhs8928 lhs8928 Awaiting requested review from lhs8928 lhs8928 is a code owner

@songgot songgot Awaiting requested review from songgot songgot is a code owner

@jihochu jihochu Awaiting requested review from jihochu jihochu is a code owner

@SeoHyungjun SeoHyungjun Awaiting requested review from SeoHyungjun SeoHyungjun is a code owner

@baek2sm baek2sm Awaiting requested review from baek2sm baek2sm is a code owner

@EunjuYang EunjuYang Awaiting requested review from EunjuYang EunjuYang is a code owner

Assignees

@DonghakPark DonghakPark

Labels
PR/READY2MERGE
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@DonghakPark @taos-ci @jijoongmoon @djeong20 @skykongkong8