Add MCore FSDP2 support #11216
Open
Add MCore FSDP2 support #11216
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: jasonwan <[email protected]>
Signed-off-by: jasonwan <[email protected]>
Signed-off-by: jasonwan <[email protected]>
Signed-off-by: blahBlahhhJ <[email protected]>
Signed-off-by: BoxiangW <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: BoxiangW <[email protected]>
Signed-off-by: BoxiangW <[email protected]>
Signed-off-by: Boxiang Wang <[email protected]>
Signed-off-by: BoxiangW <[email protected]>
ericharper
reviewed
Nov 14, 2024
| @@ -193,6 +195,7 @@ def __init__( | |||
| ckpt_load_optimizer: bool = True, | |||
| ckpt_save_optimizer: bool = True, | |||
| ddp: Union[DDPLiteral, DistributedDataParallelConfig] = "megatron", | |||
| fsdp: bool = False, | |||
There was a problem hiding this comment.
should we rename it torch_fsdp because we know we'll have mcore_fsdp eventually?
There was a problem hiding this comment.
Do you think we can make this arg a string? ['mcore', 'torch'] or None.
Or expand this arg to fsdp defualts to false, and use_torch_fsdp defaults to false?
So:
fsdp=Ture, use_torch_fsdp=True -> torch_fsdp
fsdp=Ture -> mcore_fsdp (to be supported)
nothing -> ddp
Just trying to make the interface less confusing?
Signed-off-by: Huiying Li <[email protected]>
…11631) * Fix Optimizer & LR scheduler Resume * fix unit test Signed-off-by: Chen Cui <[email protected]> * Apply isort and black reformatting Signed-off-by: cuichenx <[email protected]> * typo Signed-off-by: Chen Cui <[email protected]> * Fix consume samples * Fix unit tests * Apply isort and black reformatting Signed-off-by: suiyoubi <[email protected]> --------- Signed-off-by: Chen Cui <[email protected]> Signed-off-by: cuichenx <[email protected]> Signed-off-by: suiyoubi <[email protected]> Co-authored-by: Chen Cui <[email protected]> Co-authored-by: cuichenx <[email protected]> Co-authored-by: suiyoubi <[email protected]>
* Add vlm inference * Add init * Apply isort and black reformatting Signed-off-by: meatybobby <[email protected]> * Add KV cache and xattn cache in inference * Fix position id for KV cache * Apply isort and black reformatting Signed-off-by: meatybobby <[email protected]> * Add doc string * pylint fix * Remove max_output_len in inference controller * Modify generate script * Apply isort and black reformatting Signed-off-by: meatybobby <[email protected]> * Rename wrapped model * Rename var --------- Signed-off-by: meatybobby <[email protected]> Co-authored-by: meatybobby <[email protected]>
Signed-off-by: oliver könig <[email protected]>
* Add slimpajama example Signed-off-by: Hemil Desai <[email protected]> * Apply isort and black reformatting Signed-off-by: hemildesai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> * Fixes Signed-off-by: Hemil Desai <[email protected]> * Fixes Signed-off-by: Hemil Desai <[email protected]> * Fixes Signed-off-by: Hemil Desai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> * Add notebook Signed-off-by: Hemil Desai <[email protected]> * Fix Signed-off-by: Hemil Desai <[email protected]> * Add basic pretraining notebook for slimpajama Signed-off-by: Hemil Desai <[email protected]> * Add docs for pretraining notebook Signed-off-by: Hemil Desai <[email protected]> * PR feedback Signed-off-by: Hemil Desai <[email protected]> * PR feedback Signed-off-by: Hemil Desai <[email protected]> * Pylint fixes Signed-off-by: Hemil Desai <[email protected]> * Apply isort and black reformatting Signed-off-by: hemildesai <[email protected]> * PR feedback Signed-off-by: Hemil Desai <[email protected]> * Update README Signed-off-by: Hemil Desai <[email protected]> * PR feedback Signed-off-by: Hemil Desai <[email protected]> --------- Signed-off-by: Hemil Desai <[email protected]> Signed-off-by: hemildesai <[email protected]> Co-authored-by: hemildesai <[email protected]>
* remove nemo1 docs Signed-off-by: Chen Cui <[email protected]> * fix link Signed-off-by: Chen Cui <[email protected]> --------- Signed-off-by: Chen Cui <[email protected]>
* Change peft merge model to bf16 Signed-off-by: HuiyingLi <[email protected]> * Apply isort and black reformatting Signed-off-by: HuiyingLi <[email protected]> --------- Signed-off-by: HuiyingLi <[email protected]> Signed-off-by: HuiyingLi <[email protected]> Co-authored-by: HuiyingLi <[email protected]>
…11609) * Add depth pruning (layer dropping) to megatron_gpt_prune.py Signed-off-by: Keval Morabia <[email protected]> * Apply isort and black reformatting Signed-off-by: kevalmorabia97 <[email protected]> --------- Signed-off-by: Keval Morabia <[email protected]> Signed-off-by: kevalmorabia97 <[email protected]> Co-authored-by: kevalmorabia97 <[email protected]>
* add torch dist support Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * add changes Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * revert changes Signed-off-by: Dmytro Pykhtar <[email protected]> * revert changes Signed-off-by: Dmytro Pykhtar <[email protected]> * add deprecation notes Signed-off-by: Dmytro Pykhtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * add readme Signed-off-by: Dmytro Pykhtar <[email protected]> * update readme Signed-off-by: Dmytro Pykhtar <[email protected]> * update readme Signed-off-by: Dmytro Pykhtar <[email protected]> * update readme Signed-off-by: Dmytro Pykhtar <[email protected]> * rename script Signed-off-by: Dmytro Pykhtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * update readme Signed-off-by: Dmytro Pykhtar <[email protected]> * fix style Signed-off-by: dimapihtar <[email protected]> * fix styling Signed-off-by: dimapihtar <[email protected]> * fix styling Signed-off-by: dimapihtar <[email protected]> * Apply isort and black reformatting Signed-off-by: dimapihtar <[email protected]> * remove unused import Signed-off-by: dimapihtar <[email protected]> --------- Signed-off-by: dimapihtar <[email protected]> Signed-off-by: dimapihtar <[email protected]> Signed-off-by: Dmytro Pykhtar <[email protected]> Co-authored-by: dimapihtar <[email protected]> Co-authored-by: Dmytro Pykhtar <[email protected]>
…_ASR.ipynb and ASR_CTC_Language_Finetuning.ipynb (#11675) * Downgrading the 'datasets' package from 3.0.0 to 2.21.0 for Multilang_ASR.ipynb Signed-off-by: Weiqing Wang <[email protected]> * Downgrading the 'datasets' package from 3.0.0 to 2.21.0 for ASR_CTC_Language_Finetuning.ipynb Signed-off-by: Weiqing Wang <[email protected]> --------- Signed-off-by: Weiqing Wang <[email protected]>
…point context io.json (#11648) * Utils to detect and drop deprecated arguments in io.json Signed-off-by: Jan Lasek <[email protected]> * Unit tests for drop_unexpected_params Signed-off-by: Jan Lasek <[email protected]> * Apply isort and black reformatting Signed-off-by: janekl <[email protected]> * Add copyright header Signed-off-by: Jan Lasek <[email protected]> --------- Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Co-authored-by: janekl <[email protected]>
* Remove trt_compile from __init__ as it triggers imports from nemo.utils Signed-off-by: Jan Lasek <[email protected]> * Get tokenizer for NeMo 2 from model.yaml using local SP or HF classes Signed-off-by: Jan Lasek <[email protected]> * Apply isort and black reformatting Signed-off-by: janekl <[email protected]> --------- Signed-off-by: Jan Lasek <[email protected]> Signed-off-by: janekl <[email protected]> Co-authored-by: janekl <[email protected]>
Signed-off-by: Abhishree <[email protected]>
* Fix baichuan export Signed-off-by: Chen Cui <[email protected]> * update import Signed-off-by: Chen Cui <[email protected]> --------- Signed-off-by: Chen Cui <[email protected]>
* rename multimodal data module * Apply isort and black reformatting Signed-off-by: yashaswikarnati <[email protected]> * fix long lengths * fix lint issues * fix long lint issues --------- Signed-off-by: yashaswikarnati <[email protected]> Co-authored-by: yashaswikarnati <[email protected]>
Signed-off-by: oliver könig <[email protected]>
…Sortformer model (#11671) * Fixing the device assignment issues during inference (test_batch) Signed-off-by: taejinp <[email protected]> * Removing the commented code lines Signed-off-by: taejinp <[email protected]> --------- Signed-off-by: taejinp <[email protected]>
* add timestamp support Signed-off-by: kevinhu <[email protected]> * Add unit test, fix a branch, and refactor. Signed-off-by: kevinhu <[email protected]> * Apply isort and black reformatting Signed-off-by: kevinhu-nv <[email protected]> Signed-off-by: kevinhu <[email protected]> * Apply isort and black reformatting Signed-off-by: kevinhu-nv <[email protected]> --------- Signed-off-by: kevinhu <[email protected]> Signed-off-by: kevinhu-nv <[email protected]> Co-authored-by: kevinhu-nv <[email protected]>
* Make LinearAdapter a nn.Linear child to maintain ckpt structure Signed-off-by: Alexandros Koumparoulis <[email protected]> * add _is_fsdp_v1 attribute Signed-off-by: Alexandros Koumparoulis <[email protected]> * lora+fsdp Signed-off-by: Alexandros Koumparoulis <[email protected]> * set precision=bf16 in nl.Trainer Signed-off-by: Alexandros Koumparoulis <[email protected]> * add test Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * fix Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * add unit tests Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> * Update docs Signed-off-by: Alexandros Koumparoulis <[email protected]> * Apply isort and black reformatting Signed-off-by: akoumpa <[email protected]> --------- Signed-off-by: Alexandros Koumparoulis <[email protected]> Signed-off-by: akoumpa <[email protected]> Co-authored-by: akoumpa <[email protected]>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base. Your code was analyzed with PyLint. The following annotations have been identified: Mitigation guide:
By applying these rules, we reduce the occurance of this message in future. Thank you for improving NeMo's documentation! |
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Add MCore FSDP2 support
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information