Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Shared t5 code #286

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open

WIP: Shared t5 code #286

wants to merge 25 commits into from

Conversation

thomasw21
Copy link
Member

@thomasw21 thomasw21 commented Jun 21, 2022 •
edited
Loading

requires: microsoft/DeepSpeed#2035

TODO:

  • Make sure we can run shared enc/dec with MLM
  • Add test making sure that it runs. with MLM
  • Make sure we can load a BLOOM 6b model in a single A100 GPU
  • Make sure we can load a BLOOM model checkpoint (+ checkpoing manipulation script to share)

@Muennighoff Muennighoff mentioned this pull request Jun 24, 2022
Merged
@thomasw21 thomasw21 force-pushed the thomas/add_shared_t5 branch from 9b4a741 to 48ed32f Compare June 27, 2022 17:15
@thomasw21 thomasw21 force-pushed the thomas/add_shared_t5 branch from 5fd7456 to 9da7588 Compare June 27, 2022 18:49
adammoody pushed a commit to adammoody/Megatron-DeepSpeed that referenced this pull request Dec 18, 2023
@CokeDong
Add tgs metrics (bigscience-workshop#286)
155ce98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@thomasw21