Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Zamba2 #34517

Merged
merged 90 commits into from
Jan 27, 2025
Merged
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
acd25b7
First commit
pglorio Oct 24, 2024
70639b8
Finish model implementation
pglorio Oct 28, 2024
d111b98
First commit
pglorio Oct 24, 2024
8f36dba
Finish model implementation
pglorio Oct 28, 2024
f0c547c
Merge branch 'zamba2' of https://github.com/Zyphra/transformers_zamba…
pglorio Oct 29, 2024
700fbf0
Register zamba2
pglorio Oct 30, 2024
70a6021
generated modeling and configuration
pglorio Nov 4, 2024
88c4b26
Merge pull request #2 from Zyphra/main
pglorio Nov 5, 2024
685906a
generated modeling and configuration
pglorio Nov 5, 2024
4da8d5f
added hybrid cache
pglorio Nov 5, 2024
6b5a9be
fix attention_mask in mamba
pglorio Nov 5, 2024
248350d
dropped unused loras
pglorio Nov 5, 2024
d1d2c66
fix flash2
pglorio Nov 5, 2024
eb6063e
Merge pull request #3 from Zyphra/main
pglorio Nov 5, 2024
5f5d01e
config docstrings
Nov 6, 2024
c1b7647
fix config and fwd pass
pglorio Nov 7, 2024
979b99b
make fixup fixes
pglorio Nov 7, 2024
9d9b2eb
text_modeling_zamba2
pglorio Nov 9, 2024
3a457f5
Merge pull request #4 from Zyphra/main
pglorio Nov 9, 2024
549d4cb
small fixes
pglorio Nov 9, 2024
987bba9
make fixup fixes
pglorio Nov 11, 2024
ffc2a58
Merge pull request #5 from Zyphra/main
pglorio Nov 11, 2024
9adf85e
Fix modular model converter
pglorio Nov 11, 2024
904da4e
added inheritances in modular, renamed zamba cache
pglorio Nov 19, 2024
4725983
Merge pull request #6 from Zyphra/main
pglorio Nov 19, 2024
0be27d7
modular rebase
pglorio Nov 19, 2024
cc0c549
Rebase
pglorio Nov 19, 2024
ac77a09
new modular conversion
pglorio Nov 20, 2024
e59980e
fix generated modeling file
pglorio Nov 20, 2024
73a647a
fixed import for Zamba2RMSNormGated
pglorio Nov 20, 2024
c2b72a5
modular file cleanup
pglorio Nov 21, 2024
0eb39a5
rebase
pglorio Nov 21, 2024
10a0b1e
make fixup and model tests
pglorio Nov 21, 2024
0270667
dropped inheritance for Zamba2PreTrainedModel
pglorio Nov 23, 2024
189c8c5
make fixup and unit tests
pglorio Nov 23, 2024
fa5f79e
Add inheritance of rope from GemmaRotaryEmbedding
pglorio Dec 5, 2024
8079ae0
moved rope to model init
pglorio Dec 5, 2024
d6206eb
drop del self.self_attn and del self.feed_forward
pglorio Dec 5, 2024
f832699
Rebase onto upstream
pglorio Dec 5, 2024
cf613b7
fix tests
pglorio Dec 5, 2024
337faed
renamed lora -> adapter
pglorio Dec 7, 2024
f1b31a1
rewrote adapter implementation
pglorio Dec 7, 2024
8925c15
rebase
pglorio Dec 7, 2024
11fdd47
fixed tests
pglorio Dec 7, 2024
02dd042
Merge branch 'main' into zamba2
pglorio Dec 18, 2024
5d0a5d4
Fix torch_forward in mamba2 layer
pglorio Dec 19, 2024
ef055c9
Fix torch_forward in mamba2 layer
pglorio Dec 19, 2024
b993a78
Fix torch_forward in mamba2 layer
pglorio Dec 19, 2024
bf93251
Dropped adapter in-place sum
pglorio Dec 19, 2024
99708af
removed rope from attention init
pglorio Dec 19, 2024
d9b4a50
updated rope
pglorio Dec 19, 2024
095d853
created get_layers method
pglorio Dec 19, 2024
10ebad5
rebase
pglorio Dec 20, 2024
99e343e
make fixup fix
pglorio Dec 20, 2024
4e40975
make fixup fixes
pglorio Dec 20, 2024
61bb32f
make fixup fixes
pglorio Dec 20, 2024
bb9b24b
fix merge conflicts
pglorio Jan 7, 2025
cb90bb4
update to new attention standard
pglorio Jan 13, 2025
8ed701e
fixes for merge
pglorio Jan 13, 2025
1dbc8c7
update to new attention standard
pglorio Jan 13, 2025
f24e452
make fixup fixes
pglorio Jan 13, 2025
676f862
rebase
pglorio Jan 16, 2025
2b29338
minor fixes
pglorio Jan 16, 2025
b212cb2
cache_position
pglorio Jan 16, 2025
1e3b51e
removed cache_position postion_ids use_cache
pglorio Jan 16, 2025
5ace701
remove config from modular
pglorio Jan 16, 2025
535b631
removed config from modular (2)
pglorio Jan 16, 2025
5a16aa9
rebase
pglorio Jan 16, 2025
1c92266
import apply_rotary_pos_emb from llama
pglorio Jan 16, 2025
99bde93
fixed rope_kwargs
pglorio Jan 16, 2025
baf2ed3
Instantiate cache in Zamba2Model
pglorio Jan 16, 2025
9afb57e
fix cache
pglorio Jan 17, 2025
d1687f9
fix @slow decorator
pglorio Jan 17, 2025
4299889
rebase
pglorio Jan 20, 2025
a0545bf
rebase
pglorio Jan 21, 2025
903f6dc
small fix in modular file
pglorio Jan 21, 2025
14396d7
Update docs/source/en/model_doc/zamba2.md
pglorio Jan 23, 2025
02f5807
several minor fixes
pglorio Jan 23, 2025
bfb0267
inherit mamba2decoder fwd and drop position_ids in mamba
pglorio Jan 23, 2025
b222943
removed docstrings from modular
pglorio Jan 23, 2025
b114ad8
rebase
pglorio Jan 23, 2025
929ee67
reinstate zamba2 attention decoder fwd
pglorio Jan 23, 2025
9007a52
use regex for tied keys
pglorio Jan 24, 2025
f701dbd
Revert "use regex for tied keys"
pglorio Jan 24, 2025
87b938b
use regex for tied keys
pglorio Jan 24, 2025
5e09290
add cpu to slow forward tests
pglorio Jan 24, 2025
8ed2353
dropped config.use_shared_mlp_adapter
pglorio Jan 24, 2025
a9bbd9c
Update docs/source/en/model_doc/zamba2.md
pglorio Jan 24, 2025
1e82757
rebase
pglorio Jan 27, 2025
37bff34
re-convert from modular
pglorio Jan 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rebase
pglorio committed Jan 23, 2025
commit b114ad85b013946afc5c69c1f35cdbfb42794a0f

This merge commit was added into this branch cleanly.

There are no new changes to show, but you can still view the diff.