Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attention with scaling #115

Merged
merged 103 commits into from
Jul 30, 2024
Merged

Attention with scaling #115

merged 103 commits into from
Jul 30, 2024

Conversation

larascipio
Copy link
Contributor

Changes:

  • In selection.py I created one single function that handles all steps of the process (instead of two separate functions)
  • In attention module I changed the corruption functions to one-sided or two-sided corruption, and I also added some variables that I used for testing (e.g. 'test_sides_corruption'). These might be unnecessary for you.

bdvllrs
bdvllrs requested changes Jul 30, 2024
View reviewed changes
Copy link
Collaborator

@bdvllrs bdvllrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mypy is not passing... Will investigate before merging

larascipio added 25 commits July 30, 2024 12:47
@larascipio @bdvllrs
made a new function for testing all corruption combi
ca71135
@larascipio @bdvllrs
found a bug in corruption scaling (default should be 0.0)
2cac9e2
@larascipio @bdvllrs
remove prints
e575f3c
@larascipio @bdvllrs
add prints
59cbec9
@larascipio @bdvllrs
change corruption
d961cd4
@larascipio @bdvllrs
remove prints
01abbce
@larascipio @bdvllrs
removed batch corruption function
72d506d
@larascipio @bdvllrs
made a new function to train corruption on both sides
861879b
@larascipio @bdvllrs
remove prints for training
135dfdf
@larascipio @bdvllrs
bug
1cefd56
@larascipio @bdvllrs
added second step attention in query attention
e3c97a1
@larascipio @bdvllrs
add corruption vector to device
0081e92
@larascipio @bdvllrs
removed
af9995f
@larascipio @bdvllrs
added step 2
6a7a2df
@larascipio @bdvllrs
added even anoter step
2399b2f
@larascipio @bdvllrs
deleted added steps
b689b4a
@larascipio @bdvllrs
added multiple steps as a parameter for query attention
2326a4e
@larascipio @bdvllrs
found bug in variable corruption
61cc1d5
@larascipio @bdvllrs
removed static attention and changed into step 0
658526f
@larascipio @bdvllrs
changed corruption functions to one and two sided corr
9dbc65a
@larascipio @bdvllrs
spelling mistake
ec484cf
@larascipio @bdvllrs
again mistake
0962a7c
@larascipio @bdvllrs
remove prints
c7bb498
@larascipio @bdvllrs
Added some comments
61d0c7e
@larascipio @bdvllrs
line was too long
a5b26a8
@bdvllrs bdvllrs force-pushed the attention-with-scaling branch from bc3ba39 to a5b26a8 Compare July 30, 2024 12:48
@bdvllrs bdvllrs merged commit 774ff2f into main Jul 30, 2024
1 check passed
@bdvllrs bdvllrs deleted the attention-with-scaling branch July 30, 2024 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bdvllrs bdvllrs bdvllrs approved these changes

@RolandBERTINJOHANNET RolandBERTINJOHANNET Awaiting requested review from RolandBERTINJOHANNET

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@larascipio @bdvllrs