Code

Attention with scaling#115

Code

Merged

bdvllrs merged 103 commits intomainruflab/shimmer:mainfrom attention-with-scalingruflab/shimmer:attention-with-scalingCopy head branch name to clipboardJul 30, 2024

+217-91

Commits

Commits on Jul 30, 2024

• add scaling as input to attentionbase

  

  larascipio

  authored and

  bdvllrs

  committed
  2a11cd4

• delete print

  

  larascipio

  authored and

  bdvllrs

  committed
  40d8dc4

• added fixed corruption vector

  

  larascipio

  authored and

  bdvllrs

  committed
  f064af0

• add corruption per batch

  

  larascipio

  authored and

  bdvllrs

  committed
  b4aa045

• add prints for debugging

  

  larascipio

  authored and

  bdvllrs

  committed
  6a4f9fd

• add more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  178aa42

• hard code domain names to check bug

  

  larascipio

  authored and

  bdvllrs

  committed
  bbf856a

• change frozenset to list

  

  larascipio

  authored and

  bdvllrs

  committed
  686b997

• added more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  f0a8e00

• add print

  

  larascipio

  authored and

  bdvllrs

  committed
  d27d660

• debug

  

  larascipio

  authored and

  bdvllrs

  committed
  12e0735

• remove prints after debugging

  

  larascipio

  authored and

  bdvllrs

  committed
  c90ee14

• add prints for debugging

  

  larascipio

  authored and

  bdvllrs

  committed
  b73118c

• add shape print

  

  larascipio

  authored and

  bdvllrs

  committed
  9686201

• change corruption vector

  

  larascipio

  authored and

  bdvllrs

  committed
  86b2cd8

• print

  

  larascipio

  authored and

  bdvllrs

  committed
  3353164

• change dim

  

  larascipio

  authored and

  bdvllrs

  committed
  cc0d831

• change to cuda

  

  larascipio

  authored and

  bdvllrs

  committed
  c33d39e

• remove print

  

  larascipio

  authored and

  bdvllrs

  committed
  f87b5d7

• prints

  

  larascipio

  authored and

  bdvllrs

  committed
  be4c075

• check if loop works

  

  larascipio

  authored and

  bdvllrs

  committed
  343c081

• checking

  

  larascipio

  authored and

  bdvllrs

  committed
  fb7365e

• add other prints

  

  larascipio

  authored and

  bdvllrs

  committed
  668e166

• remove unnessary part of the loop

  

  larascipio

  authored and

  bdvllrs

  committed
  6bf8ecc

• add prints for debugging

  

  larascipio

  authored and

  bdvllrs

  committed
  718a54b

• more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  218d33e

• change to 5

  

  larascipio

  authored and

  bdvllrs

  committed
  32b3b4a

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  477636c

• change corruption from 5 to 25

  

  larascipio

  authored and

  bdvllrs

  committed
  3321bc4

• changed corruption to 5 again

  

  larascipio

  authored and

  bdvllrs

  committed
  982a91e

• add prints for debugging

  

  larascipio

  authored and

  bdvllrs

  committed
  4440055

• more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  2fe0074

• add more

  

  larascipio

  authored and

  bdvllrs

  committed
  6117c24

• add more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  6d89051

• made another corruption function

  

  larascipio

  authored and

  bdvllrs

  committed
  5560d2b

• add prints

  

  larascipio

  authored and

  bdvllrs

  committed
  b6f717b

• change

  

  larascipio

  authored and

  bdvllrs

  committed
  6a09873

• change

  

  larascipio

  authored and

  bdvllrs

  committed
  23304bb

• remove prints in new corruption function

  

  larascipio

  authored and

  bdvllrs

  committed
  7f1528a

• add prints for debug

  

  larascipio

  authored and

  bdvllrs

  committed
  504b74c

• change tensor size

  

  larascipio

  authored and

  bdvllrs

  committed
  d6b7260

• add more noise

  

  larascipio

  authored and

  bdvllrs

  committed
  0232ae2

• add prints

  

  larascipio

  authored and

  bdvllrs

  committed
  929c20a

• changed corruption

  

  larascipio

  authored and

  bdvllrs

  committed
  8e0fbd8

• change corrruption

  

  larascipio

  authored and

  bdvllrs

  committed
  84141c1

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  c1dbe9d

• change corruption

  

  larascipio

  authored and

  bdvllrs

  committed
  2d59b51

• change to old corruption

  

  larascipio

  authored and

  bdvllrs

  committed
  4944fce

• made a new corruption functor on tensors

  

  larascipio

  authored and

  bdvllrs

  committed
  638d5f1

• debug

  

  larascipio

  authored and

  bdvllrs

  committed
  c7a3fc4

• add prints

  

  larascipio

  authored and

  bdvllrs

  committed
  2c54d3e

• more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  903e3c8

• change

  

  larascipio

  authored and

  bdvllrs

  committed
  12493fa

• nog meer

  

  larascipio

  authored and

  bdvllrs

  committed
  ccf8282

• prints

  

  larascipio

  authored and

  bdvllrs

  committed
  18ac69c

• more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  b57fbaa

• prints

  

  larascipio

  authored and

  bdvllrs

  committed
  37431d0

• add more prints

  

  larascipio

  authored and

  bdvllrs

  committed
  b6ed759

• changed function

  

  larascipio

  authored and

  bdvllrs

  committed
  61b52fa

• debug

  

  larascipio

  authored and

  bdvllrs

  committed
  2e220b2

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  054d218

• check how the attention scores change

  

  larascipio

  authored and

  bdvllrs

  committed
  2905ad6

• print attention dict

  

  larascipio

  authored and

  bdvllrs

  committed
  dc25da8

• prints

  

  larascipio

  authored and

  bdvllrs

  committed
  f1d85fb

• change corruption to corrupt side as well

  

  larascipio

  authored and

  bdvllrs

  committed
  28a40d5

• add print to check

  

  larascipio

  authored and

  bdvllrs

  committed
  8899f98

• change sides

  

  larascipio

  authored and

  bdvllrs

  committed
  a41946e

• other change

  

  larascipio

  authored and

  bdvllrs

  committed
  58dc5ad

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  369e555

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  2453317

• add attention scores to log

  

  larascipio

  authored and

  bdvllrs

  committed
  56f15a1

• add calculate mean attention

  

  larascipio

  authored and

  bdvllrs

  committed
  dcb72db

• print attention scores

  

  larascipio

  authored and

  bdvllrs

  committed
  54ee31e

• mean attention

  

  larascipio

  authored and

  bdvllrs

  committed
  6fd2d88

• remove print attentions scores

  

  larascipio

  authored and

  bdvllrs

  committed
  ec643fe

• omkeren

  

  larascipio

  authored and

  bdvllrs

  committed
  0141dd2

• add variable corruption

  

  larascipio

  authored and

  bdvllrs

  committed
  c09e245

• made a new function for testing all corruption combi

  

  larascipio

  authored and

  bdvllrs

  committed
  ca71135

• found a bug in corruption scaling (default should be 0.0)

  

  larascipio

  authored and

  bdvllrs

  committed
  2cac9e2

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  e575f3c

• add prints

  

  larascipio

  authored and

  bdvllrs

  committed
  59cbec9

• change corruption

  

  larascipio

  authored and

  bdvllrs

  committed
  d961cd4

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  01abbce

• removed batch corruption function

  

  larascipio

  authored and

  bdvllrs

  committed
  72d506d

• made a new function to train corruption on both sides

  

  larascipio

  authored and

  bdvllrs

  committed
  861879b

• remove prints for training

  

  larascipio

  authored and

  bdvllrs

  committed
  135dfdf

• bug

  

  larascipio

  authored and

  bdvllrs

  committed
  1cefd56

• added second step attention in query attention

  

  larascipio

  authored and

  bdvllrs

  committed
  e3c97a1

• add corruption vector to device

  

  larascipio

  authored and

  bdvllrs

  committed
  0081e92

• removed

  

  larascipio

  authored and

  bdvllrs

  committed
  af9995f

• added step 2

  

  larascipio

  authored and

  bdvllrs

  committed
  6a7a2df

• added even anoter step

  

  larascipio

  authored and

  bdvllrs

  committed
  2399b2f

• deleted added steps

  

  larascipio

  authored and

  bdvllrs

  committed
  b689b4a

• added multiple steps as a parameter for query attention

  

  larascipio

  authored and

  bdvllrs

  committed
  2326a4e

• found bug in variable corruption

  

  larascipio

  authored and

  bdvllrs

  committed
  61cc1d5

• removed static attention and changed into step 0

  

  larascipio

  authored and

  bdvllrs

  committed
  658526f

• changed corruption functions to one and two sided corr

  

  larascipio

  authored and

  bdvllrs

  committed
  9dbc65a

• spelling mistake

  

  larascipio

  authored and

  bdvllrs

  committed
  ec484cf

• again mistake

  

  larascipio

  authored and

  bdvllrs

  committed
  0962a7c

• remove prints

  

  larascipio

  authored and

  bdvllrs

  committed
  c7bb498

• Added some comments

  

  larascipio

  authored and

  bdvllrs

  committed
  61d0c7e

• line was too long

  

  larascipio

  authored and

  bdvllrs

  committed
  a5b26a8

• Fix mypy issues

  bdvllrs

  committed
  4929c3f