Code

[Kernel] Flash attention 2#275

Code

Open

remi-or wants to merge 14 commits intolinkedin:mainlinkedin/Liger-Kernel:mainfrom remi-or:mainremi-or/Liger-Kernel:mainCopy head branch name to clipboard

+3,048-3

Commits

Commits on Sep 26, 2024

• Added flash attention kernel

  remi0r

  committed
  2f42f74

• Added monkey patching for SDPA

  remi0r

  committed
  8412147

• Added and passed tests in test_mini_models

  remi0r

  committed
  5731707

• Small changes to the fa ops

  remi0r

  committed
  638882f

• Added and passed tests in no_logits (skipped in multi)

  remi0r

  committed
  e04dc87

• Unit tests, nearly all passed

  remi0r

  committed
  23d0df9

• Benchmarking

  remi0r

  committed
  6c03dcf

• Reduced test numbers

  remi0r

  committed
  b32cf9f

• Formatted the style

  remi0r

  committed
  b213b0e

• Further style

  remi0r

  committed
  c316b87

• Changed the test threshold in accordance with FA repo

  remi0r

  committed
  15cfc63

Commits on Sep 27, 2024

• Merge branch 'main' into main

  lancerts

  authored
  eb2683b

• Changed the test threshold in accordance with FA repo

  remi0r

  committed
  6a31c03

Commits on Sep 30, 2024

• Merge branch 'main' into main

  lancerts

  authored
  cc71c5d