Code

fp8 backward#119

Code

Merged

micmelesse merged 38 commits intomain_perfROCm/flash-attention:main_perffrom micmelesse/fp8_bwdROCm/flash-attention:micmelesse/fp8_bwdCopy head branch name to clipboardMar 5, 2025

+2,366-1,465

Commits

Commits on Feb 25, 2025

• fp8 BWD after figuring out varlen problem

  Show description for b09e323
  micmelesse

  committed
  b09e323

  Copy full SHA for b09e323

• rm require grad

  micmelesse

  committed
  92cb15f

  Copy full SHA for 92cb15f

• decast fp8 for ref input, use fp16 as input

  Show description for 592e69b
  micmelesse

  committed
  592e69b

  Copy full SHA for 592e69b

Commits on Feb 26, 2025

• disable causal

  micmelesse

  committed
  5fa886a

  Copy full SHA for 5fa886a

• fix bug

  micmelesse

  committed
  1279940

  Copy full SHA for 1279940

• pass strides

  micmelesse

  committed
  fc8de83

  Copy full SHA for fc8de83

• DEBUG modes work only with interp

  micmelesse

  committed
  b9a7e05

  Copy full SHA for b9a7e05

• zero out varlen bwd grads

  micmelesse

  committed
  6bd2ee4

  Copy full SHA for 6bd2ee4

• zero out everything

  micmelesse

  committed
  dd0549c

  Copy full SHA for dd0549c

• varlen dropout and causal works

  micmelesse

  committed
  3e59caa

  Copy full SHA for 3e59caa

Commits on Feb 27, 2025

• add descale factors to other apis

  micmelesse

  committed
  acf6194

  Copy full SHA for acf6194

• save

  micmelesse

  committed
  c790dfd

  Copy full SHA for c790dfd

• unify tests

  micmelesse

  committed
  d21e892

  Copy full SHA for d21e892

• add packing flag

  micmelesse

  committed
  8a12ff1

  Copy full SHA for 8a12ff1

Commits on Feb 28, 2025

• fix copy grad bug

  micmelesse

  committed
  1d7bfc4

  Copy full SHA for 1d7bfc4

• add types, flags for zeroing tensors and

  Show description for 4622888
  micmelesse

  committed
  4622888

  Copy full SHA for 4622888

Commits on Mar 3, 2025

• just pass the output tensors

  micmelesse

  committed
  eb342c7

  Copy full SHA for eb342c7

• accumlate forwad in fp32

  micmelesse

  committed
  2e965de

  Copy full SHA for 2e965de

• fp8 in and fp8 out

  micmelesse

  committed
  dfc3824

  Copy full SHA for dfc3824

• return descale factors works for out

  micmelesse

  committed
  edcd20e

  Copy full SHA for edcd20e

• start fp8 return for bwd

  micmelesse

  committed
  49eba04

  Copy full SHA for 49eba04

• return dq, dv, dk descale factors

  micmelesse

  committed
  9295353

  Copy full SHA for 9295353

• save what you have

  micmelesse

  committed
  045c16c

  Copy full SHA for 045c16c

Commits on Mar 4, 2025

• custom fp8 api function

  micmelesse

  committed
  653deab

  Copy full SHA for 653deab

• add varlen function

  micmelesse

  committed
  afe34d8

  Copy full SHA for afe34d8

• test backward with varlen

  micmelesse

  committed
  4df0361

  Copy full SHA for 4df0361

• test fp8

  micmelesse

  committed
  fde6f14

  Copy full SHA for fde6f14

• kv cache fix

  micmelesse

  committed
  a98870d

  Copy full SHA for a98870d

• clean up interface

  micmelesse

  committed
  04c2774

  Copy full SHA for 04c2774

• add packed api

  micmelesse

  committed
  2bf8ef2

  Copy full SHA for 2bf8ef2

• fix qkv bug

  micmelesse

  committed
  413096e

  Copy full SHA for 413096e

• disable bench

  micmelesse

  committed
  337f0e9

  Copy full SHA for 337f0e9

• run big tests at the end

  micmelesse

  committed
  fe28e6d

  Copy full SHA for fe28e6d

• run in parrallel

  micmelesse

  committed
  b61c019

  Copy full SHA for b61c019

• Update utils.py

  micmelesse

  authored
  a9f0d45

  Copy full SHA for a9f0d45

Commits on Mar 5, 2025

• Update amd_tests.yml

  micmelesse

  authored
  76443a4

  Copy full SHA for 76443a4

• add train script

  micmelesse

  committed
  89c2e39

  Copy full SHA for 89c2e39

• use local configs for testing

  micmelesse

  committed
  dfaf216

  Copy full SHA for dfaf216