Code

FP8 AllGather Support in Fairscale#1185

Code

Open

levendlee wants to merge 21 commits intongoyal_changes_for_pp_fp8_jiecaoyu_debugfacebookresearch/fairscale:ngoyal_changes_for_pp_fp8_jiecaoyu_debugfrom shikaili_fp8_allgather_no_pp_fixfacebookresearch/fairscale:shikaili_fp8_allgather_no_pp_fixCopy head branch name to clipboard

+322-111

Commits

Commits on Mar 29, 2024

• added option for no PG validation for faster init (#1161)

  Show description for 73ce4b4
  

  2 peopleauthored and

  levendlee

  committed
  73ce4b4

  Copy full SHA for 73ce4b4

• Mirros Jiecao's change.

  levendlee

  committed
  33457b3

  Copy full SHA for 33457b3

• Debug non-determinism issues.

  Show description for 70f5ff5
  levendlee

  committed
  70f5ff5

  Copy full SHA for 70f5ff5

• Moves amax update logic into params downcasting function.

  levendlee

  committed
  16c682d

  Copy full SHA for 16c682d

Commits on Apr 1, 2024

• Cleans up code.

  levendlee

  committed
  24a769f

  Copy full SHA for 24a769f

Commits on Apr 2, 2024

• Fix main_grad attribute checking.

  Show description for 3e2e77f
  levendlee

  committed
  3e2e77f

  Copy full SHA for 3e2e77f

Commits on Apr 9, 2024

• Fix no pp hanging error.

  Show description for 1be7aa0
  levendlee

  committed
  1be7aa0

  Copy full SHA for 1be7aa0

Commits on Apr 10, 2024

• Clean up shard offset calculation logic.

  levendlee

  committed
  57eb557

  Copy full SHA for 57eb557

Commits on Apr 17, 2024

• Unify compute dtype setting.

  levendlee

  committed
  e9e8f8e

  Copy full SHA for e9e8f8e

• Have FP16 and FP8 sharded in the same way.

  levendlee

  committed
  21f8e05

  Copy full SHA for 21f8e05

Commits on May 20, 2024

• added option for no PG validation for faster init (#1161)

  Show description for 8ec7c1d
  

  2 peopleauthored and

  levendlee

  committed
  8ec7c1d

  Copy full SHA for 8ec7c1d

• Mirros Jiecao's change.

  levendlee

  committed
  f27ab17

  Copy full SHA for f27ab17

• Debug non-determinism issues.

  Show description for fa9cf77
  levendlee

  committed
  fa9cf77

  Copy full SHA for fa9cf77

• Moves amax update logic into params downcasting function.

  levendlee

  committed
  6fa19e0

  Copy full SHA for 6fa19e0

• Cleans up code.

  levendlee

  committed
  80ffd54

  Copy full SHA for 80ffd54

• Fix main_grad attribute checking.

  Show description for afb2ca1
  levendlee

  committed
  afb2ca1

  Copy full SHA for afb2ca1

• Fix no pp hanging error.

  Show description for 25b2322
  levendlee

  committed
  25b2322

  Copy full SHA for 25b2322

• Clean up shard offset calculation logic.

  levendlee

  committed
  0d1502b

  Copy full SHA for 0d1502b

• Unify compute dtype setting.

  levendlee

  committed
  5edb109

  Copy full SHA for 5edb109

• Have FP16 and FP8 sharded in the same way.

  levendlee

  committed
  2df199f

  Copy full SHA for 2df199f

• Merge branch 'shikaili_fp8_allgather_no_pp_fix' of github.com:facebookresearch/fairscale into shikaili_fp8_allgather_no_pp_fix

  levendlee

  committed
  da36e31

  Copy full SHA for da36e31