Code

Add segment ids/sequence masking#827

Code

Merged

dlwh merged 14 commits intomainstanford-crfm/levanter:mainfrom segment_idsstanford-crfm/levanter:segment_idsCopy head branch name to clipboardDec 3, 2024

+202-38

Commits

Commits on Dec 1, 2024

• add segment ids

  dlwh

  committed
  039f319

  Copy full SHA for 039f319

• better test, combine segment masks

  dlwh

  committed
  9131078

  Copy full SHA for 9131078

Commits on Dec 2, 2024

• add with_segment_ids

  dlwh

  committed
  039ecd7

  Copy full SHA for 039ecd7

• actually use segment_ids

  dlwh

  committed
  8bf34bd

  Copy full SHA for 8bf34bd

• comments

  dlwh

  committed
  92247ac

  Copy full SHA for 92247ac

• cleanup

  dlwh

  committed
  381ce3a

  Copy full SHA for 381ce3a

• sigh

  dlwh

  committed
  de04991

  Copy full SHA for de04991

• tol

  dlwh

  committed
  bf14a2b

  Copy full SHA for bf14a2b

• sigh

  dlwh

  committed
  df0f0eb

  Copy full SHA for df0f0eb

• kk

  dlwh

  committed
  c29e071

  Copy full SHA for c29e071

• better vmap

  dlwh

  committed
  1bbb411

  Copy full SHA for 1bbb411

• this?

  dlwh

  committed
  4fff4c3

  Copy full SHA for 4fff4c3

• this?

  dlwh

  committed
  5052f36

  Copy full SHA for 5052f36

Commits on Dec 3, 2024

• enough device puts and we're good

  dlwh

  committed
  1a7729a

  Copy full SHA for 1a7729a