Code

Adding configs related to DCLM#663

Code

Closed

abhinavg4 wants to merge 101 commits intofineweb_datastanford-crfm/levanter:fineweb_datafrom dclmstanford-crfm/levanter:dclmCopy head branch name to clipboard

+11,781-5,770

Commits

Commits on Jul 18, 2024

• Adding configs related to DCLM

  abhinavg4

  committed
  4aa2f2c

  Copy full SHA for 4aa2f2c

Commits on Jul 19, 2024

• Adding configs related to DCLM

  abhinavg4

  committed
  dde9ed0

  Copy full SHA for dde9ed0

• Adding Z loss

  abhinavg4

  committed
  b991e29

  Copy full SHA for b991e29

• pre commit changes

  abhinavg4

  committed
  bb674bb

  Copy full SHA for bb674bb

• Adding z_loss as part of train_lm.py

  abhinavg4

  committed
  6c99dfb

  Copy full SHA for 6c99dfb

• Reverting changes to llama.py for z_loss

  abhinavg4

  committed
  24469e7

  Copy full SHA for 24469e7

Commits on Jul 21, 2024

• Address capacity_type and env variables (#665)

  Ivan-Zhou

  authored
  76092c4

  Copy full SHA for 76092c4

Commits on Jul 24, 2024

• fix best effort test (#662)

  dlwh

  authored
  2e55856

  Copy full SHA for 2e55856

Commits on Jul 26, 2024

• Enable multislice in launch script (#666)

  Show description for 2e64f14
  blahBlahhhJ

  authored
  2e64f14

  Copy full SHA for 2e64f14

• Fineweb Text + Partial revert of kiloshard (#669)

  Show description for 4950a8e
  

  3 peopleauthored
  4950a8e

  Copy full SHA for 4950a8e

• log run_progress for a special x axis. Fixes #671 (#674)

  dlwh

  authored
  c17f653

  Copy full SHA for c17f653

Commits on Jul 28, 2024

• refactor trainer to always need a loss function, add z_loss (#672)

  dlwh

  authored
  ac0882d

  Copy full SHA for ac0882d

Commits on Aug 4, 2024

• Specify node_count as int in launch.py (#682)

  Ivan-Zhou

  authored
  cb3638e

  Copy full SHA for cb3638e

Commits on Aug 11, 2024

• Bump ray[default] from 2.32.0 to 2.34.0 (#683)

  Show description for 8111f29
  dependabot[bot]

  authored
  8111f29

  Copy full SHA for 8111f29

Commits on Aug 12, 2024

• wandb seems to be broken in latest release (#688)

  Show description for 04b0904
  dlwh

  authored
  04b0904

  Copy full SHA for 04b0904

Commits on Aug 14, 2024

• switch to setup tools and forget the config thing (#691)

  dlwh

  authored
  8c10a7a

  Copy full SHA for 8c10a7a

• set logging level to INFO

  dlwh

  committed
  e8b6003

  Copy full SHA for e8b6003

• update docker image, build it in ci, make the args point to the new version (#693)

  dlwh

  authored
  441af5c

  Copy full SHA for 441af5c

Commits on Aug 15, 2024

• RE-Allow adding extrenal directory to docker image (#695)

  Show description for ef6349c
  blahBlahhhJ

  authored
  ef6349c

  Copy full SHA for ef6349c

Commits on Aug 20, 2024

• Merge remote-tracking branch 'origin/dclm' into dclm

  dlwh

  committed
  e12c1b6

  Copy full SHA for e12c1b6

• match specs in dclm

  dlwh

  committed
  c9ebc88

  Copy full SHA for c9ebc88

Commits on Aug 21, 2024

• publish dev build

  dlwh

  committed
  7727696

  Copy full SHA for 7727696

• wip

  dlwh

  committed
  55e4d98

  Copy full SHA for 55e4d98

Commits on Aug 22, 2024

• fix imports and such

  dlwh

  committed
  de51236

  Copy full SHA for de51236

• get default zone from gcloud config

  dlwh

  committed
  7863989

  Copy full SHA for 7863989

• factor out docker command, build

  dlwh

  committed
  a550bb5

  Copy full SHA for a550bb5

• Merge remote-tracking branch 'origin/main' into dclm

  dlwh

  committed
  6341252

  Copy full SHA for 6341252

• Update beta2=0.95 (#701)

  Show description for 715a04a
  dlwh

  authored
  715a04a

  Copy full SHA for 715a04a

Commits on Aug 23, 2024

• publish full tpu image (#703)

  Show description for c0ae0f9
  dlwh

  authored
  c0ae0f9

  Copy full SHA for c0ae0f9

• fix incremental build on CI (#704)

  dlwh

  authored
  ca7c9a6

  Copy full SHA for ca7c9a6

• sigh

  dlwh

  committed
  d16482b

  Copy full SHA for d16482b

• grr (#705)

  dlwh

  authored
  c823c75

  Copy full SHA for c823c75

Commits on Aug 26, 2024

• Adding multiple configs (#685)

  Show description for 7ec7bb5
  abhinavg4

  authored
  7ec7bb5

  Copy full SHA for 7ec7bb5

• Expose infra as a package, publish dev builds (#696)

  Show description for 20faff3
  dlwh

  authored
  20faff3

  Copy full SHA for 20faff3

• Llama mixture (#706)

  Show description for 5c53a19
  abhinavg4

  authored
  5c53a19

  Copy full SHA for 5c53a19

• Fix base again (#707)

  Show description for 277e728
  dlwh

  authored
  277e728

  Copy full SHA for 277e728

Commits on Aug 27, 2024

• Fix tpu vm autoshutdown (#708)

  Show description for 0c628d5
  dlwh

  authored
  0c628d5

  Copy full SHA for 0c628d5

Commits on Aug 28, 2024

• suppress stderr in describe_tpu since it usually logs a dumb error (#710)

  dlwh

  authored
  97358f9

  Copy full SHA for 97358f9

• Merge remote-tracking branch 'origin/main' into dclm

  dlwh

  committed
  e9ca517

  Copy full SHA for e9ca517

Commits on Aug 29, 2024

• wip

  dlwh

  committed
  d674dd9

  Copy full SHA for d674dd9

• fix pyprojec.toml and pre-commit wandb issues (#712)

  dlwh

  authored
  4913df2

  Copy full SHA for 4913df2

• wip

  dlwh

  committed
  06dc304

  Copy full SHA for 06dc304

• fix device kind for mfu v5e (#713)

  dlwh

  authored
  ffa8e28

  Copy full SHA for ffa8e28

Commits on Sep 2, 2024

• add haps configuration (cycle lr schedule) (#709)

  Show description for fd7888d
  blahBlahhhJ

  authored
  fd7888d

  Copy full SHA for fd7888d

Commits on Sep 4, 2024

• Bump ray[default] from 2.34.0 to 2.35.0 (#714)

  Show description for 8dd32c6
  dependabot[bot]

  authored
  8dd32c6

  Copy full SHA for 8dd32c6

• use hf config from checkpoint by default (#715)

  dlwh

  authored
  ea4ea25

  Copy full SHA for ea4ea25

Commits on Sep 5, 2024

• Completely rework dataset/cache system: instant resume, perfect shuffle, stable mixtures and more (#716)

  Show description for fbe27bc
  dlwh

  authored
  fbe27bc

  Copy full SHA for fbe27bc

• unpin ray (#718)

  dlwh

  authored
  944a19f

  Copy full SHA for 944a19f

• bump equinox

  dlwh

  committed
  f13cfde

  Copy full SHA for f13cfde

Commits on Sep 6, 2024

• wip

  dlwh

  committed
  8d3dfe0

  Copy full SHA for 8d3dfe0

• 768

  dlwh

  committed
  8ecb7ea

  Copy full SHA for 8ecb7ea

Commits on Sep 10, 2024

• Update gcsfs requirement from <2024.7,>=2024.2 to >=2024.2,<2024.10 (#723)

  Show description for 9ba6b20
  dependabot[bot]

  authored
  9ba6b20

  Copy full SHA for 9ba6b20

• Update fsspec[http] requirement (#722)

  Show description for 78da902
  dependabot[bot]

  authored
  78da902

  Copy full SHA for 78da902

• Bump equinox from 0.11.4 to 0.11.5 (#721)

  Show description for 5b685c3
  dependabot[bot]

  authored
  5b685c3

  Copy full SHA for 5b685c3

Commits on Sep 11, 2024

• fix extra context docker build bug (#724)

  blahBlahhhJ

  authored
  a91ef81

  Copy full SHA for a91ef81

Commits on Sep 12, 2024

• Fix eqx (#725)

  Show description for 5c18557
  dlwh

  authored
  5c18557

  Copy full SHA for 5c18557

Commits on Sep 13, 2024

• get rid of eraconfig b/c draccus can't handle it

  dlwh

  committed
  b6f334e

  Copy full SHA for b6f334e

• ugh

  dlwh

  committed
  e33a905

  Copy full SHA for e33a905

• missed some prints

  dlwh

  committed
  2645efb

  Copy full SHA for 2645efb

• attempt at launching small fast in CI, add tqdm_loggable (#719)

  dlwh

  authored
  5fc4084

  Copy full SHA for 5fc4084

Commits on Sep 16, 2024

• Update datasets requirement from ~=2.18 to >=2.18,<4.0 (#732)

  Show description for d05036c
  dependabot[bot]

  authored
  d05036c

  Copy full SHA for d05036c

• Bump tensorstore from 0.1.64 to 0.1.65 (#731)

  Show description for ca16aa0
  dependabot[bot]

  authored
  ca16aa0

  Copy full SHA for ca16aa0

• Bump equinox from 0.11.3 to 0.11.6 (#730)

  Show description for 79fa64c
  dependabot[bot]

  authored
  79fa64c

  Copy full SHA for 79fa64c

Commits on Sep 18, 2024

• add bits-per-byte calculation to levanter (#729)

  dlwh

  authored
  07b3f16

  Copy full SHA for 07b3f16

Commits on Sep 22, 2024

• fix sequence parallel attention in splash attention (#738)

  Show description for fe3e2f3
  dlwh

  authored
  fe3e2f3

  Copy full SHA for fe3e2f3

Commits on Sep 24, 2024

• fix llama 3 rotary embeddings (#740)

  dlwh

  authored
  9fa3aaa

  Copy full SHA for 9fa3aaa

• Support for running in a Ray cluster (#737)

  dlwh

  authored
  2b42bfb

  Copy full SHA for 2b42bfb

• see if it's this file in particular (#742)

  dlwh

  authored
  8ad3074

  Copy full SHA for 8ad3074

• Update README.md (#656)

  devactivity-team

  authored
  541ff12

  Copy full SHA for 541ff12

Commits on Sep 25, 2024

• bump levanter version (#743)

  dlwh

  authored
  91be677

  Copy full SHA for 91be677

• Make new tokenization ~67% faster (#744)

  dlwh

  authored
  cd82fb3

  Copy full SHA for cd82fb3

• Adding supervised data config

  TheQuantumFractal

  committed
  43268e0

  Copy full SHA for 43268e0

• Fixing linter error

  TheQuantumFractal

  committed
  d6ad71f

  Copy full SHA for d6ad71f

Commits on Sep 26, 2024

• Tweaks to Ray TPU stuff (#747)

  Show description for 71bd696
  dlwh

  authored
  71bd696

  Copy full SHA for 71bd696

Commits on Sep 27, 2024

• Fixing supervised training

  TheQuantumFractal

  committed
  f5b32cd

  Copy full SHA for f5b32cd

• Making linter happy

  TheQuantumFractal

  committed
  6483b42

  Copy full SHA for 6483b42

• Making linter happy

  TheQuantumFractal

  committed
  45d41d8

  Copy full SHA for 45d41d8

Commits on Oct 4, 2024

• Simplify tokenization pipeline, make it work with large numbers of shards again, (re)add configuration metadata to cache (#752)

  Show description for b41838f
  

  dlwh

  and

  ahmeda14960

  authored
  b41838f

  Copy full SHA for b41838f

Commits on Oct 5, 2024

• allow mixture components to override cache_dir (#754)

  dlwh

  authored
  3bae9d3

  Copy full SHA for 3bae9d3

• a few final tweaks for marin runs (#755)

  dlwh

  authored
  9847728

  Copy full SHA for 9847728

Commits on Oct 9, 2024

• Update Audio Data Loader to Support Mixture Dataset (#758)

  Show description for 36b29fd
  Helw150

  authored
  36b29fd

  Copy full SHA for 36b29fd

• Update src/levanter/data/text.py

  Show description for 5370c72
  

  ahmeda14960

  and

  dlwh

  authored
  5370c72

  Copy full SHA for 5370c72

• Merge remote-tracking branch 'origin/main' into ksalahi/supervised-data

  ahmeda14960

  committed
  1063fd8

  Copy full SHA for 1063fd8

• address david's comments

  ahmeda14960

  committed
  2f625d3

  Copy full SHA for 2f625d3

• lint and minor

  ahmeda14960

  committed
  cf2c9e5

  Copy full SHA for cf2c9e5

• Adding supervised data config (#746)

  ahmeda14960

  authored
  8bed0aa

  Copy full SHA for 8bed0aa

Commits on Oct 10, 2024

• Add an actor pool for batch processing, switch to a thread for writing batches instead of a ray actor/task (#757)

  Show description for adf4b6d
  dlwh

  authored
  adf4b6d

  Copy full SHA for adf4b6d

• pre-commit

  dlwh

  committed
  36459da

  Copy full SHA for 36459da

• flaky hf

  dlwh

  committed
  6499656

  Copy full SHA for 6499656

• Fix actor pool in python 3.11, add better scaling down logic (#760)

  dlwh

  authored
  074477f

  Copy full SHA for 074477f

Commits on Oct 11, 2024

• Fix ray docs (#761)

  blahBlahhhJ

  authored
  1c0e10e

  Copy full SHA for 1c0e10e

• ensure everything always uses at least some CPU to avoid flooding ray head node, add code to change max size of actor pool

  dlwh

  committed
  51f9bf1

  Copy full SHA for 51f9bf1

• cap the size of the core writer task rather than the number of batches (#762)

  Show description for c3b3dd8
  dlwh

  authored
  c3b3dd8

  Copy full SHA for c3b3dd8

Commits on Oct 13, 2024

• add parquet support

  nikil-ravi

  committed
  52bff4f

  Copy full SHA for 52bff4f

• lint, shard name fix

  nikil-ravi

  committed
  af78281

  Copy full SHA for af78281

• pre-commit

  nikil-ravi

  committed
  8d09cfd

  Copy full SHA for 8d09cfd

• read as binary file

  nikil-ravi

  committed
  50715e9

  Copy full SHA for 50715e9

Commits on Oct 14, 2024

• simplify test

  nikil-ravi

  committed
  3fe8995

  Copy full SHA for 3fe8995

• Support Parquet files in ShardedDataSource (#764)

  Show description for fc26c74
  nikil-ravi

  authored
  fc26c74

  Copy full SHA for fc26c74

• fix crash in data loader caused by using stale array (#765)

  dlwh

  authored
  02f34ac

  Copy full SHA for 02f34ac

• Merge remote-tracking branch 'origin/main' into dclm

  dlwh

  committed
  0ea3eb4

  Copy full SHA for 0ea3eb4