Code

Use new ResourceEnvs from Haliax#444

Code

Open

dlwh wants to merge 236 commits intomainstanford-crfm/levanter:mainfrom use_jampstanford-crfm/levanter:use_jampCopy head branch name to clipboard

+3,907-1,144

Commits

Commits on Nov 10, 2023

• wip

  dlwh

  committed
  fff4dfb

  Copy full SHA for fff4dfb

• wip

  dlwh

  committed
  740ad68

  Copy full SHA for 740ad68

• almost got new logger working

  dlwh

  committed
  4ad74a6

  Copy full SHA for 4ad74a6

• move the metrics stuff to its own file

  dlwh

  committed
  ad708e3

  Copy full SHA for ad708e3

• refactor and move stuff around

  dlwh

  committed
  6930fa9

  Copy full SHA for 6930fa9

• use generic infrastructure for summary

  dlwh

  committed
  abf7ec3

  Copy full SHA for abf7ec3

• wip towards a clean tracker package

  dlwh

  committed
  547cea8

  Copy full SHA for 547cea8

• wip

  dlwh

  committed
  2f481ed

  Copy full SHA for 2f481ed

• remove more wandb deps

  dlwh

  committed
  0b080fb

  Copy full SHA for 0b080fb

• tiny cleanup

  dlwh

  committed
  a324ae5

  Copy full SHA for a324ae5

• add some tests

  dlwh

  committed
  cfdcbb9

  Copy full SHA for cfdcbb9

• migrate alpaca-lora to new logger

  dlwh

  committed
  2ddc558

  Copy full SHA for 2ddc558

• sort of get tb to work

  dlwh

  committed
  9b0df08

  Copy full SHA for 9b0df08

Commits on Nov 15, 2023

• wip

  dlwh

  committed
  4fd2526

  Copy full SHA for 4fd2526

Commits on Nov 16, 2023

• wip

  dlwh

  committed
  a608a65

  Copy full SHA for a608a65

Commits on Nov 17, 2023

• Merge remote-tracking branch 'origin/main' into generic_logger

  dlwh

  committed
  176e5fa

  Copy full SHA for 176e5fa

• update configs, expose a method to find trackers

  dlwh

  committed
  8d34f6f

  Copy full SHA for 8d34f6f

• use trainer more to set logging

  dlwh

  committed
  42d7f2c

  Copy full SHA for 42d7f2c

• test the tracker get name stuff

  dlwh

  committed
  b887761

  Copy full SHA for b887761

• minor

  dlwh

  committed
  3ebd161

  Copy full SHA for 3ebd161

Commits on Nov 18, 2023

• making speccing the loss function simpler

  dlwh

  committed
  0d2efbc

  Copy full SHA for 0d2efbc

• stop requiring a loss function for every model definition

  dlwh

  committed
  f085287

  Copy full SHA for f085287

Commits on Nov 19, 2023

• wip

  dlwh

  committed
  f21cf4b

  Copy full SHA for f21cf4b

• jkacjkac

  dlwh

  committed
  01c8b87

  Copy full SHA for 01c8b87

Commits on Nov 22, 2023

• tweak

  dlwh

  committed
  e374697

  Copy full SHA for e374697

• register default hooks by default...

  dlwh

  committed
  921acf8

  Copy full SHA for 921acf8

Commits on Nov 24, 2023

• wip

  dlwh

  committed
  c8a5d6c

  Copy full SHA for c8a5d6c

• make it so we can evaluate if we have a cache but no sources

  dlwh

  committed
  639d334

  Copy full SHA for 639d334

• Merge branch 'cache_only' into extensible_trainer

  dlwh

  committed
  a3fdbaf

  Copy full SHA for a3fdbaf

Commits on Nov 25, 2023

• about got the checkpoint refactor done

  dlwh

  committed
  ec35e9b

  Copy full SHA for ec35e9b

• about got the checkpoint refactor done

  dlwh

  committed
  ed13502

  Copy full SHA for ed13502

• minor dead code removal

  dlwh

  committed
  634407e

  Copy full SHA for 634407e

Commits on Nov 26, 2023

• fix tests

  dlwh

  committed
  4208e03

  Copy full SHA for 4208e03

• cleanup

  dlwh

  committed
  9584884

  Copy full SHA for 9584884

• cleanup

  dlwh

  committed
  5a18678

  Copy full SHA for 5a18678

• minor

  dlwh

  committed
  c355106

  Copy full SHA for c355106

• Merge branch 'extensible_trainer' into doremi

  dlwh

  committed
  7a2ffc3

  Copy full SHA for 7a2ffc3

• wip

  dlwh

  committed
  d2e0de1

  Copy full SHA for d2e0de1

Commits on Nov 28, 2023

• register default hooks by default...

  dlwh

  committed
  be99631

  Copy full SHA for be99631

• wip

  dlwh

  committed
  5d033eb

  Copy full SHA for 5d033eb

• make it so we can evaluate if we have a cache but no sources

  dlwh

  committed
  c4a9160

  Copy full SHA for c4a9160

• about got the checkpoint refactor done

  dlwh

  committed
  b888065

  Copy full SHA for b888065

• about got the checkpoint refactor done

  dlwh

  committed
  c47ae97

  Copy full SHA for c47ae97

• minor dead code removal

  dlwh

  committed
  f0613c7

  Copy full SHA for f0613c7

• fix tests

  dlwh

  committed
  85c5678

  Copy full SHA for 85c5678

• cleanup

  dlwh

  committed
  8f84822

  Copy full SHA for 8f84822

• cleanup

  dlwh

  committed
  e54bad0

  Copy full SHA for e54bad0

• minor

  dlwh

  committed
  85dd89b

  Copy full SHA for 85dd89b

• generalize and extract the checkpoint loading logic so it can be used separately from trainer

  dlwh

  committed
  c61824e

  Copy full SHA for c61824e

• Revert "Temporarily Revert "Generic Tracker interface, support for TB logging (#367)""

  Show description for 7391475
  dlwh

  committed
  7391475

  Copy full SHA for 7391475

• wip

  dlwh

  committed
  2387f26

  Copy full SHA for 2387f26

• just about workable logger stuff

  dlwh

  committed
  6446bc0

  Copy full SHA for 6446bc0

• fix logging of config with a new levanter.initialize

  dlwh

  committed
  1b821d1

  Copy full SHA for 1b821d1

• missed a sopt

  dlwh

  committed
  afb6459

  Copy full SHA for afb6459

Commits on Nov 29, 2023

• on second thought, don't use tb in small_fast

  dlwh

  committed
  9d916bd

  Copy full SHA for 9d916bd

Commits on Nov 30, 2023

• Merge remote-tracking branch 'origin/dev' into extensible_trainer

  dlwh

  committed
  3d67552

  Copy full SHA for 3d67552

Commits on Dec 1, 2023

• main->dev (#375)

  Show description for 4d8cd68
  dlwh

  authored
  4d8cd68

  Copy full SHA for 4d8cd68

• Merge remote-tracking branch 'origin/dev' into extensible_trainer

  dlwh

  committed
  272e1e1

  Copy full SHA for 272e1e1

• Merge remote-tracking branch 'origin/main' into dev

  dlwh

  committed
  48ccdd3

  Copy full SHA for 48ccdd3

• supporting new trainer in gsm8k example

  dlwh

  committed
  3b27a08

  Copy full SHA for 3b27a08

Commits on Dec 2, 2023

• Merge branch 'dev' into extensible_trainer

  dlwh

  committed
  dcbed88

  Copy full SHA for dcbed88

• Merge remote-tracking branch 'origin/main' into extensible_trainer

  dlwh

  committed
  bbac4ef

  Copy full SHA for bbac4ef

Commits on Dec 7, 2023

• Add Sophia-H, some WIP support for Sophia-G (#372)

  Show description for f2842e9
  dlwh

  authored
  f2842e9

  Copy full SHA for f2842e9

Commits on Dec 10, 2023

• Merge remote-tracking branch 'origin/main' into dev

  dlwh

  committed
  6d6ae21

  Copy full SHA for 6d6ae21

• fix missing test changes

  dlwh

  committed
  83bea6e

  Copy full SHA for 83bea6e

Commits on Dec 11, 2023

• should use a tempdir

  dlwh

  committed
  92a615f

  Copy full SHA for 92a615f

• update gsm8k lora for sophia refactors

  dlwh

  committed
  cbee427

  Copy full SHA for cbee427

Commits on Dec 13, 2023

• Allow val change wandb dev (#384)

  Show description for e048581
  dlwh

  authored
  e048581

  Copy full SHA for e048581

• oops

  dlwh

  committed
  2bdf08b

  Copy full SHA for 2bdf08b

Commits on Dec 14, 2023

• do loss in fp32

  dlwh

  committed
  8f4aff3

  Copy full SHA for 8f4aff3

Commits on Dec 17, 2023

• Merge remote-tracking branch 'origin/dev' into extensible_trainer

  dlwh

  committed
  91eb588

  Copy full SHA for 91eb588

• more dead code removal

  dlwh

  committed
  2002832

  Copy full SHA for 2002832

• refix merge issues

  dlwh

  committed
  efa70a1

  Copy full SHA for efa70a1

• refix merge issues

  dlwh

  committed
  4cca0d1

  Copy full SHA for 4cca0d1

Commits on Dec 19, 2023

• Merge remote-tracking branch 'origin/main' into dev

  dlwh

  committed
  15e223d

  Copy full SHA for 15e223d

• Merge branch 'dev' into extensible_trainer

  dlwh

  committed
  904497b

  Copy full SHA for 904497b

• allow train_batch_size to be -1 if per_device_parallelism isn't -1

  dlwh

  committed
  2a90f57

  Copy full SHA for 2a90f57

Commits on Dec 21, 2023

• wip

  dlwh

  committed
  f05739a

  Copy full SHA for f05739a

• Merge remote-tracking branch 'origin/main' into extensible_trainer

  dlwh

  committed
  321bb30

  Copy full SHA for 321bb30

• fix performance regression in trainer.py

  dlwh

  committed
  38db3d5

  Copy full SHA for 38db3d5

• wth

  dlwh

  committed
  9b2813b

  Copy full SHA for 9b2813b

• mdkladmlkad

  dlwh

  committed
  e014c45

  Copy full SHA for e014c45

• jfakmfa

  dlwh

  committed
  95a391f

  Copy full SHA for 95a391f

• try this other approach to steps in TrainerState

  dlwh

  committed
  9f40f10

  Copy full SHA for 9f40f10

• fix checkpoint tests

  dlwh

  committed
  6df53f4

  Copy full SHA for 6df53f4

• fix gsm8k

  dlwh

  committed
  94aa8fa

  Copy full SHA for 94aa8fa

Commits on Dec 24, 2023

• update for new Haliax reduction functions

  dlwh

  committed
  5af6cb2

  Copy full SHA for 5af6cb2

• Merge branch 'extensible_trainer' into doremi

  dlwh

  committed
  e2b086f

  Copy full SHA for e2b086f

• wip

  dlwh

  committed
  84d3b33

  Copy full SHA for 84d3b33

Commits on Dec 25, 2023

• refactor grad_accum to have a separate microbatched

  dlwh

  committed
  85b42b0

  Copy full SHA for 85b42b0

Commits on Dec 27, 2023

• remove accumulate_gradients_sharded and just use microbatched directly

  dlwh

  committed
  c47c188

  Copy full SHA for c47c188

• add dtype for grad accum

  dlwh

  committed
  70b766f

  Copy full SHA for 70b766f

• small refactor

  dlwh

  committed
  57725ea

  Copy full SHA for 57725ea

• small refactor

  dlwh

  committed
  85f777b

  Copy full SHA for 85f777b

• fix key handling in grad accum

  dlwh

  committed
  f8d98fc

  Copy full SHA for f8d98fc

Commits on Dec 28, 2023

• make sophia work with non-trainables again

  dlwh

  committed
  5a8c77a

  Copy full SHA for 5a8c77a

• factor out some methods in train_step

  dlwh

  committed
  ff59e51

  Copy full SHA for ff59e51

• Merge branch 'extensible_trainer' into doremi

  dlwh

  committed
  c1718dd

  Copy full SHA for c1718dd

Commits on Dec 29, 2023

• make the initialize_from logic just use load_checkpoint_or_initialize

  dlwh

  committed
  d7a060d

  Copy full SHA for d7a060d

Commits on Dec 30, 2023

• on second thought load_from_checkpoint_or_initialize is the wrong abstraction

  dlwh

  committed
  8c44e64

  Copy full SHA for 8c44e64

• wip

  dlwh

  committed
  72f1e47

  Copy full SHA for 72f1e47

• on second thought load_from_checkpoint_or_initialize is the wrong abstraction

  dlwh

  committed
  add3df4

  Copy full SHA for add3df4

• Merge branch 'extensible_trainer' into doremi

  dlwh

  committed
  3ba7bf1

  Copy full SHA for 3ba7bf1

• wip factoring out the initial state stuff, again

  dlwh

  committed
  b6535b5

  Copy full SHA for b6535b5

• almost ready to try out doremi

  dlwh

  committed
  0d6f357

  Copy full SHA for 0d6f357

Commits on Jan 2, 2024

• almost ready to try out doremi

  dlwh

  committed
  7395e3c

  Copy full SHA for 7395e3c

Commits on Jan 3, 2024

• cleanup typing.overloads

  dlwh

  committed
  08996e6

  Copy full SHA for 08996e6

• use auto_sharded internally, undeprecate it b/c it has a point

  dlwh

  committed
  710900c

  Copy full SHA for 710900c

Commits on Jan 4, 2024

• fix docs

  dlwh

  committed
  5f9d96d

  Copy full SHA for 5f9d96d

• use new dot syntax in doremi

  dlwh

  committed
  04a74a1

  Copy full SHA for 04a74a1

Commits on Jan 8, 2024

• Merge remote-tracking branch 'origin/main' into doremi

  dlwh

  committed
  3249ca1

  Copy full SHA for 3249ca1

Commits on Jan 9, 2024

• fix mixture init with prngkey

  dlwh

  committed
  6a20c95

  Copy full SHA for 6a20c95

• add a simple InMemoryDataset that takes a list

  dlwh

  committed
  fd6d343

  Copy full SHA for fd6d343

• make keyiterator support just an int seed

  dlwh

  committed
  f5b8d00

  Copy full SHA for f5b8d00

• dumb bug in grad accum

  dlwh

  committed
  288e7fb

  Copy full SHA for 288e7fb

• fix some dumb bugs in new trainer

  dlwh

  committed
  c4da125

  Copy full SHA for c4da125

• test for doremi and associated fixes

  dlwh

  committed
  9257597

  Copy full SHA for 9257597

• depend on haliax dev for levanter dev

  dlwh

  committed
  317b10d

  Copy full SHA for 317b10d

• fix gsm8k_lora

  dlwh

  committed
  e4d1385

  Copy full SHA for e4d1385

• add a small_pile configuration

  dlwh

  committed
  ddcdac7

  Copy full SHA for ddcdac7

• make it len 2048

  dlwh

  committed
  792f769

  Copy full SHA for 792f769

Commits on Jan 10, 2024

• add doremi main

  dlwh

  committed
  e16b3af

  Copy full SHA for e16b3af

• we install haliax from source with the pyprojec.toml

  dlwh

  committed
  a272ca9

  Copy full SHA for a272ca9

• fix doremi test when doing multidevice

  dlwh

  committed
  e8d4b9d

  Copy full SHA for e8d4b9d

• add a pile_mixture.yaml

  dlwh

  committed
  5c489c1

  Copy full SHA for 5c489c1

• add a config for the small pile mixture

  dlwh

  committed
  1672148

  Copy full SHA for 1672148

• reduce default rows per chunk and see if that helps with these big subcorpora

  dlwh

  committed
  f485c5f

  Copy full SHA for f485c5f

• add some more logging to see if we can figure out why it's running out of memory still

  dlwh

  committed
  b2d8a58

  Copy full SHA for b2d8a58

Commits on Jan 11, 2024

• add some more logging to see if we can figure out why it's running out of memory still

  dlwh

  committed
  f76e466

  Copy full SHA for f76e466

• dumb

  dlwh

  committed
  fc78716

  Copy full SHA for fc78716

• don't run the slow tests in CI

  dlwh

  committed
  4927f67

  Copy full SHA for 4927f67

Commits on Jan 12, 2024

• wip

  dlwh

  committed
  1ceb00a

  Copy full SHA for 1ceb00a

Commits on Jan 13, 2024

• move the script, make it read off fsspec

  dlwh

  committed
  bc7108c

  Copy full SHA for bc7108c

• update for reverted Haliax change

  dlwh

  committed
  69ca4a4

  Copy full SHA for 69ca4a4

• update for reverted Haliax change

  dlwh

  committed
  ff5cb6d

  Copy full SHA for ff5cb6d

Commits on Jan 15, 2024

• update paths for pile mixture

  dlwh

  committed
  d6bf2c0

  Copy full SHA for d6bf2c0

• fix new import

  dlwh

  committed
  cc6044c

  Copy full SHA for cc6044c

• sigh

  dlwh

  committed
  415158a

  Copy full SHA for 415158a

• isjfo

  dlwh

  committed
  d2a90ae

  Copy full SHA for d2a90ae

• mdklmdlm

  dlwh

  committed
  058a9e0

  Copy full SHA for 058a9e0

• make logging list names of caches

  dlwh

  committed
  9f16fbe

  Copy full SHA for 9f16fbe

• lower resource requirements to see if this gets us processing faster

  dlwh

  committed
  b80ef6a

  Copy full SHA for b80ef6a

• let's make the chunkcachebuilders free

  dlwh

  committed
  6983ff0

  Copy full SHA for 6983ff0

• minimize use of optax internals

  dlwh

  committed
  5f42ad8

  Copy full SHA for 5f42ad8

Commits on Jan 16, 2024

• fix a crash i don't understand

  dlwh

  committed
  e6e8d27

  Copy full SHA for e6e8d27

• let's reduce requirements some more to see if we can keep everything running. should really solve this correctly

  dlwh

  committed
  ab29e92

  Copy full SHA for ab29e92

• let's reduce requirements some more to see if we can keep everything running. should really solve this correctly

  dlwh

  committed
  83f0616

  Copy full SHA for 83f0616

• silly

  dlwh

  committed
  def45cc

  Copy full SHA for def45cc

• ok so we're ok maybe

  dlwh

  committed
  de821ca

  Copy full SHA for de821ca

• don't fetch local

  dlwh

  committed
  cbddab8

  Copy full SHA for cbddab8

• wtf

  dlwh

  committed
  5d0f987

  Copy full SHA for 5d0f987

• what

  dlwh

  committed
  13cc556

  Copy full SHA for 13cc556

• ok, think we figured it out

  dlwh

  committed
  5afac01

  Copy full SHA for 5afac01

• less logging

  dlwh

  committed
  41ac362

  Copy full SHA for 41ac362

Commits on Jan 17, 2024

• toward turning the reader process into an actor too

  dlwh

  committed
  c621a08

  Copy full SHA for c621a08

• did we do it?

  dlwh

  committed
  4d92af9

  Copy full SHA for 4d92af9

• wandb: only force a step if commit is true

  dlwh

  committed
  257dfa7

  Copy full SHA for 257dfa7

• don't crash if n == 0

  dlwh

  committed
  23865a1

  Copy full SHA for 23865a1

• wandb: maybe this gives the behavior i want?

  dlwh

  committed
  70c00f1

  Copy full SHA for 70c00f1

• mklafmlkafml

  dlwh

  committed
  4c54365

  Copy full SHA for 4c54365

• Merge branch 'main' into dev

  dlwh

  committed
  1edeeef

  Copy full SHA for 1edeeef

• minimize use of optax internals

  dlwh

  committed
  6148381

  Copy full SHA for 6148381

Commits on Jan 18, 2024

• what

  dlwh

  committed
  8274cad

  Copy full SHA for 8274cad

• actually this is probably better

  dlwh

  committed
  b980c9f

  Copy full SHA for b980c9f

• actually this is probably better

  dlwh

  committed
  36f25a0

  Copy full SHA for 36f25a0

• dumb

  dlwh

  committed
  4da7112

  Copy full SHA for 4da7112

• mkladmlkad

  dlwh

  committed
  6147520

  Copy full SHA for 6147520

• fix key order for doremi

  dlwh

  committed
  e166a78

  Copy full SHA for e166a78

• remove excess log

  dlwh

  committed
  e6b581b

  Copy full SHA for e6b581b

• remove a redundant log message

  dlwh

  committed
  8c64be5

  Copy full SHA for 8c64be5

• fixed more bugs

  dlwh

  committed
  e89e709

  Copy full SHA for e89e709

• almost there

  dlwh

  committed
  33600fd

  Copy full SHA for 33600fd

• don't log a value for domains with no data on a step

  dlwh

  committed
  efbdd31

  Copy full SHA for efbdd31

Commits on Jan 30, 2024

• bring over the trainer-abstraction doc

  dlwh

  committed
  a810242

  Copy full SHA for a810242

• remove the wrapped loss_fn thing from trainer

  dlwh

  committed
  e49fb38

  Copy full SHA for e49fb38

• factor out a take_opt_step. need to decide where to put it

  dlwh

  committed
  13dc392

  Copy full SHA for 13dc392

• explicitly expose microbatch_size, use it in microbatched

  dlwh

  committed
  514da05

  Copy full SHA for 514da05

Commits on Jan 31, 2024

• comment about custom_jvp on microbatched

  dlwh

  committed
  f797a85

  Copy full SHA for f797a85

• unneeded cast

  dlwh

  committed
  4301930

  Copy full SHA for 4301930

• rename to mixed-precision.md

  dlwh

  committed
  d3416b1

  Copy full SHA for d3416b1

• cleanup ctors for BatchLoaders some

  dlwh

  committed
  9552909

  Copy full SHA for 9552909

• misc cleanup

  dlwh

  committed
  888d35e

  Copy full SHA for 888d35e

• wip

  dlwh

  committed
  49a409b

  Copy full SHA for 49a409b

• stable point: migrating to resourceenvs

  dlwh

  committed
  78d9342

  Copy full SHA for 78d9342

• require the jamp branch

  dlwh

  committed
  8e4e183

  Copy full SHA for 8e4e183

• knknajkdnjakd

  dlwh

  committed
  9a0ea6d

  Copy full SHA for 9a0ea6d

• try this?

  dlwh

  committed
  7c19f47

  Copy full SHA for 7c19f47

• cleanup and explain the issue

  dlwh

  committed
  d98a885

  Copy full SHA for d98a885

• see if we get the just-in-time conversion to bf16 that we want

  dlwh

  committed
  015dfb3

  Copy full SHA for 015dfb3

Commits on Feb 1, 2024

• wtf

  dlwh

  committed
  cddaf20

  Copy full SHA for cddaf20

• bypass microbatching if we don't need it?

  dlwh

  committed
  27949f8

  Copy full SHA for 27949f8

• switch to using hnn.Embedding in gpt2, which means we get the mixed precision right

  dlwh

  committed
  c3a9ce1

  Copy full SHA for c3a9ce1

• switch to using compute_envs where posisble use .shard instead

  dlwh

  committed
  0e91352

  Copy full SHA for 0e91352

• please pre-commit

  dlwh

  committed
  b57e1c7

  Copy full SHA for b57e1c7

• ok maybe we can do it?

  dlwh

  committed
  7fd46cb

  Copy full SHA for 7fd46cb

• sigh

  dlwh

  committed
  2ca4d97

  Copy full SHA for 2ca4d97

• Merge branch 'dev' into use_jamp

  dlwh

  authored
  b1e99e5

  Copy full SHA for b1e99e5

• fix test_weight_decay_mask.py

  dlwh

  committed
  a237a57

  Copy full SHA for a237a57

• use param_env everywhere

  dlwh

  committed
  5282694

  Copy full SHA for 5282694

• makldmlkad

  dlwh

  committed
  a013c4c

  Copy full SHA for a013c4c

Commits on Feb 2, 2024

• Merge remote-tracking branch 'origin/main' into use_jamp

  dlwh

  committed
  3049f89

  Copy full SHA for 3049f89

• Merge remote-tracking branch 'origin/main' into dev

  dlwh

  committed
  e4fcd67

  Copy full SHA for e4fcd67

• Merge branch 'dev' into use_jamp

  dlwh

  committed
  4a8d07a

  Copy full SHA for 4a8d07a

• Merge remote-tracking branch 'origin/main' into dev

  dlwh

  committed
  1983a1f

  Copy full SHA for 1983a1f

• Merge remote-tracking branch 'origin/main' into dev

  dlwh

  committed
  5312b87

  Copy full SHA for 5312b87

• Merge remote-tracking branch 'origin/main' into use_jamp

  dlwh

  committed
  90ed9cd

  Copy full SHA for 90ed9cd

• Merge branch 'dev' into use_jamp

  dlwh

  committed
  74096fa

  Copy full SHA for 74096fa

• wip debugging devices

  dlwh

  committed
  58ca1d7

  Copy full SHA for 58ca1d7

• let's try this?

  dlwh

  committed
  6ee6d8f

  Copy full SHA for 6ee6d8f

• so confused

  dlwh

  committed
  b485673

  Copy full SHA for b485673

• sigh

  dlwh

  committed
  de3162b

  Copy full SHA for de3162b

• ok i think i got it

  dlwh

  committed
  343367f

  Copy full SHA for 343367f

Commits on Feb 3, 2024

• Merge branch 'dev' into simple_first_cleanup

  dlwh

  committed
  1198bb2

  Copy full SHA for 1198bb2

• Merge branch 'simple_first_cleanup' into use_jamp

  dlwh

  committed
  a2d5934

  Copy full SHA for a2d5934

Commits on Feb 5, 2024

• wtf

  dlwh

  committed
  71b755e

  Copy full SHA for 71b755e

• this async seems like a bad idea

  dlwh

  committed
  07b5797

  Copy full SHA for 07b5797

• log perf numbers?

  dlwh

  committed
  b6e0c1d

  Copy full SHA for b6e0c1d

• more logging

  dlwh

  committed
  a3f9c7f

  Copy full SHA for a3f9c7f

• moar

  dlwh

  committed
  ce2db7b

  Copy full SHA for ce2db7b

• oops

  dlwh

  committed
  ea57bde

  Copy full SHA for ea57bde

• reduce logging some, try to figure out this stupid books problem

  dlwh

  committed
  d352b37

  Copy full SHA for d352b37

• ka dkla dkl

  dlwh

  committed
  0effef0

  Copy full SHA for 0effef0

• admaldl

  dlwh

  committed
  1e85d16

  Copy full SHA for 1e85d16

Commits on Feb 6, 2024

• fix the unnecessarily long time outs

  dlwh

  committed
  a25a8ce

  Copy full SHA for a25a8ce

• break really long docs into shorter docs b/c tokenizers is quadratic

  dlwh

  committed
  7c163a8

  Copy full SHA for 7c163a8

• kmklamdklad

  dlwh

  committed
  4125d3f

  Copy full SHA for 4125d3f

• maybe don't do the workaround so often?

  dlwh

  committed
  99a87e8

  Copy full SHA for 99a87e8

• is this the leak?!?

  dlwh

  committed
  5245e10

  Copy full SHA for 5245e10

• update for latest datasets

  dlwh

  committed
  8a6f59b

  Copy full SHA for 8a6f59b

• add a test to ensure we use the workaround for llama tokenizer

  dlwh

  committed
  002989b

  Copy full SHA for 002989b

• tweak timeouts in test

  dlwh

  committed
  3dfebe2

  Copy full SHA for 3dfebe2

• less spammy logging

  dlwh

  committed
  5a5a1f1

  Copy full SHA for 5a5a1f1

• cleanup, see if we can avoid crashing when one cache finishes

  dlwh

  committed
  4e6df52

  Copy full SHA for 4e6df52

• tweaks to tokenization/shard_cache throughput (#456)

  dlwh

  authored
  c2dccf2

  Copy full SHA for c2dccf2

• Merge remote-tracking branch 'origin/main' into use_jamp_harfleur

  dlwh

  committed
  f7a3d0a

  Copy full SHA for f7a3d0a

• Merge branch 'use_jamp_harfleur' into use_jamp

  dlwh

  committed
  c95ba8b

  Copy full SHA for c95ba8b