Code

Training script#31

Code

Merged

jaidhyani merged 150 commits intomaindelphi-suite/delphi:mainfrom training-scriptdelphi-suite/delphi:training-scriptCopy head branch name to clipboardMar 19, 2024

+2,033-38

Commits

Commits on Mar 3, 2024

• add llama2.c submodule

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  5aa939a

  Copy full SHA for 5aa939a

• rename submodule to avoid import errors

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  27e7df4

  Copy full SHA for 27e7df4

• add llama2.c wrapper

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  f9dbff2

  Copy full SHA for f9dbff2

• draft training.py

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  1b394e0

  Copy full SHA for 1b394e0

• updated draft

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  9fba402

  Copy full SHA for 9fba402

• Adding a Mamba Class

  

  SrGonao

  authored and

  jettjaniak

  committed
  e11de88

  Copy full SHA for e11de88

• Moving stuff to the correct place

  

  SrGonao

  authored and

  jettjaniak

  committed
  6287121

  Copy full SHA for 6287121

• Not ready but idea there

  

  SrGonao

  authored and

  jettjaniak

  committed
  e42a8db

  Copy full SHA for e42a8db

• updated training script

  

  jannik-brinkmann

  authored and

  jettjaniak

  committed
  6642256

  Copy full SHA for 6642256

• formatting

  jettjaniak

  committed
  b9cdc78

  Copy full SHA for b9cdc78

• remove gitmodules

  jettjaniak

  committed
  294e792

  Copy full SHA for 294e792

• moved llama2c submodule

  jettjaniak

  committed
  c95c28b

  Copy full SHA for c95c28b

• llama2c update

  jettjaniak

  committed
  27891b5

  Copy full SHA for 27891b5

• fix import

  jettjaniak

  committed
  cd2c5f7

  Copy full SHA for cd2c5f7

• remove unused files

  jettjaniak

  committed
  145f8aa

  Copy full SHA for 145f8aa

• rename training_old -> training

  jettjaniak

  committed
  fe834b0

  Copy full SHA for fe834b0

• Moved Mamba

  

  SrGonao

  authored and

  jettjaniak

  committed
  f7eacd7

  Copy full SHA for f7eacd7

• Added type hinting

  

  SrGonao

  authored and

  jettjaniak

  committed
  a88be1f

  Copy full SHA for a88be1f

• Removed not needed file

  

  SrGonao

  authored and

  jettjaniak

  committed
  0d356af

  Copy full SHA for 0d356af

• Removed compile, amp train fp32

  

  SrGonao

  authored and

  jettjaniak

  committed
  646b3e7

  Copy full SHA for 646b3e7

• fixing black and isort

  

  SrGonao

  authored and

  jettjaniak

  committed
  b04975a

  Copy full SHA for b04975a

• add submodules to checkout in CI

  jettjaniak

  committed
  2b056ca

  Copy full SHA for 2b056ca

• pyproject.toml, moved isort cfg, excl. llama2c

  jettjaniak

  committed
  23f8a55

  Copy full SHA for 23f8a55

• isort: llama2c known_third_party

  jettjaniak

  committed
  f7cc6b7

  Copy full SHA for f7cc6b7

• limit pytest to tests/ directory

  jettjaniak

  committed
  e863604

  Copy full SHA for e863604

Commits on Mar 8, 2024

• Training_script_refactor (#54)

  Show description for a00425b
  jaidhyani

  authored
  a00425b

  Copy full SHA for a00425b

• It's actually a script now

  jaidhyani

  committed
  2aa2ea6

  Copy full SHA for 2aa2ea6

• lol copypasting

  jaidhyani

  committed
  ce4a6ca

  Copy full SHA for ce4a6ca

• cleanup

  jaidhyani

  committed
  83496e2

  Copy full SHA for 83496e2

• Adding support for config files

  jaidhyani

  committed
  30a1b30

  Copy full SHA for 30a1b30

• comments

  jaidhyani

  committed
  4098036

  Copy full SHA for 4098036

• flag arguments take priority over config file values

  jaidhyani

  committed
  96f2361

  Copy full SHA for 96f2361

• comments

  jaidhyani

  committed
  584c55d

  Copy full SHA for 584c55d

• gitignore .DS_Store file on macos

  jaidhyani

  committed
  d79d50c

  Copy full SHA for d79d50c

• remove training.sh

  jaidhyani

  committed
  9e1e9d8

  Copy full SHA for 9e1e9d8

• meeting notes and tweaks

  jaidhyani

  committed
  27f5d43

  Copy full SHA for 27f5d43

• configurable device

  jaidhyani

  committed
  e542237

  Copy full SHA for e542237

• Adding mamba implementation

  

  SrGonao

  authored and

  jaidhyani

  committed
  0524030

  Copy full SHA for 0524030

• mamba hacks, please forgive me

  jaidhyani

  committed
  29e986e

  Copy full SHA for 29e986e

• experimenting with cuda support in gh actions

  jaidhyani

  committed
  d8831bd

  Copy full SHA for d8831bd

• welp, that didn't work

  jaidhyani

  committed
  95d534d

  Copy full SHA for 95d534d

• remove tokenized_chunks_dataset

  jaidhyani

  committed
  8716bf7

  Copy full SHA for 8716bf7

• separate batch ordering and torch seeds

  jaidhyani

  committed
  7352eec

  Copy full SHA for 7352eec

• remove mamba.py

  jaidhyani

  committed
  c3e5ef7

  Copy full SHA for c3e5ef7

• refactoring

  jaidhyani

  committed
  7cb4ca7

  Copy full SHA for 7cb4ca7

• rm TODO

  jaidhyani

  committed
  7213aa4

  Copy full SHA for 7213aa4

• refactoring

  jaidhyani

  committed
  a8f7143

  Copy full SHA for a8f7143

• bughunt

  jaidhyani

  committed
  e038f31

  Copy full SHA for e038f31

Commits on Mar 9, 2024

• debugger config

  jaidhyani

  committed
  59fce94

  Copy full SHA for 59fce94

• typing improvements and bugfixes

  jaidhyani

  committed
  734b92e

  Copy full SHA for 734b92e

• add support for "x.y.z = val" style config

  jaidhyani

  committed
  2ed386c

  Copy full SHA for 2ed386c

• first steps towards Llama2HF support

  jaidhyani

  committed
  c928d3b

  Copy full SHA for c928d3b

• more debugging stuff

  jaidhyani

  committed
  54b095a

  Copy full SHA for 54b095a

• initial HF llama2 support

  jaidhyani

  committed
  fee0497

  Copy full SHA for fee0497

Commits on Mar 10, 2024

• debug more

  jaidhyani

  committed
  2665633

  Copy full SHA for 2665633

• Add support for preset model configs in script, specifying multiple config files

  jaidhyani

  committed
  2f65438

  Copy full SHA for 2f65438

• bughunt

  jaidhyani

  committed
  4c64774

  Copy full SHA for 4c64774

• fix beartype Callalble deprecation warning

  jaidhyani

  committed
  a9ac3dd

  Copy full SHA for a9ac3dd

• rm llamaconfig json accidentally added before

  jaidhyani

  committed
  3d7711a

  Copy full SHA for 3d7711a

• asdf

  jaidhyani

  committed
  c4e69d2

  Copy full SHA for c4e69d2

• script tweaks

  jaidhyani

  committed
  656228c

  Copy full SHA for 656228c

• better gigaconfig defaults

  jaidhyani

  committed
  27fdc79

  Copy full SHA for 27fdc79

• debug config is now just another preset; better documentation for subfield arguments

  jaidhyani

  committed
  398f1de

  Copy full SHA for 398f1de

• fix imports

  jaidhyani

  committed
  366b4b5

  Copy full SHA for 366b4b5

• remove upload_tokens

  jaidhyani

  committed
  d4a81e8

  Copy full SHA for d4a81e8

• Whoops. I should probably test things more before pushing them.

  jaidhyani

  committed
  5dc23e6

  Copy full SHA for 5dc23e6

• cleanup

  jaidhyani

  committed
  2f1a0a4

  Copy full SHA for 2f1a0a4

• script tweaks

  jaidhyani

  committed
  1f37228

  Copy full SHA for 1f37228

• added support for prioritizing configs

  jaidhyani

  committed
  adfd4b4

  Copy full SHA for adfd4b4

• refactoring (config_utils) to support notebook use

  jaidhyani

  committed
  e3b326c

  Copy full SHA for e3b326c

Commits on Mar 11, 2024

• fix Llama2ConfigData bug in gigaconfig (use default_factory)

  jaidhyani

  committed
  551a8de

  Copy full SHA for 551a8de

• make run_training return ModelTrainingState

  jaidhyani

  committed
  cd9a5b1

  Copy full SHA for cd9a5b1

• more config_utils

  jaidhyani

  committed
  ab19879

  Copy full SHA for ab19879

• cleanup run_training script

  jaidhyani

  committed
  bc8b43d

  Copy full SHA for bc8b43d

• training_demo notebook (for colab)

  jaidhyani

  committed
  859ae09

  Copy full SHA for 859ae09

• static files tweak

  jaidhyani

  committed
  fc3f021

  Copy full SHA for fc3f021

• estimate_mfu for llama2hf

  jaidhyani

  committed
  70e82ee

  Copy full SHA for 70e82ee

• Don't break if model export not available

  jaidhyani

  committed
  697f729

  Copy full SHA for 697f729

Commits on Mar 12, 2024

• 100k quick config

  jaidhyani

  committed
  a8f7a4f

  Copy full SHA for a8f7a4f

• torch.use_deterministic_algorithms for training

  jaidhyani

  committed
  997ec3a

  Copy full SHA for 997ec3a

• import Callable from collections.abc

  jaidhyani

  committed
  f9bd899

  Copy full SHA for f9bd899

• Move up torch.manual_seed before calling anything in torch

  jaidhyani

  committed
  89cee7c

  Copy full SHA for 89cee7c

• add wandb to requirements

  jaidhyani

  committed
  698365f

  Copy full SHA for 698365f

• factor out training config package + wandb_config

  jaidhyani

  committed
  7f6c180

  Copy full SHA for 7f6c180

• unused import

  jaidhyani

  committed
  662555d

  Copy full SHA for 662555d

• isort

  jaidhyani

  committed
  0699026

  Copy full SHA for 0699026

• initial mamba support

  jaidhyani

  committed
  4007b6a

  Copy full SHA for 4007b6a

• pip install wheel

  jaidhyani

  committed
  594033e

  Copy full SHA for 594033e

• pip install packaging

  jaidhyani

  committed
  cc010da

  Copy full SHA for cc010da

• come on, mamba_ssm, get it together

  jaidhyani

  committed
  bfb28c1

  Copy full SHA for bfb28c1

• requirements-nocuda.txt for gh actions

  jaidhyani

  committed
  f85d015

  Copy full SHA for f85d015

Commits on Mar 13, 2024

• Merge branch 'main' into training-script

  jaidhyani

  committed
  3c0010b

  Copy full SHA for 3c0010b

Commits on Mar 14, 2024

• mv ModelTypes to constants

  jaidhyani

  committed
  4870d92

  Copy full SHA for 4870d92

• deprecate llama2c support

  jaidhyani

  committed
  9eeb960

  Copy full SHA for 9eeb960

• clear out more llama2c stuff

  jaidhyani

  committed
  02ec1a1

  Copy full SHA for 02ec1a1

• we still need max_seq_len

  jaidhyani

  committed
  6cb9d52

  Copy full SHA for 6cb9d52

• factoring out optimizer params from config

  jaidhyani

  committed
  35cb7c4

  Copy full SHA for 35cb7c4

Commits on Mar 15, 2024

• fix broken test

  jaidhyani

  committed
  5e10db2

  Copy full SHA for 5e10db2

• model_args overhaul

  jaidhyani

  committed
  af6c0db

  Copy full SHA for af6c0db

• rm llama2c

  jaidhyani

  committed
  2a9d2c2

  Copy full SHA for 2a9d2c2

• replace DataLoader

  jaidhyani

  committed
  1225d93

  Copy full SHA for 1225d93

• run_dir to gigaconfig; output_run_dir; fix Generator type warning

  jaidhyani

  committed
  a9a791b

  Copy full SHA for a9a791b

• save results when training is done

  jaidhyani

  committed
  10b1a36

  Copy full SHA for 10b1a36

• save step in results

  jaidhyani

  committed
  fbfeaa5

  Copy full SHA for fbfeaa5

• include architecture and priority in llama preset configs

  jaidhyani

  committed
  5a54078

  Copy full SHA for 5a54078

• Merge branch 'training-script' into mamba_dev

  jaidhyani

  committed
  19e779f

  Copy full SHA for 19e779f

• update training demo

  jaidhyani

  committed
  a98aa42

  Copy full SHA for a98aa42

• mamba expectedly imports correctly

  jaidhyani

  committed
  53a6adf

  Copy full SHA for 53a6adf

• rm export_model

  jaidhyani

  committed
  0d69ede

  Copy full SHA for 0d69ede

• estimate_loss no longer depends on architecture

  jaidhyani

  committed
  e24fec4

  Copy full SHA for e24fec4

• add combine_configs (working towards frozen config for type safety)

  jaidhyani

  committed
  b9f682c

  Copy full SHA for b9f682c

• renaming/simplification

  jaidhyani

  committed
  13db6eb

  Copy full SHA for 13db6eb

• model_config refactor to approach type safety + frozen dataclasses

  jaidhyani

  committed
  15983a3

  Copy full SHA for 15983a3

• rm architectures.py

  jaidhyani

  committed
  ac8fb6a

  Copy full SHA for ac8fb6a

Commits on Mar 16, 2024

• new config system with type safety!

  jaidhyani

  committed
  43dd1b2

  Copy full SHA for 43dd1b2

• Support for optional config types (mamba and llama)

  jaidhyani

  committed
  6a21f2c

  Copy full SHA for 6a21f2c

• fix sample configs

  jaidhyani

  committed
  be2354f

  Copy full SHA for be2354f

• remove some unused model config args

  jaidhyani

  committed
  c128f96

  Copy full SHA for c128f96

• remove unused mamba.py

  jaidhyani

  committed
  28076ed

  Copy full SHA for 28076ed

• I thought I already deleted this?

  jaidhyani

  committed
  384b140

  Copy full SHA for 384b140

• rename to "initialize_model_training_state"

  jaidhyani

  committed
  5f64802

  Copy full SHA for 5f64802

Commits on Mar 17, 2024

• Support for mandatory fields in run_training

  jaidhyani

  committed
  e4ecd21

  Copy full SHA for e4ecd21

• ModelTypes

  jaidhyani

  committed
  a1a150d

  Copy full SHA for a1a150d

• output_dir is output_dir

  jaidhyani

  committed
  ec04579

  Copy full SHA for ec04579

• cleaner imports

  jaidhyani

  committed
  fb0c509

  Copy full SHA for fb0c509

• error if ModelConfig doesn't include config for chosen model type

  jaidhyani

  committed
  129587d

  Copy full SHA for 129587d

• no-op renames for clarity

  jaidhyani

  committed
  ab208aa

  Copy full SHA for ab208aa

• log levels

  jaidhyani

  committed
  076cd8b

  Copy full SHA for 076cd8b

Commits on Mar 18, 2024

• shebang & chmod +x on run_training.py

  jettjaniak

  committed
  be352fe

  Copy full SHA for be352fe

• renamed corpus dataset

  jettjaniak

  committed
  10d718e

  Copy full SHA for 10d718e

• removed llama2c references from pyproject.toml

  jettjaniak

  committed
  77c15ef

  Copy full SHA for 77c15ef

• removed .gitmodules

  jettjaniak

  committed
  a8d03a8

  Copy full SHA for a8d03a8

• removed scripts/upload_stories.py

  jettjaniak

  committed
  2d86063

  Copy full SHA for 2d86063

• test wandb_utils

  jaidhyani

  committed
  bed480c

  Copy full SHA for bed480c

• no llama2c, no .view, no need for enforcing contigious tensors

  jaidhyani

  committed
  f168310

  Copy full SHA for f168310

• Fix _unoptionalize

  jaidhyani

  committed
  6a67a02

  Copy full SHA for 6a67a02

• run_training.py --help when no args

  jaidhyani

  committed
  52745bc

  Copy full SHA for 52745bc

Commits on Mar 19, 2024

• script improvements: model-specific args moved to their own --help; fixed preset args; set default logging to INFO

  jaidhyani

  committed
  9e759a3

  Copy full SHA for 9e759a3

• rename llama to llama2

  jaidhyani

  committed
  5dfdb94

  Copy full SHA for 5dfdb94

• unused imports

  jaidhyani

  committed
  12dc6c4

  Copy full SHA for 12dc6c4

• set run name from config file

  jaidhyani

  committed
  a81f9e5

  Copy full SHA for a81f9e5

• set default output_dir based on run_name

  jaidhyani

  committed
  63cb45b

  Copy full SHA for 63cb45b

• remove in-progress testing file added by mistake

  jaidhyani

  committed
  be3069d

  Copy full SHA for be3069d

• add huggingface config

  jaidhyani

  committed
  d15f467

  Copy full SHA for d15f467

• fix config json that got broken somehow

  jaidhyani

  committed
  9c27671

  Copy full SHA for 9c27671

• save/load fix + huggingface uploading

  jaidhyani

  committed
  c259e97

  Copy full SHA for c259e97

• fix test that broken when renaming llama to llama2

  jaidhyani

  committed
  0a04b99

  Copy full SHA for 0a04b99

• unused import

  jaidhyani

  committed
  083cb1b

  Copy full SHA for 083cb1b

• fix validation sampling

  jaidhyani

  committed
  580d3c6

  Copy full SHA for 580d3c6

• remove eval_only

  jaidhyani

  committed
  14dc55e

  Copy full SHA for 14dc55e