Code

Add Molmo (7B-D, 7B-O, 70B)#33962

Code

Open

molbap wants to merge 145 commits intohuggingface:mainhuggingface/transformers:mainfrom molbap:add_molmomolbap/transformers:add_molmoCopy head branch name to clipboard

+6,894-9

Commits

Commits on Oct 1, 2024

• add base convert keys + chat template

  molbap

  committed
  dc6fcac

  Copy full SHA for dc6fcac

Commits on Oct 2, 2024

• Merge branch 'main' into add_molmo

  molbap

  committed
  574e01f

  Copy full SHA for 574e01f

Commits on Oct 4, 2024

• draft: add up modular files for molmo

  molbap

  committed
  0bd413b

  Copy full SHA for 0bd413b

Commits on Oct 8, 2024

• Squashed commit of the following:

  Show description for 9e454e4
  molbap

  committed
  9e454e4

  Copy full SHA for 9e454e4

• sync changes

  molbap

  committed
  d82c471

  Copy full SHA for d82c471

• push a simple fix

  ArthurZucker

  committed
  339a8d3

  Copy full SHA for 339a8d3

• finish fixing

  ArthurZucker

  committed
  c0c25d6

  Copy full SHA for c0c25d6

• Merge branch 'main' into add_molmo

  molbap

  committed
  5ee6a44

  Copy full SHA for 5ee6a44

• suppress diff

  molbap

  committed
  33e43ec

  Copy full SHA for 33e43ec

Commits on Oct 10, 2024

• Merge branch 'main' into add_molmo

  molbap

  committed
  d23e1c1

  Copy full SHA for d23e1c1

• fix

  ArthurZucker

  committed
  c8c12fe

  Copy full SHA for c8c12fe

• style

  ArthurZucker

  committed
  0909c02

  Copy full SHA for 0909c02

• add config + 2d pooling

  molbap

  committed
  1799d20

  Copy full SHA for 1799d20

• suppress changes

  molbap

  committed
  fb133d4

  Copy full SHA for fb133d4

• Merge branch 'add_molmo' of github.com:molbap/transformers into add_molmo

  molbap

  committed
  5ba4105

  Copy full SHA for 5ba4105

• fix

  ArthurZucker

  committed
  a2a6a9b

  Copy full SHA for a2a6a9b

• Merge branch 'add_molmo' of github.com:molbap/transformers into add_molmo

  ArthurZucker

  committed
  8fe7a9f

  Copy full SHA for 8fe7a9f

Commits on Oct 11, 2024

• conversion works 🙌

  molbap

  committed
  20681f5

  Copy full SHA for 20681f5

• fixup

  molbap

  committed
  c85af98

  Copy full SHA for c85af98

• handle missing MOLMO_VISION_ATTENTION_CLASSES

  molbap

  committed
  35ea3cc

  Copy full SHA for 35ea3cc

• fix

  molbap

  committed
  ab79d0e

  Copy full SHA for ab79d0e

Commits on Oct 15, 2024

• fix fused keys mismatch

  molbap

  committed
  b9bdf99

  Copy full SHA for b9bdf99

• fix

  molbap

  committed
  98d5ccd

  Copy full SHA for 98d5ccd

• [Modular-breaking] add manually vision attention classes list

  molbap

  committed
  3bca742

  Copy full SHA for 3bca742

• finish weight conversion script

  molbap

  committed
  a13fe05

  Copy full SHA for a13fe05

Commits on Oct 16, 2024

• add more keys

  molbap

  committed
  fac8dfd

  Copy full SHA for fac8dfd

• flipped the linear layers

  molbap

  committed
  c1e5f19

  Copy full SHA for c1e5f19

• add pooling forward + draft general forward

  molbap

  committed
  a68e5f5

  Copy full SHA for a68e5f5

• modeling file with swiglu, forward(input_ids) passing

  molbap

  committed
  8298b80

  Copy full SHA for 8298b80

Commits on Oct 23, 2024

• BIG push of image processor

  molbap

  committed
  9f69c6b

  Copy full SHA for 9f69c6b

• add missing objects to init

  molbap

  committed
  0711e08

  Copy full SHA for 0711e08

Commits on Nov 5, 2024

• Merge branch 'main' into add_molmo

  molbap

  committed
  7efe22e

  Copy full SHA for 7efe22e

Commits on Nov 7, 2024

• fix up wrong channel dimension

  molbap

  committed
  f5bd3b0

  Copy full SHA for f5bd3b0

• fix typo

  molbap

  committed
  3ae884f

  Copy full SHA for 3ae884f

Commits on Nov 19, 2024

• add missing image token indices used in forward

  molbap

  committed
  3ef60c0

  Copy full SHA for 3ef60c0

• pad patch orderings

  molbap

  committed
  cf9d4ab

  Copy full SHA for cf9d4ab

• clean up conversion script

  molbap

  committed
  91a2d3c

  Copy full SHA for 91a2d3c

• remind that tests are TODO

  molbap

  committed
  0f7904f

  Copy full SHA for 0f7904f

Commits on Nov 21, 2024

• merge main

  zucchini-nlp

  committed
  577e347

  Copy full SHA for 577e347

Commits on Nov 24, 2024

• at least it runs like this

  zucchini-nlp

  committed
  b514041

  Copy full SHA for b514041

Commits on Nov 27, 2024

• add bos token

  molbap

  committed
  cf6cb5d

  Copy full SHA for cf6cb5d

• add bos token in prompt

  molbap

  committed
  26c517d

  Copy full SHA for 26c517d

• fix processor, missing batching img_mask

  molbap

  committed
  35c168d

  Copy full SHA for 35c168d

• fix image masks + batching

  molbap

  committed
  e7275c7

  Copy full SHA for e7275c7

• working version

  zucchini-nlp

  committed
  3e7530d

  Copy full SHA for 3e7530d

• +1 only on non masked indices

  zucchini-nlp

  committed
  4bbc89b

  Copy full SHA for 4bbc89b

• attemp 1 to make modular work

  zucchini-nlp

  committed
  54e072b

  Copy full SHA for 54e072b

• update conversion to fit all ckpt + chat template + clean up a bit

  zucchini-nlp

  committed
  1e99752

  Copy full SHA for 1e99752

• fix processing tests

  zucchini-nlp

  committed
  92a1f31

  Copy full SHA for 92a1f31

• add more tests (failing for now)

  zucchini-nlp

  committed
  42330e0

  Copy full SHA for 42330e0

• fix the conversion

  zucchini-nlp

  committed
  932f6d1

  Copy full SHA for 932f6d1

• done!

  zucchini-nlp

  committed
  aafb827

  Copy full SHA for aafb827

• nit

  zucchini-nlp

  committed
  36cc6dd

  Copy full SHA for 36cc6dd

• some tests are failing, coming back tomorrow

  zucchini-nlp

  committed
  f399c3a

  Copy full SHA for f399c3a

• adapt to any image format

  molbap

  committed
  7322227

  Copy full SHA for 7322227

• Merge branch 'add_molmo' of github.com:molbap/transformers into add_molmo

  molbap

  committed
  e4db50a

  Copy full SHA for e4db50a

Commits on Nov 28, 2024

• try to get batched generation working

  molbap

  committed
  205a755

  Copy full SHA for 205a755

• fix other tests, should work now

  zucchini-nlp

  committed
  eb61617

  Copy full SHA for eb61617

• adjust test for batching

  zucchini-nlp

  committed
  b77d947

  Copy full SHA for b77d947

• little bit of style

  zucchini-nlp

  committed
  ba4dd50

  Copy full SHA for ba4dd50

• docs + imports + automapping

  zucchini-nlp

  committed
  0e2d184

  Copy full SHA for 0e2d184

• remove images kwargs

  zucchini-nlp

  committed
  9a83706

  Copy full SHA for 9a83706

• some unused config attributes

  zucchini-nlp

  committed
  171eb8e

  Copy full SHA for 171eb8e

• remove additional vocab size and pad lm head

  zucchini-nlp

  committed
  35b517a

  Copy full SHA for 35b517a

• remove einops dependency

  molbap

  committed
  6a0cbc5

  Copy full SHA for 6a0cbc5

• Merge branch 'add_molmo' of github.com:molbap/transformers into add_molmo

  molbap

  committed
  5c7b141

  Copy full SHA for 5c7b141

• dont skip these tests

  zucchini-nlp

  committed
  434d4b1

  Copy full SHA for 434d4b1

• format + add integration testing

  molbap

  committed
  4645f97

  Copy full SHA for 4645f97

• Merge branch 'add_molmo' of github.com:molbap/transformers into add_molmo

  molbap

  committed
  48f2e21

  Copy full SHA for 48f2e21

Commits on Nov 29, 2024

• fix tests + fix 72B conversion

  molbap

  committed
  4bb4e48

  Copy full SHA for 4bb4e48

• fix format

  molbap

  committed
  e676782

  Copy full SHA for e676782

• modualr kinda works but adds extra classes like VisionVisionModel :(

  zucchini-nlp

  committed
  a74bda2

  Copy full SHA for a74bda2

• accomodate 7B-O version as well (broken)

  molbap

  committed
  2c428ae

  Copy full SHA for 2c428ae

• merge, fix conflicts and clean up modular extra code

  molbap

  committed
  d338153

  Copy full SHA for d338153

Commits on Dec 2, 2024

• fix 7B-O

  zucchini-nlp

  committed
  00376c4

  Copy full SHA for 00376c4

• remove unused code path

  zucchini-nlp

  committed
  48354fe

  Copy full SHA for 48354fe

Commits on Dec 3, 2024

• nit

  zucchini-nlp

  committed
  d738493

  Copy full SHA for d738493

• make modular work mostly

  zucchini-nlp

  committed
  d0e90d4

  Copy full SHA for d0e90d4

• fix imports

  zucchini-nlp

  committed
  f06b6d9

  Copy full SHA for f06b6d9

• update modulat last time

  zucchini-nlp

  committed
  9fc25c0

  Copy full SHA for 9fc25c0

• fix copies

  zucchini-nlp

  committed
  38dc9e8

  Copy full SHA for 38dc9e8

Commits on Dec 4, 2024

• fix copies

  zucchini-nlp

  committed
  eb77f3c

  Copy full SHA for eb77f3c

• fix tests

  zucchini-nlp

  committed
  190cc35

  Copy full SHA for 190cc35

Commits on Dec 6, 2024

• initial push of fast processor

  molbap

  committed
  84ed244

  Copy full SHA for 84ed244

• Merge branch 'add_molmo' of github.com:molbap/transformers into add_molmo

  molbap

  committed
  b4d48d5

  Copy full SHA for b4d48d5

Commits on Dec 10, 2024

• Merge branch 'main' into add_molmo

  molbap

  committed
  1298d08

  Copy full SHA for 1298d08

• fix various issues + tests

  molbap

  committed
  6687d43

  Copy full SHA for 6687d43

• add Molmo submodules as private

  molbap

  committed
  5f79577

  Copy full SHA for 5f79577

• do not test submodules

  molbap

  committed
  9e72758

  Copy full SHA for 9e72758

• [run-slow] molmo

  molbap

  committed
  439aed6

  Copy full SHA for 439aed6

• underscore prefixed method is not public

  molbap

  committed
  5a6a965

  Copy full SHA for 5a6a965

• fix tests

  molbap

  committed
  b9746a8

  Copy full SHA for b9746a8

• fix docs

  molbap

  committed
  2090ed6

  Copy full SHA for 2090ed6

• [run-slow] molmo

  molbap

  committed
  8ad3a25

  Copy full SHA for 8ad3a25

• Merge branch 'main' into add_molmo

  molbap

  committed
  0d10ee4

  Copy full SHA for 0d10ee4

• fix cache shape

  molbap

  committed
  9bd96f5

  Copy full SHA for 9bd96f5

• [run-slow] molmo

  molbap

  committed
  af5468b

  Copy full SHA for af5468b

• trigger CI

  molbap

  committed
  c02c6de

  Copy full SHA for c02c6de

• mark flaky test

  molbap

  committed
  5f35055

  Copy full SHA for 5f35055

• add missing objects

  molbap

  committed
  2b7af87

  Copy full SHA for 2b7af87

• add config to init

  molbap

  committed
  9f0f09d

  Copy full SHA for 9f0f09d

• more init fixes

  molbap

  committed
  74ebb24

  Copy full SHA for 74ebb24

• fix style

  molbap

  committed
  8b00c44

  Copy full SHA for 8b00c44

• fix?

  molbap

  committed
  d6403ad

  Copy full SHA for d6403ad

• fix

  molbap

  committed
  eb43cb9

  Copy full SHA for eb43cb9

• what is this again

  molbap

  committed
  33f0624

  Copy full SHA for 33f0624

• Merge branch 'main' into add_molmo

  molbap

  committed
  cc59007

  Copy full SHA for cc59007

• is this real life

  molbap

  committed
  23ae692

  Copy full SHA for 23ae692

• it was real life, fix broken eager

  molbap

  committed
  4c456e7

  Copy full SHA for 4c456e7

• fix attribtues

  molbap

  committed
  91f2820

  Copy full SHA for 91f2820

• this attention should be fixed

  molbap

  committed
  e2df6bc

  Copy full SHA for e2df6bc

Commits on Dec 11, 2024

• set 7b test to bf16

  molbap

  committed
  ae77cc6

  Copy full SHA for ae77cc6

• [run-slow] molmo

  molbap

  committed
  166b28a

  Copy full SHA for 166b28a

• Merge branch 'main' into add_molmo

  molbap

  committed
  50bcb7c

  Copy full SHA for 50bcb7c

• [run-slow] molmo

  molbap

  committed
  bf012d8

  Copy full SHA for bf012d8

• fix text (variability T4/A100)

  molbap

  committed
  6e0634b

  Copy full SHA for 6e0634b

Commits on Dec 12, 2024

• push clean Fast (x3!) image processor

  molbap

  committed
  8569fd0

  Copy full SHA for 8569fd0

• Merge branch 'main' into add_molmo

  molbap

  committed
  fd401bc

  Copy full SHA for fd401bc

• fix modular changes from main

  molbap

  committed
  86acf22

  Copy full SHA for 86acf22

Commits on Dec 16, 2024

• Merge branch 'main' into add_molmo

  molbap

  committed
  1ebea3c

  Copy full SHA for 1ebea3c

Commits on Dec 23, 2024

• push fast image proc with device check

  molbap

  committed
  5ebc6f0

  Copy full SHA for 5ebc6f0

• push fast image proc with device check

  molbap

  committed
  19d2689

  Copy full SHA for 19d2689

• format

  molbap

  committed
  c652bb9

  Copy full SHA for c652bb9

• images kwargs were missing

  molbap

  committed
  50c21e5

  Copy full SHA for 50c21e5

• merge and fix conflicts

  molbap

  committed
  092da76

  Copy full SHA for 092da76

• style

  molbap

  committed
  1254eac

  Copy full SHA for 1254eac

• update with modular conversion

  molbap

  committed
  bd39143

  Copy full SHA for bd39143

• add torch import

  molbap

  committed
  3efcb13

  Copy full SHA for 3efcb13

• style

  molbap

  committed
  56ae76f

  Copy full SHA for 56ae76f

• protect import

  molbap

  committed
  9417ff7

  Copy full SHA for 9417ff7

• fix modular

  molbap

  committed
  51f9336

  Copy full SHA for 51f9336

Commits on Jan 7, 2025

• Merge branch 'main' into add_molmo

  molbap

  committed
  3719481

  Copy full SHA for 3719481

Commits on Jan 8, 2025

• cherry-pick: cohere (from 67c3fcd)

  molbap

  committed
  f394b02

  Copy full SHA for f394b02

• fix modular with cohere interface

  molbap

  committed
  e418aa3

  Copy full SHA for e418aa3

• fixup cohere all imports

  molbap

  committed
  5af0b57

  Copy full SHA for 5af0b57

• fix bf16 test output

  molbap

  committed
  a574b93

  Copy full SHA for a574b93

• fix

  molbap

  committed
  9f3018d

  Copy full SHA for 9f3018d

• style

  molbap

  committed
  e2d1ba8

  Copy full SHA for e2d1ba8

Commits on Jan 9, 2025

• Merge branch 'main' into add_molmo

  molbap

  committed
  c872095

  Copy full SHA for c872095

• uniformize fast image processor

  molbap

  committed
  41ab3a7

  Copy full SHA for 41ab3a7

• Merge branch 'main' into add_molmo

  molbap

  committed
  dd74b78

  Copy full SHA for dd74b78

• fix merge

  molbap

  committed
  d052666

  Copy full SHA for d052666

• unbloat modular a tad

  molbap

  committed
  0a822f4

  Copy full SHA for 0a822f4

• fix import

  molbap

  committed
  8ebf44f

  Copy full SHA for 8ebf44f

• fix modular

  molbap

  committed
  4e6070f

  Copy full SHA for 4e6070f