Code

Baseline data#61

Code

Draft

IanMagnusson wants to merge 69 commits intomainallenai/dolma:mainfrom baselinesallenai/dolma:baselinesCopy head branch name to clipboard

+3,692-3

Commits

Commits on Aug 28, 2023

• testing warc

  soldni

  committed
  4da316e

  Copy full SHA for 4da316e

• ignore

  soldni

  committed
  7fc2c9c

  Copy full SHA for 7fc2c9c

• testing slow

  soldni

  committed
  b35e6ee

  Copy full SHA for b35e6ee

• langdetect

  soldni

  committed
  4979ea7

  Copy full SHA for 4979ea7

• optional import

  soldni

  committed
  b6f96c3

  Copy full SHA for b6f96c3

Commits on Aug 29, 2023

• refactoring

  soldni

  committed
  bd903b2

  Copy full SHA for bd903b2

Commits on Aug 30, 2023

• wip

  soldni

  committed
  8c9e00a

  Copy full SHA for 8c9e00a

• style

  soldni

  committed
  2d97f76

  Copy full SHA for 2d97f76

Commits on Sep 6, 2023

• wip

  soldni

  committed
  d25287d

  Copy full SHA for d25287d

Commits on Sep 7, 2023

• test

  soldni

  committed
  0350a5f

  Copy full SHA for 0350a5f

Commits on Sep 8, 2023

• wip

  soldni

  committed
  c08bc6f

  Copy full SHA for c08bc6f

Commits on Sep 20, 2023

• configs

  soldni

  committed
  294ffca

  Copy full SHA for 294ffca

• hash sample

  soldni

  committed
  32610e5

  Copy full SHA for 32610e5

• small improvements

  soldni

  committed
  ab0d741

  Copy full SHA for ab0d741

Commits on Sep 21, 2023

• updated with output

  soldni

  committed
  d0cde79

  Copy full SHA for d0cde79

• more details

  soldni

  committed
  5562666

  Copy full SHA for 5562666

• updated readme

  soldni

  committed
  3909b7f

  Copy full SHA for 3909b7f

Commits on Sep 24, 2023

• decon wip

  soldni

  committed
  f1e463a

  Copy full SHA for f1e463a

• new confits

  soldni

  committed
  f9ed26d

  Copy full SHA for f9ed26d

• taggging content

  soldni

  committed
  a3b08c3

  Copy full SHA for a3b08c3

• Merge pull request #49 from allenai/main

  Show description for 07a745c
  soldni

  authored
  07a745c

  Copy full SHA for 07a745c

• changed name of file

  soldni

  committed
  ba2a413

  Copy full SHA for ba2a413

• fixes

  soldni

  committed
  4b2fb1b

  Copy full SHA for 4b2fb1b

• deal with empty docs/local files

  soldni

  committed
  534c3c2

  Copy full SHA for 534c3c2

• increased bloom size

  soldni

  committed
  0ccf67c

  Copy full SHA for 0ccf67c

Commits on Sep 25, 2023

• configs for rest of splits

  soldni

  committed
  59559d8

  Copy full SHA for 59559d8

• switching to option2

  soldni

  committed
  39405e8

  Copy full SHA for 39405e8

• forgot to do two more

  soldni

  committed
  8c2af40

  Copy full SHA for 8c2af40

Commits on Sep 26, 2023

• finding puctuation

  soldni

  committed
  c1c5b54

  Copy full SHA for c1c5b54

• tokenizer porting

  soldni

  committed
  abaf44d

  Copy full SHA for abaf44d

Commits on Sep 27, 2023

• configs

  soldni

  committed
  1363dff

  Copy full SHA for 1363dff

• books config

  soldni

  committed
  637ee26

  Copy full SHA for 637ee26

• more sources

  soldni

  committed
  b387f6a

  Copy full SHA for b387f6a

• configs

  soldni

  committed
  a099d59

  Copy full SHA for a099d59

• updated paths

  soldni

  committed
  5fe9e2b

  Copy full SHA for 5fe9e2b

• new c4

  soldni

  committed
  f7796d7

  Copy full SHA for f7796d7

• cleaned up

  soldni

  committed
  a8ff9dc

  Copy full SHA for a8ff9dc

• sampling

  soldni

  committed
  33cb671

  Copy full SHA for 33cb671

• sample

  soldni

  committed
  9cd6dcc

  Copy full SHA for 9cd6dcc

Commits on Sep 28, 2023

• sampling

  soldni

  committed
  6b369b4

  Copy full SHA for 6b369b4

• added tokenizer

  soldni

  committed
  2cebbe2

  Copy full SHA for 2cebbe2

• update all

  soldni

  committed
  7168c8b

  Copy full SHA for 7168c8b

• style

  soldni

  committed
  e60bb46

  Copy full SHA for e60bb46

• updated

  soldni

  committed
  cae806a

  Copy full SHA for cae806a

• configs

  soldni

  committed
  d64f225

  Copy full SHA for d64f225

• tokenizer cli wip

  soldni

  committed
  383c1cb

  Copy full SHA for 383c1cb

Commits on Oct 2, 2023

• cli

  soldni

  committed
  14b4724

  Copy full SHA for 14b4724

Commits on Oct 5, 2023

• wip big refactor

  soldni

  committed
  1e17a8f

  Copy full SHA for 1e17a8f

Commits on Oct 6, 2023

• fixed small bugs

  soldni

  committed
  110eaee

  Copy full SHA for 110eaee

• tokenizer log

  soldni

  committed
  e2d3f75

  Copy full SHA for e2d3f75

• fixed tokenizer paths

  soldni

  committed
  9c76d04

  Copy full SHA for 9c76d04

• added tokenizer small

  soldni

  committed
  b1d48a9

  Copy full SHA for b1d48a9

Commits on Oct 13, 2023

• fixed glob issue

  soldni

  committed
  d1659f1

  Copy full SHA for d1659f1

Commits on Oct 19, 2023

• falcon baseline thru mixing

  IanMagnusson

  committed
  e00b8c3

  Copy full SHA for e00b8c3

Commits on Oct 24, 2023

• trying to debug decon

  IanMagnusson

  committed
  3fe273e

  Copy full SHA for 3fe273e

Commits on Oct 25, 2023

• Merge branch 'main' into baselines

  soldni

  committed
  2b97d61

  Copy full SHA for 2b97d61

Commits on Oct 26, 2023

• Merge remote-tracking branch 'origin/main' into baselines

  IanMagnusson

  committed
  591b6bc

  Copy full SHA for 591b6bc

• pile

  IanMagnusson

  committed
  6d78c2a

  Copy full SHA for 6d78c2a

Commits on Oct 28, 2023

• redpajama

  IanMagnusson

  committed
  43ea152

  Copy full SHA for 43ea152

• Had to remove this to make decon work.

  IanMagnusson

  committed
  b30b206

  Copy full SHA for b30b206

Commits on Nov 7, 2023

• bump dolma version

  IanMagnusson

  committed
  c90345b

  Copy full SHA for c90345b

Commits on Nov 8, 2023

• pile tokenization

  IanMagnusson

  committed
  921e7ad

  Copy full SHA for 921e7ad

• falcon

  IanMagnusson

  committed
  50ab5a9

  Copy full SHA for 50ab5a9

Commits on Nov 18, 2023

• fix pile tokenization

  IanMagnusson

  committed
  5f30bd4

  Copy full SHA for 5f30bd4

• c4 decon

  IanMagnusson

  committed
  48e323e

  Copy full SHA for 48e323e

• c4 mixing

  IanMagnusson

  committed
  b09a89c

  Copy full SHA for b09a89c

• c4 tokenization

  IanMagnusson

  committed
  50ac33b

  Copy full SHA for 50ac33b

Commits on Nov 21, 2023

• mc4

  IanMagnusson

  committed
  dfb1c3b

  Copy full SHA for dfb1c3b

Commits on Dec 3, 2023

• cc only dolma

  IanMagnusson

  committed
  16295d8

  Copy full SHA for 16295d8