Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Combined dataset feature #256

Closed
wants to merge 100 commits into from
Closed

Combined dataset feature #256

wants to merge 100 commits into from

Conversation

le1nux
Copy link
Member

@le1nux le1nux commented Sep 16, 2024 •
edited
Loading

What does this PR do?

This PR addresses issue #258

General Changes

  • ..

Breaking Changes

  • ..

Checklist before submitting final PR

  • My PR is minimal and addresses one issue in isolation
  • I have merged the latest version of the target branch into this feature branch
  • I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
  • I have run a sample config for model training
  • I have checked that all tests run through (python tests/tests.py)
  • I have updated the internal changelog (CHANGELOG_DEV.md)

le1nux and others added 30 commits August 29, 2024 13:41
@le1nux
refactor: introduced ResultItem to EvaluationResultBatch
2e0fedf
@le1nux
fix: fixed max_length warning in tokenizer
fc9d7b5
@le1nux
refactor: removed excessive print statements
3b7d74d
@le1nux
refactor: added ResultItem to other components
cf598a0
@le1nux
chore: removed more print statements
c0bcad4
@le1nux
refactor: removed unused parameter from IndexGenerator constructor
99bd571
@le1nux
feat: added configs for demo
431766a
@le1nux
feat: added demo diagrams
335c783
@le1nux
feat: added tokenizer config
1867232
@le1nux
feat: added demo jupyternobook
8d2f0b2
@le1nux
feat: added img
0dab792
@le1nux
refactor: more demo adaptations
3dd1f7b
@le1nux
chore: added banner
6398d62
@le1nux
chore: moved the diagrams to new tutorial
70edf9d
@le1nux
feat: added notebooks disclaimer
e6c5a59
@le1nux
feat: added tokenizer and training config
1f83d68
@le1nux
feat: added getting started jupyter notebook
fce1daa
@le1nux
refactor: updated modalities demo
fce1d5f
@le1nux
feat: added tokenizer configs for tutorial
1899bfd
@le1nux
feat: added wandb_storage to gitignore
cf2f7db
@le1nux
chore: renamed tutorial folder
4fce366
@le1nux
chore: removed old debug print statements
2f80136
@le1nux
chore: Merge branch 'main' into live_demo
9281405
@le1nux
fix: removed the max_length tag in huggingfae tokenizer. Setting it t…
42e7b1c 
…o a high value lead to huge memory allocation
@le1nux
fix: fixed failing warmstart test
5af079a
@le1nux @mali-git
Update src/modalities/config/component_factory.py
226719c 
Co-authored-by: Mehdi Ali <[email protected]>
@le1nux
feat: added optional rounding for metrics
7f3f8fa
@le1nux
refactor: lr now logged with full precision
61dba25
@le1nux
feat: added evaluator logging
fe6e38a
@le1nux
refactor: added logging of number parameters again
78fd763
le1nux and others added 17 commits September 15, 2024 14:23
@le1nux
feat: added information on missed out tokens percentages
522694d
@le1nux
feat: added warmstart tutorial
55db9a2
@le1nux
feat: updated components.md
02b09f2
@le1nux
chore: removed unnecessary math.ceil call
d183fdd
@le1nux
chore: Merge branch 'main' into warmstart_infrastructure_switch
be0d424
@flxst
chore: added short description for modalities in 15mins tutorial to R…
23bb463 
…EADME
@le1nux
feat: added README to getting started tutorial
e71f537
@le1nux
chore: further shortened path explanations in jupyter notebook
4756f39
@le1nux @flxst
Update README.md
c9c1ce4 
Co-authored-by: Felix Stollenwerk <[email protected]>
@le1nux
refactor: renamed examples to tutorials
658d0d0
@le1nux
chore: fixed type in variable name
f70e7b1
@le1nux @flxst
Update README.md
f7923f7 
Co-authored-by: Felix Stollenwerk <[email protected]>
@le1nux
refactor: consistent usage of progress_subscriber name
17b7a0c
@le1nux
chore: minor config renaming
9a3ff8c
@le1nux
feat: combined datasets implementation
dcb866f
@le1nux
chore: Merge branch 'warmstart_infrastructure_switch' into combined_d…
04c3189 
…ataset_feature
@le1nux
feat: added DistributedSampler (unmodified pytorch version)
63b370e
@le1nux le1nux changed the base branch from main to main_copy September 17, 2024 11:29
@le1nux le1nux changed the base branch from main_copy to main September 17, 2024 11:29
@le1nux le1nux changed the base branch from main to main_copy September 17, 2024 11:31
@le1nux le1nux changed the base branch from main_copy to main September 17, 2024 11:31
@le1nux le1nux closed this Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@le1nux @mali-git @flxst