Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Count dataset tokens #1055

Draft
wants to merge 38 commits into
base: develop
Choose a base branch
from
Draft

Count dataset tokens #1055

wants to merge 38 commits into from

Commits on Oct 11, 2024

  1. Update GenerateOutput type

    @plaguss
    plaguss committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    caea13f View commit details
    Browse the repository at this point in the history

  2. Add draft function to compute number of tokens given a tokenizer

    @plaguss
    plaguss committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    f7d7a0e View commit details
    Browse the repository at this point in the history

  3. Refactor llm generation to return generations and statistics

    @plaguss
    plaguss committed Oct 11, 2024
    Configuration menu
    Copy the full SHA
    dbcfafa View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2024

  1. Move statistics from the LLM to distilabel_metadata row

    @plaguss
    plaguss committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    a0bf204 View commit details
    Browse the repository at this point in the history

  2. Update tests and LLM outputs to run with generations and statistics a… 

    …s the outputs
    @plaguss
    plaguss committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    a51ce59 View commit details
    Browse the repository at this point in the history

  3. Openai computed tokens

    @plaguss
    plaguss committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    fe5d4c5 View commit details
    Browse the repository at this point in the history

  4. First version of async llms with statistics

    @plaguss
    plaguss committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    394984f View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Return generations with list of strings and token count from _raw_res… 

    …ponse
    @plaguss
    plaguss committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    9003fa0 View commit details
    Browse the repository at this point in the history

  2. Passing tests for inference endpoints

    @plaguss
    plaguss committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    4cf0e2f View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Testing vLLM with statistics

    @plaguss
    plaguss committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    1b6d15c View commit details
    Browse the repository at this point in the history

  2. Refactor statistics module to utils and output preparation to avoid c… 

    …ode duplication
    @plaguss
    plaguss committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    8923880 View commit details
    Browse the repository at this point in the history

  3. Refactor to remove code duplication

    @plaguss
    plaguss committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    608e8b6 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Fix async llms not returning properly the generations grouped by num_… 

    …generations
    @plaguss
    plaguss committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    8c35af5 View commit details
    Browse the repository at this point in the history

  2. Fix async llms not processing multiple generations

    @plaguss
    plaguss committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    6d19de7 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. Fix vllm sorting mechanism and add mocked generate method to the test… 

    … suite
    @plaguss
    plaguss committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    9746d75 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Checkpoint

    @plaguss
    plaguss committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    f108670 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Fix tests from merge responses and group generations

    @plaguss
    plaguss committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    c8063a4 View commit details
    Browse the repository at this point in the history

  2. Move import to guarded type hint

    @plaguss
    plaguss committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    6f6769a View commit details
    Browse the repository at this point in the history

  3. Fix tests to work with statistics

    @plaguss
    plaguss committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    4971f26 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Return void list in case of no generations

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    74f81ad View commit details
    Browse the repository at this point in the history

  2. Update function to allow flatten inner list in values of dicts, and a… 

    …dd new merge_dicts to help merging user-assistant messages in magpie
    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    8ff6e13 View commit details
    Browse the repository at this point in the history

  3. Fix dummy magpie llm

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d8f2a8b View commit details
    Browse the repository at this point in the history

  4. Update tests for magpie

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    70898da View commit details
    Browse the repository at this point in the history

  5. Create statistics entry in distilabel_metadata with the name of the step

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    314e171 View commit details
    Browse the repository at this point in the history

  6. Update magpie code to work with the new llm.generate behaviour

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    e3e81d9 View commit details
    Browse the repository at this point in the history

  7. Update tests with the llm generate output format

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    241d899 View commit details
    Browse the repository at this point in the history

  8. Merge branch 'develop' of https://github.com/argilla-io/distilabel in… 

    …to llm-generate-upgrade
    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    a657859 View commit details
    Browse the repository at this point in the history

  9. Fix pending tests

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    40e408d View commit details
    Browse the repository at this point in the history

  10. Fix test failing with vllm version upgrade

    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6c2e1fd View commit details
    Browse the repository at this point in the history

  11. Another fix including tokenizer for our llm to work and to avoid outl… 

    …ines complaining
    @plaguss
    plaguss committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    d28a798 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Fix dummy offline batch generation

    @plaguss
    plaguss committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    1bc28ba View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. Merge and fix conflict

    @plaguss
    plaguss committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    13f42b0 View commit details
    Browse the repository at this point in the history

  2. Compute tokens using the tokenizer if available

    @plaguss
    plaguss committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    edbea28 View commit details
    Browse the repository at this point in the history

  3. Update docs to include references to the new outputs of the LLMs incl… 

    …uding statistics
    @plaguss
    plaguss committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    e97f901 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2024

  1. Merge branch 'develop' of https://github.com/argilla-io/distilabel in… 

    …to llm-generate-upgrade
    @plaguss
    plaguss committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    991320a View commit details
    Browse the repository at this point in the history

  2. Update template card to include the statistics table

    @plaguss
    plaguss committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    84cfe8f View commit details
    Browse the repository at this point in the history

  3. Add draft with statistics table with token summary

    @plaguss
    plaguss committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    6644d10 View commit details
    Browse the repository at this point in the history

  4. Fix jinja2 template

    @plaguss
    plaguss committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    5741cd1 View commit details
    Browse the repository at this point in the history