Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Tiktoken support for TRTLLM #10306

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Add Tiktoken support for TRTLLM #10306

wants to merge 13 commits into from

Commits on Aug 15, 2024

  1. Add tiktoken tokenizer

    @meatybobby
    meatybobby committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    16e8eb7 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Merge branch 'main' into bobchen/add_tiktoken

    @meatybobby
    meatybobby committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    4cd21a3 View commit details
    Browse the repository at this point in the history

  2. Add special token

    @meatybobby
    meatybobby committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    f24af5f View commit details
    Browse the repository at this point in the history

  3. Remove unused import

    @meatybobby
    meatybobby committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    a61a02f View commit details
    Browse the repository at this point in the history

  4. Apply isort and black reformatting 

    Signed-off-by: meatybobby <[email protected]>
    @meatybobby
    meatybobby committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    b857102 View commit details
    Browse the repository at this point in the history

  5. Remove unused import

    @meatybobby
    meatybobby committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    672f4ef View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into bobchen/add_tiktoken

    @meatybobby
    meatybobby authored Aug 29, 2024
    Configuration menu
    Copy the full SHA
    b1b7a50 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Merge branch 'main' into bobchen/add_tiktoken

    @jubick1337
    jubick1337 authored Sep 17, 2024
    Configuration menu
    Copy the full SHA
    9df9f91 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. Merge branch 'main' into bobchen/add_tiktoken

    @meatybobby
    meatybobby committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    38b8042 View commit details
    Browse the repository at this point in the history

  2. Fix after merge

    @meatybobby
    meatybobby committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    986966d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into bobchen/add_tiktoken

    @meatybobby
    meatybobby authored Nov 20, 2024
    Configuration menu
    Copy the full SHA
    b367e08 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2024

  1. Change qnemo loading

    @meatybobby
    meatybobby committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    57762b9 View commit details
    Browse the repository at this point in the history

  2. Apply isort and black reformatting 

    Signed-off-by: meatybobby <[email protected]>
    @meatybobby
    meatybobby committed Nov 23, 2024
    Configuration menu
    Copy the full SHA
    5806398 View commit details
    Browse the repository at this point in the history