Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support new OS models: Zephyr and Yi #392

Draft
wants to merge 58 commits into
base: develop
Choose a base branch
from
Draft

Support new OS models: Zephyr and Yi #392

wants to merge 58 commits into from

Commits on Oct 17, 2023

  1. Add context length info. Refactor BuiltinTask and models to facilitat… 

    …e this.
    @rmitsch
    rmitsch committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    3aba660 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into feat/inf-doc-len 

    # Conflicts:
#	spacy_llm/tasks/builtin_task.py
    @rmitsch
    rmitsch committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    5699773 View commit details
    Browse the repository at this point in the history

  3. Add token count estimator plumbing.

    @rmitsch
    rmitsch committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    4213372 View commit details
    Browse the repository at this point in the history

  4. Add plumbing for mapper and reducer.

    @rmitsch
    rmitsch committed Oct 17, 2023
    Configuration menu
    Copy the full SHA
    f440ca4 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2023

  1. Add ShardMapper prototype.

    @rmitsch
    rmitsch committed Oct 18, 2023
    Configuration menu
    Copy the full SHA
    e47f762 View commit details
    Browse the repository at this point in the history

Commits on Oct 19, 2023

  1. Integrating mapping into prompt generation workflow.

    @rmitsch
    rmitsch committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    89a5510 View commit details
    Browse the repository at this point in the history

Commits on Oct 20, 2023

  1. Update response parsing and component to support sharding (WIP).

    @rmitsch
    rmitsch committed Oct 20, 2023
    Configuration menu
    Copy the full SHA
    086dec9 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. Fix shard & prompt flow.

    @rmitsch
    rmitsch committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    23718fc View commit details
    Browse the repository at this point in the history

  2. Fix shard & prompt flow.

    @rmitsch
    rmitsch committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    7ce670d View commit details
    Browse the repository at this point in the history

  3. Remove todo comments.

    @rmitsch
    rmitsch committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    0d75ea8 View commit details
    Browse the repository at this point in the history

  4. Fix Anthropic, Cohere, NoOp model tests.

    @rmitsch
    rmitsch committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    9da7098 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Merge branch 'develop' into feat/inf-doc-len

    @rmitsch
    rmitsch authored Oct 30, 2023
    Configuration menu
    Copy the full SHA
    0cb9afd View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Fix test_llm_pipe().

    @rmitsch
    rmitsch committed Oct 31, 2023
    Configuration menu
    Copy the full SHA
    f368412 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Fix type checking test.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b1f111d View commit details
    Browse the repository at this point in the history

  2. Fix span parsing tests.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    44a2787 View commit details
    Browse the repository at this point in the history

  3. Fix internal tests.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    6d8cdc7 View commit details
    Browse the repository at this point in the history

  4. Fix _CountTask.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    e712f41 View commit details
    Browse the repository at this point in the history

  5. Fix sentiment and summarization tasks and tests.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    985fd68 View commit details
    Browse the repository at this point in the history

  6. Fix Azure connection URL. Fix Model test pings.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    98842a2 View commit details
    Browse the repository at this point in the history

  7. Fix Lemma parsing.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b54a3d9 View commit details
    Browse the repository at this point in the history

  8. Start work on doc-to-shard property copying.

    @rmitsch
    rmitsch committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    9bf365d View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. Fix REL doc preprocessing.

    @rmitsch
    rmitsch committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    dddfaab View commit details
    Browse the repository at this point in the history

  2. Remove comment on doc attribute handling during sharding, as this is … 

    …done by spaCy's slicing directly.
    @rmitsch
    rmitsch committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    3af21b5 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Add reducer implementations.

    @rmitsch
    rmitsch committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    fee9ca7 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. Implement outstanding task reducers.

    @rmitsch
    rmitsch committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    e508499 View commit details
    Browse the repository at this point in the history

  2. Resolve merge conflicts.

    @rmitsch
    rmitsch committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    3218541 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. Add shardable/non-shardable LLM task typing distinction. Add support … 

    …for handling both types of tasks. Update tests.
    @rmitsch
    rmitsch committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    c104387 View commit details
    Browse the repository at this point in the history

Commits on Nov 21, 2023

  1. Merge branch 'develop' into feat/inf-doc-len 

    # Conflicts:
#	spacy_llm/tests/tasks/test_rel.py
    @rmitsch
    rmitsch committed Nov 21, 2023
    Configuration menu
    Copy the full SHA
    2c6d899 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2023

  1. Fix EL task.

    @rmitsch
    rmitsch committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    2502c4d View commit details
    Browse the repository at this point in the history

  2. Fix EL tokenization and highlighting partially.

    @rmitsch
    rmitsch committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    03055c5 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. Fix tokenization and whitespaces for EL task.

    @rmitsch
    rmitsch committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    4e4a2cd View commit details
    Browse the repository at this point in the history

  2. Fix merge conflicts.

    @rmitsch
    rmitsch committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    865acec View commit details
    Browse the repository at this point in the history

  3. Add new registry handlers (with context length and arbitrary model na… 

    …mes) for all REST models.
    @rmitsch
    rmitsch committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    694d5da View commit details
    Browse the repository at this point in the history

  4. Add sharding test with simple count task.

    @rmitsch
    rmitsch committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    5295400 View commit details
    Browse the repository at this point in the history

  5. Fix sharding algorithm.

    @rmitsch
    rmitsch committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    70e3643 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. Add test with simple count task.

    @rmitsch
    rmitsch committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    4321483 View commit details
    Browse the repository at this point in the history

  2. Add context length as init arg in HF models.

    @rmitsch
    rmitsch committed Nov 27, 2023
    Configuration menu
    Copy the full SHA
    ef6e738 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. Fix tests. Don't stringify IO lists if sharded.

    @rmitsch
    rmitsch committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    e3ff37d View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. Fix tests.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    056730a View commit details
    Browse the repository at this point in the history

  2. Add NER sharding test.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    196c235 View commit details
    Browse the repository at this point in the history

  3. Add REL and sentiment sharding tests.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    1f51a4a View commit details
    Browse the repository at this point in the history

  4. Add summary sharding tests.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    e18b302 View commit details
    Browse the repository at this point in the history

  5. Add EL sharding task. Fix bug in shard mapper.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    7c092ca View commit details
    Browse the repository at this point in the history

  6. Fix REL error with RELExample parsing.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    358ba72 View commit details
    Browse the repository at this point in the history

  7. Use regex for punctuation in REL conversion.

    @rmitsch
    rmitsch committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    0c96fb6 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2023

  1. Maintain custom doc attributes, incl. test.

    @rmitsch
    rmitsch committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    dc926bd View commit details
    Browse the repository at this point in the history

  2. Filter merge warnings in textcat reduction.

    @rmitsch
    rmitsch committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    5585174 View commit details
    Browse the repository at this point in the history

  3. Add Zephyr and Yi classes.

    @rmitsch
    rmitsch committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    6d3a4c8 View commit details
    Browse the repository at this point in the history

  4. Fix Yi model.

    @rmitsch
    rmitsch committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    57acfe4 View commit details
    Browse the repository at this point in the history

  5. Fix Yi model.

    @rmitsch
    rmitsch committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    2f1a905 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. Fix Yi and Zephyr processing.

    @rmitsch
    rmitsch committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    9821063 View commit details
    Browse the repository at this point in the history

  2. Remove deprecated comment.

    @rmitsch
    rmitsch committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    98e3e6c View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. Fix merge conflicts.

    @rmitsch
    rmitsch committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    513c2fb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'develop' into feat/new-os-models

    @rmitsch
    rmitsch committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    482af35 View commit details
    Browse the repository at this point in the history

  3. Change model used for Yi tests.

    @rmitsch
    rmitsch committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    3747a2f View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. Incorporate feedback.

    @rmitsch
    rmitsch committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    b2dff8f View commit details
    Browse the repository at this point in the history

Commits on Dec 13, 2023

  1. Skip Yi test failing in CI, but suceeding locally.

    @rmitsch
    rmitsch committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    dfe89ee View commit details
    Browse the repository at this point in the history

  2. Extend readme with links for Zephyr and Yi.

    @rmitsch
    rmitsch committed Dec 13, 2023
    Configuration menu
    Copy the full SHA
    69c3c76 View commit details
    Browse the repository at this point in the history