diff --git a/LICENSE b/LICENSE index b09cd78..b97e57f 100644 --- a/LICENSE +++ b/LICENSE @@ -1,201 +1,20 @@ -Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - - TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - - 1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - - 2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - - 3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - - 4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - - 5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - - 6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - - 7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - - 8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - - 9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - - END OF TERMS AND CONDITIONS - - APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - - Copyright [yyyy] [name of copyright owner] - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. +Copyright (c) 2023-2025 Sang T. Truong et al. + +Permission is hereby granted, free of charge, to any person obtaining +a copy of this software and associated documentation files (the +"Software"), to deal in the Software without restriction, including +without limitation the rights to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell copies of the Software, and to +permit persons to whom the Software is furnished to do so, subject to +the following conditions: + +The above copyright notice and this permission notice shall be +included in all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE +LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION +WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml index dca667d..e1fb10b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,10 +23,11 @@ classifiers = [ "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", ] -license = { text = "Apache 2.0 License" } +license = { file = "LICENSE" } dependencies = [ "vllm>=0.5.2", @@ -36,7 +37,7 @@ dependencies = [ "transformers>=4.31.0", "ranx", "trl>=0.9.4", - "xformers==0.0.12", + "xformers>=0.0.12", "scipy>=1.10.0", "tiktoken", "tqdm>=4.65.0", diff --git a/requirements.txt b/requirements.txt index 2107d73..579ddd5 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,178 +1,33 @@ -aiofiles==23.2.1 -aiohttp==3.9.3 -aiosignal==1.3.1 -alabaster==0.7.16 -altair==5.3.0 -annotated-types==0.6.0 -anyio==4.3.0 -attrs==23.2.0 -Babel==2.14.0 -beautifulsoup4==4.12.3 +vllm>=0.5.2 +accelerate>=0.30.1 +peft>=0.11.1 +bitsandbytes>=0.40.2 +transformers>=4.31.0 +ranx +trl>=0.9.4 +xformers>=0.0.12 +scipy>=1.10.0 +tiktoken +tqdm>=4.65.0 +python-dotenv>=0.21.0 +openai>=1.14.3 +backoff>=2.2.1 +uncertainty-calibration +jiwer>=2.6.0 +underthesea>=2.4.1 +sentencepiece>=0.1.97 +nptyping>=1.4.0 +evaluate>=0.4.0 +scikit-learn>=1.2.2 +python-Levenshtein>=0.20.9 bert-score==0.3.13 -blanc==0.3.4 -blis==0.7.11 -boto3==1.34.74 -botocore==1.34.74 -catalogue==2.0.10 -cbor==1.0.0 -cbor2==5.6.2 -certifi==2024.2.2 -charset-normalizer==3.3.2 -click==8.1.7 -cloudpathlib==0.16.0 -colorama==0.4.6 -confection==0.1.4 -contourpy==1.2.0 -cramjam==2.8.3 -cycler==0.12.1 -cymem==2.0.8 -Cython==3.0.9 -datasets==2.18.0 -dill==0.3.8 -docutils==0.20.1 -emoji==2.11.0 -en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl#sha256=86cc141f63942d4b2c5fcee06630fd6f904788d2f0ab005cce45aadb8fb73889 -evaluate==0.4.1 -ez_setup==0.9 -fastapi==0.110.0 -fastparquet==2024.2.0 -ffmpy==0.3.2 -filelock==3.13.3 -fonttools==4.50.0 -frozenlist==1.4.1 -fsspec==2024.2.0 -furo==2024.1.29 -gin-config==0.5.0 -gradio==4.37.1 -gradio_client==0.5.0 -h11==0.14.0 +rouge-score==0.1.2 hLepor==0.0.4 -httpcore==1.0.5 -httpx==0.27.0 -huggingface-hub==0.22.2 -idna==3.6 -ijson==3.2.3 -imagesize==1.4.1 -importlib_resources==6.4.0 -inscriptis==2.5.0 -ir-datasets==0.5.6 -Jinja2==3.1.3 -jmespath==1.0.1 -joblib==1.3.2 -jsonschema==4.21.1 -jsonschema-specifications==2023.12.1 -kiwisolver==1.4.5 -langcodes==3.3.0 -Levenshtein==0.25.0 -llvmlite==0.42.0 -lxml==5.1.1 -lz4==4.3.3 -markdown-it-py==3.0.0 -MarkupSafe==2.1.5 -matplotlib==3.8.3 -maturin==1.5.1 -mdurl==0.1.2 -moverscore==1.0.3 -mpmath==1.3.0 -multidict==6.0.5 -multiprocess==0.70.16 -murmurhash==1.0.10 -networkx==3.2.1 -nltk==3.8.1 -nptyping==2.5.0 -numba==0.59.1 -numpy==1.26.4 -orjson==3.10.0 -packaging==24.0 -pandas==2.2.1 -parameterized==0.9.0 -pbr==6.0.0 -pillow==10.2.0 -portalocker==2.8.2 -preshed==3.0.9 -protobuf==5.26.1 -psutil==5.9.8 -pyarrow==15.0.2 -pyarrow-hotfix==0.6 -pyautocorpus==0.1.12 -pydantic==2.6.4 -pydantic_core==2.16.3 -pydub==0.25.1 +spacy>=3.5.0 +cython>=0.29.33 +stanza>=1.4.2 +moverscore>=0.0.19 pyemd==0.5.1 -Pygments==2.17.2 -pyparsing==3.1.2 -python-crfsuite==0.9.10 -python-dateutil==2.9.0.post0 -python-Levenshtein==0.25.0 -python-multipart==0.0.9 -pytorch-pretrained-bert==0.6.2 -pytz==2024.1 -PyYAML==6.0.1 -ranx==0.3.19 -rapidfuzz==3.7.0 -referencing==0.34.0 -regex==2023.12.25 -requests==2.31.0 -responses==0.18.0 -rich==13.7.1 -rpds-py==0.18.0 -s3transfer==0.10.1 -sacrebleu==2.4.1 -sacremoses==0.1.1 -safetensors==0.4.2 -scikit-learn==1.4.1.post1 -scipy==1.12.0 -seaborn==0.13.2 -semantic-version==2.10.0 -six==1.16.0 -smart-open==6.4.0 -sniffio==1.3.1 -snowballstemmer==2.2.0 -soupsieve==2.5 -spacy==3.7.4 -spacy-legacy==3.0.12 -spacy-loggers==1.0.5 -Sphinx==7.2.6 -sphinx-basic-ng==1.0.0b2 -sphinx-rtd-theme==2.0.0 -sphinxcontrib-apidoc==0.5.0 -sphinxcontrib-applehelp==1.0.8 -sphinxcontrib-devhelp==1.0.6 -sphinxcontrib-htmlhelp==2.0.5 -sphinxcontrib-jquery==4.1 -sphinxcontrib-jsmath==1.0.1 -sphinxcontrib-qthelp==1.0.7 -sphinxcontrib-serializinghtml==1.1.10 -srsly==2.4.8 -stanza==1.8.1 -starlette==0.36.3 -summ_eval==0.892 -sympy==1.12 -tabulate==0.9.0 -thinc==8.2.3 -threadpoolctl==3.4.0 -tokenizers==0.15.2 -toml==0.10.2 -toolz==0.12.1 -torch==2.2.2 -tqdm==4.66.2 -transformers==4.39.2 -trec-car-tools==2.6 -typer==0.9.4 -typing==3.7.4.3 -typing_extensions==4.10.0 -tzdata==2024.1 -uncertainty-calibration==0.1.4 -underthesea==6.8.0 -underthesea_core==1.0.4 -unlzw3==0.2.2 -urllib3==2.2.1 -uvicorn==0.29.0 -warc3-wet==0.2.3 -warc3-wet-clueweb09==0.2.5 -wasabi==1.1.2 -weasel==0.3.4 -websockets==11.0.3 -xxhash==3.4.1 -yarl==1.9.4 -zlib-state==0.1.6 +sacremoses>=0.0.53 +pytorch-pretrained-bert>=0.6.2 +google-generativeai>=0.3.2 \ No newline at end of file