feat(cache): use sqlite to store translation cache #324

awwaawwa · 2024-12-23T08:44:11Z

Implement database-related operations
Modify translator.py, have BaseTranslator handle cache matters
Add relevant tests

In addition, I modified the Test and Build Python Package Action to be triggered by any branch.

- add tortoise-orm>=0.22.2 to dependencies for database support

- update runner image to python 3.12 bookworm-slim for better compatibility - use uv instead of pip

- update runner to 'ubuntu-latest'

- update uv pip install command to include --system flag

- include creation of virtual environment using `uv venv`

- set ignore rules for specific flake8 checks - configure max line length to 88 characters

- include pep8-naming in the dev dependencies array - aim to improve code style adherence to PEP 8 naming conventions

This reverts commit ace5eb2.

- rename findSID to find_sid to align with snake_case naming convention

- remove tortoise-orm>=0.22.2 from dependencies - add peewee>=3.17.8 to dependencies

- introduce peewee ORM for database management - add `_TranslationCache` model with unique constraints - implement `TranslationCache` class for clean API usage - add `init_db` function to initialize database - improve translation cache handling with database storage

- bind and initialize in-memory test database for isolated testing - add tests for translation cache operations and engine distinction - include additional scenarios for parameter-based cache separation - ensure cleanup and teardown after tests to maintain test consistency

…test has no practical significance Although this test set the number of threads to 2, it did not test parallel translation paths. This test was actually directly testing BaseTranslator.

awwaawwa · 2024-12-23T10:56:13Z

For the issue described in #214, I have included both the translation engine and its parameters in the cache, and only when the engine, parameters, and original text are all identical, the cache is returned. However, parameters affecting the translation results need to be manually maintained in the translator.py file.

- implement a method to recursively sort dictionaries and lists in translation parameters - ensure translation engine parameters are serialized consistently for caching

- include tests for non-string parameter serialization to JSON - test consistent serialization of dicts regardless of key order - validate cache behavior with sorted parameter inputs - add tests for nested and complex parameter structures

- fix issue with incorrect handling of translate_engine_params types - ensure nested dictionaries are sorted recursively before conversion to JSON

- separate `update_params` method for better readability

- update parameter type annotations for clarity - improve default parameter handling in update_params method - ensure thread safety considerations are noted in comments

- add tests for thread safety in cache operations - ensure that all operations are consistent and correct under load

- rename update_params to replace_params for clarity - refactor related method calls to maintain consistency

- change method name for clarity and consistency - update related test cases to reflect the new method name

- rename the `translate` function to `do_translate`, making it easier for subclasses to override. - Implement cache-related processing in BaseTranslator. - Add parameters that affect translation results to the cache key.

…ache

- implement unit tests for translation caching functionalities - ensure cache interacts correctly with different input cases - verify cache behavior with overridden translations and parameters

awwaawwa · 2024-12-23T15:27:33Z

I tested OpenAITranslator locally, and it can cache correctly. The other translation engines have not been tested individually, but they should be fine.

…anslate method

- add test case to check translation without cache - assert results are not equal when ignoring cache

- implement test to ensure BaseTranslator raises NotImplementedError - verify that the translate method is not implemented for base class

awwaawwa · 2024-12-23T15:48:04Z

I haven't written the ignore_cache CLI argument and web UI parameter. ~~If anyone is interested in writing it, I hope it can be added after #329 is merged. Thank you very much.~~ This is quite a big task, it will need some more time to modify.

Byaidu · 2024-12-25T02:57:39Z

Thanks for contribution!

awwaawwa added 15 commits December 23, 2024 15:14

➕ feat(dependencies): add tortoise-orm dependency

5636c33

- add tortoise-orm>=0.22.2 to dependencies for database support

👷 ci(github-actions): update trigger branches for python build

a881589

🔧 chore(ci): update python build workflow to speed up

e214751

- update runner image to python 3.12 bookworm-slim for better compatibility - use uv instead of pip

👷 ci(workflow): update ci build configuration

57e5223

- update runner to 'ubuntu-latest'

🐛 fix(ci): correct dependency installation command

3568ff2

- update uv pip install command to include --system flag

👷 ci(venv): update python build workflow

2caa9e3

- include creation of virtual environment using `uv venv`

ci(github-actions): revent changes

cb330b4

🔧 chore(pyproject): add flake8 configuration

13d47bc

- set ignore rules for specific flake8 checks - configure max line length to 88 characters

➕ chore(pyproject): add pep8-naming to development dependencies

ace5eb2

- include pep8-naming in the dev dependencies array - aim to improve code style adherence to PEP 8 naming conventions

Revert "➕ chore(pyproject): add pep8-naming to development dependencies"

5f2a256

This reverts commit ace5eb2.

♻️ refactor(translator): improve function naming consistency

3e20049

- rename findSID to find_sid to align with snake_case naming convention

📦 build(dependencies): replace tortoise-orm with peewee in dependencies

4e6b8d9

- remove tortoise-orm>=0.22.2 from dependencies - add peewee>=3.17.8 to dependencies

test(converter): remove test for translation functionality since the …

ee8ae85

…test has no practical significance Although this test set the number of threads to 2, it did not test parallel translation paths. This test was actually directly testing BaseTranslator.

awwaawwa added 13 commits December 23, 2024 19:16

✨ feat(cache): add recursive sorting for translation params

ba0a1ec

- implement a method to recursively sort dictionaries and lists in translation parameters - ensure translation engine parameters are serialized consistently for caching

🐛 fix(cache): handle non-string translate_engine_params properly

b4b86d3

- fix issue with incorrect handling of translate_engine_params types - ensure nested dictionaries are sorted recursively before conversion to JSON

♻️ refactor(cache): restructure translation cache initialization

3508352

- separate `update_params` method for better readability

♻️ refactor(cache): enhance parameter handling in translation cache

2797197

- update parameter type annotations for clarity - improve default parameter handling in update_params method - ensure thread safety considerations are noted in comments

✅ test(test_cache): enhance caching tests for thread safety

053f332

- add tests for thread safety in cache operations - ensure that all operations are consistent and correct under load

♻️ refactor(cache): rename and refactor parameter methods

e5a3b09

- rename update_params to replace_params for clarity - refactor related method calls to maintain consistency

♻️ refactor(cache): rename append_params to add_params

03fd175

- change method name for clarity and consistency - update related test cases to reflect the new method name

♻️ refactor(converter): move cache-related operations to translator

26f5347

♻️ feat(translator): Implement cache-related operations

92c6b2f

- rename the `translate` function to `do_translate`, making it easier for subclasses to override. - Implement cache-related processing in BaseTranslator. - Add parameters that affect translation results to the cache key.

♻️ fix(translator): Remove redundant cache parameters

c11eeb4

format

c5fb3b4

format

8022264

awwaawwa closed this Dec 23, 2024

awwaawwa reopened this Dec 23, 2024

awwaawwa added 2 commits December 23, 2024 23:22

fix(translator): When the ignore cache flag is set, do not write to c…

ddf5426

…ache

✅ test(translator): add unit tests for translation caching

4cbacd2

- implement unit tests for translation caching functionalities - ensure cache interacts correctly with different input cases - verify cache behavior with overridden translations and parameters

awwaawwa marked this pull request as ready for review December 23, 2024 15:24

awwaawwa added 4 commits December 23, 2024 23:33

🐛 fix(translator): raise NotImplementedError in the BaseTranslator tr…

ba39fd5

…anslate method

✨ feat(translator): Add the ignore cache parameter to translate function

e4db1bd

✅ test(translator): add cache ignore tests for translation

28e5f94

- add test case to check translation without cache - assert results are not equal when ignoring cache

✅ test(test_translator): add unit test for BaseTranslator

97e9a90

- implement test to ensure BaseTranslator raises NotImplementedError - verify that the translate method is not implemented for base class

Byaidu merged commit a1eb6c8 into Byaidu:main Dec 25, 2024
4 checks passed

awwaawwa deleted the better_cache branch December 25, 2024 03:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cache): use sqlite to store translation cache #324

feat(cache): use sqlite to store translation cache #324

awwaawwa commented Dec 23, 2024 •

edited

Loading

awwaawwa commented Dec 23, 2024

awwaawwa commented Dec 23, 2024

awwaawwa commented Dec 23, 2024 •

edited

Loading

Byaidu commented Dec 25, 2024

feat(cache): use sqlite to store translation cache #324

feat(cache): use sqlite to store translation cache #324

Conversation

awwaawwa commented Dec 23, 2024 • edited Loading

awwaawwa commented Dec 23, 2024

awwaawwa commented Dec 23, 2024

awwaawwa commented Dec 23, 2024 • edited Loading

Byaidu commented Dec 25, 2024

awwaawwa commented Dec 23, 2024 •

edited

Loading

awwaawwa commented Dec 23, 2024 •

edited

Loading