Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Use forward and reverse tokenization
We enable the "reverse" tokenization mode, which allow to search both in forward and backward directions on tokens. The forward mode allows to search from left to right, while the reverse mode allows the opposite. For example, with the word "example", you can search "exam" in forward mode, and "ample" in reverse. We measured a 15-20% memory impact on enabling the reverse tokenization, compared to the forward mode. The "full" mode, allowing searching on all combinations, including in the middle of the word, comes with ~70% memory increase. So, we decided to to not enable it for now, as the cost/benefit ratio of such feature is unclear. The memory cost of enabling the reverse mode seems however reasonable.
- Loading branch information