Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

epic: dictionary-based word-breakers 🔬 #12142

Draft
wants to merge 43 commits into
base: master
Choose a base branch
from
Draft

epic: dictionary-based word-breakers 🔬 #12142

wants to merge 43 commits into from
+708 −2

Conversation

mcdurdin
Copy link
Member

@mcdurdin mcdurdin commented Aug 9, 2024

No description provided.

jahorton and others added 21 commits August 9, 2024 09:40
@jahorton
feat(common/models/wordbreakers): starting on dictionary-based wordbr…
425a0a0 
…eaking
@jahorton
feat(common/models/wordbreakers): actual first-pass implementation
7b456c1 
Only wordbreaks anything AFTER the last space / ZWNJ.  Doesn't bother with anything before it.
@jahorton
feat(common/models/wordbreakers): pass 2 - should now tokenize full c…
2da7b7c 
…ontext
@jahorton
feat(common/models/wordbreakers): dict-breaker helper unit tests (BMP)
5568167
@jahorton
feat(common/models/wordbreakers): dict-breaker helper unit tests (non…
d40a09d 
…-BMP)
@jahorton
feat(common/models/wordbreakers): dict-breaker unit tests with simple…
10f91e0 
… mocked-fixture
@jahorton
fix(common/models/wordbreakers): blocks empty span output on empty co…
1d07e9f 
…ntext
@jahorton
fix(common/models): update re base branch change
3e82434
@jahorton
docs(common/models): updates dict-breaker comments
56e4052
@jahorton
change(common/models/wordbreakers): allows wordbreaker unit tests to …
66c156e 
…use Tries
@jahorton
feat(common/models/wordbreakers): baby's first khmer wordbreaking test
3181a4c
@jahorton
chore(common/models/wordbreakers): comment tweak
07f6747
@jahorton
feat(common/models/wordbreakers): rejoins adjacent single-point spans…
8c5e54a 
… from unmatched chars in path
@jahorton
fix(common/models/wordbreakers): handling of penalty transitions
766992d
@jahorton
change(common/models): use spread operator to split on codepoints
99e1f4b 
Addresses #10973 (comment)
@jahorton
chore(common/models/wordbreakers): Merge branch 'feat/common/models/w…
51b62e6 
…ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
@jahorton
chore(common/models/wordbreakers): Merge base branch into feat/common…
295c70b 
…/models/wordbreakers/fuse-dict-unmatched-chars
@jahorton
chore(common/models): drops unit tests for replaced func
8ae6a62
@jahorton
chore(common/models/wordbreakers): Merge branch 'feat/common/models/w…
258763b 
…ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
@jahorton
chore(common/models/wordbreakers): Merge base branch fixes into feat/…
8942883 
…common/models/wordbreakers/fuse-dict-unmatched-chars
@mcdurdin
chore: establish dictionary breakers epic
001126b
@keymanapp-test-bot keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Aug 9, 2024
@keymanapp-test-bot
Copy link

keymanapp-test-bot bot commented Aug 9, 2024 •
edited
Loading

User Test Results

Test specification and instructions

ERROR: user tests have not yet been defined

Test Artifacts

@keymanapp-test-bot keymanapp-test-bot bot added this to the A18S8 milestone Aug 9, 2024
@darcywong00 darcywong00 modified the milestones: A18S8, A18S9 Aug 17, 2024
@mcdurdin mcdurdin added this to the A18S19 milestone Aug 27, 2024
jahorton and others added 9 commits August 27, 2024 10:15
@jahorton
Merge pull request #12139 from keymanapp/feat/common/models/wordbreak…
9ca5d0a 
…ers/dict-breaker-start

feat(common/models/wordbreakers): begin development of dictionary-based wordbreaking algorithm 🔬
@jahorton
Merge pull request #12140 from keymanapp/change/common/models/wordbre…
b0aaf1e 
…akers/unit-test-trie-access

change(common/models/wordbreakers): allow wordbreaker tests to access TrieModel implementation 🔬
@jahorton
Merge pull request #12141 from keymanapp/feat/common/models/wordbreak…
2e7e9d7 
…ers/fuse-dict-unmatched-chars

feat(common/models/wordbreakers): fuse adjacent unmatched characters when dictionary-breaking 🔬
@jahorton
chore: Merge branch 'epic/dict-breaker' into chore/merge-master-into-…
1df99e5 
…dict-breaker
@mcdurdin
chore: move dict.ts into src
cf43c0e
@mcdurdin
Merge pull request #12317 from keymanapp/chore/merge-master-into-dict…
dad7a6c 
…-breaker

chore: merge master into dict-breaker 🔬
@mcdurdin
chore: Merge branch 'epic/dict-breaker' into chore/merge-master-into-…
c2e80b4 
…dict-breaker
@mcdurdin
chore: fixup dependency path
c4991a7
@mcdurdin
Merge pull request #12411 from keymanapp/chore/merge-master-into-dict…
367d32e 
…-breaker

chore: merge master into dict-breaker 🔬
@github-actions github-actions bot added common/ and removed common/ labels Oct 11, 2024
@github-actions github-actions bot added common/ and removed common/ labels Oct 25, 2024
@github-actions github-actions bot added common/ and removed common/ labels Nov 10, 2024
@github-actions github-actions bot added common/ and removed common/ labels Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jahorton jahorton Awaiting requested review from jahorton jahorton will be requested when the pull request is marked ready for review jahorton is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
common/models/wordbreakers/ common/models/ common/ epic-dict-breaker user-test-missing User tests have not yet been defined for the PR web/
Projects
Keyman
Status: No status
Milestone
A18S19
Development

Successfully merging this pull request may close these issues.
3 participants
@mcdurdin @jahorton @darcywong00