This repository has been archived by the owner on Dec 24, 2024. It is now read-only.

Morphologic parsing tokenizer #35

Open

aliok opened this issue Dec 11, 2012 · 0 comments

Labels

Owner

aliok commented Dec 11, 2012

This would be good for deciding what to do when a dot char is seen.
If it makes sense:

numerals
roman numerals
etc.

don't separate it.

Same would go with other ambiguous points.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.