Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Russian articles are not extracted #223

Open
szhem opened this issue Apr 28, 2015 · 0 comments · May be fixed by #248
Open

Russian articles are not extracted #223

szhem opened this issue Apr 28, 2015 · 0 comments · May be fixed by #248

Comments

@szhem
Copy link

szhem commented Apr 28, 2015

Seems that article.cleaned_text from here #135 does not work. article_text is always empty.
Lol4t0 added a commit to Lol4t0/python-goose that referenced this issue Nov 13, 2015
@Lol4t0
Fix unicode processing +   support
964eb48 
* As STOP_WORDS are stored in unicode format we should keep our words candidates in unicode also to be able to compare candidates against dictionary correctly
* With some languages, short stopwords are linked to the next word in the sentance with no-breakable-space. To designate those stop words we should support nbsp when tokenizing.
 Russian is an example. So this fixes grangier#223
@Lol4t0 Lol4t0 linked a pull request Nov 13, 2015 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@szhem